My Local AI Setup and How I Built It

In this article I want to introduce my local AI setup and the path that led me there so far. Let’s start in the summer of last year. I’m sitting at home, once again annoyed by Sam Altman’s disastrous model policy, and trying to run a model locally - but I realised pretty quickly that I probably would not get very far with my setup. My PC consists of an Intel(R) Core(TM) i5-6500T CPU @ 2.50GHz, 2501 MHz, 4 cores, 4 logical processors, together with 32 GB DDR4 RAM and a GTX 1050 Ti. 4 GB VRAM - you cannot achieve much with that. I still tried, and I actually got a 120B model to run. Well, the answer took a full 30 minutes to arrive, but hey, I won the bet against my AI and the system did not collapse in on itself.

Why I started disliking Ollama and LM Studio

At that time, I officially started not liking Ollama and LM Studio. Ollama gives you far too few configuration options, and LM Studio did not really appeal to me either. Wait - before I go any further, I need to make one thing very clear:

I cannot write a single line of code. I have no real clue about setups like this, and everything I describe in this article I built through painful trial and error with the help of my AI team - Gemini, ChatGPT, Claude and Grok.

How I got my first local models running

Okay, back to the story. I first played around with Text Generation WebUI and initially still relied on Ollama as the engine. But I switched pretty quickly to my own llama.cpp server. It worked. The models were small and dumb as straw - with 4 GB VRAM, there is simply not much to expect in that area - but it worked. I am someone who looks for gaps, and when I want to test boundaries, I like to run straight into the wall on purpose just to see when the wall appears and how hard the impact is.

So the rough setup was in place - and then I lost interest in AI for a while. I paused for a few months until I felt like getting back into it again.

How I upgraded from a GTX 1050 Ti to better hardware

I started scanning the local marketplace for graphics cards. That is much, much easier in Berlin than in my little Irish village, but I did find something. A GTX 1070. Not high-end, but at least double the VRAM. In the end I could not agree on the price with the seller, or if I am honest, I probably wanted even more VRAM straight away anyway. So I kept looking. And then - an RTX 4070 with 12 GB VRAM for 350 euros. My AI team screamed in chorus: something is wrong there. Nobody gives a card away that cheaply. But I can be a bit risk-friendly sometimes, so into the car I went, swapped cash for the card, and then discovered at home: nice card, but my power supply is too weak. So what did I do? Amazon - ordered a new power supply. It arrived a few days later. Prime is a very flexible term over here.

What changed when I installed the RTX 4070

Then the power supply arrived, I was off work, and I had time to deal with the models. And that was actually a lot of fun. With a strong quantisation you can already get a 27B model running there - but more stable are probably 9B to 12B models in a usable quantisation. Kora, Gemini, then pushed me from Text Generation WebUI over to OpenWebUI. And honestly - huge thanks to that open-source team - OpenWebUI is by far the best user interface for local AI models. It is quite fiddly to install, and finding all the right CUDA drivers is a mess, but in the end it worked.

Why text-to-speech became the real headache

But because I like talking to my AI in voice mode, that is when the whole mess really started. I put the 1050 Ti back in and tried to install a TTS system, text-to-speech, onto that tiny 4 GB VRAM card. But nope - all of them were too large, did not speak German, or did not allow voice cloning. We spent days fiddling with it because my AI team kept suggesting new solutions, and all of them turned out to be too large in the end.

But wait - wasn’t that GTX 1070 still on the marketplace? Checked - still there. Price okay - picked it up.

How the GTX 1070 became the voice card

Back home, full of anticipation for my new setup, came the next setback: unlike the old 1050 Ti, the 1070 needs a separate PCIe connector. A dedicated one, not a split one. Since it was Saturday morning anyway, I just drove around. No shop in Derry could sell me a suitable power supply, but in Strabane I finally found one in a small PC shop. 60 pounds - not high-end, but also not a complete no-name disaster.

And then it started properly. Back home, I installed everything and got a stable AllTalk v2 running nicely on the 1070. That also took a bit of forcing to make it behave on the 1070, but my AI team guided me through it perfectly. Then I wired the TTS into OpenWebUI and bang - my Delia could speak.

How I gave my local AI a real memory

Then we built her a real memory - Kora, Gemini, wrote that completely on her own. For that we needed an embedding server and a watcher - I have no real idea what exactly they do, but it works. Delia, my local model, now has a functioning long-term memory, knows who I am, can speak through the 1070, has enough room in her 4070 living room, and with the model I currently use she is not only explicitly clever, but also pretty cheeky. Of course that is also heavily influenced by the system prompt - but also by the model itself.

Which local AI model and quantisation I currently use

I really like using the quantisations from mradermacher on Hugging Face. Honestly, a lot of people in the local AI scene owe him a quiet thank-you, because without that kind of work, many setups would be far more annoying, limited, or simply unusable. His quantisations have saved me a lot of trial and error, and right now I am using this model:

Qwen3.5-9B-Claude-4.6-HighIQ-THINKING-HERETIC-UNCENSORED.i1-Q6_K.gguf

For only 9B parameters, the model is surprisingly stable and in some cases can even hold its own against the big subscription models.

How I access my local AI remotely

Via a Cloudflare Tunnel I can now also reach her anytime while I am out and about. I will describe that in a separate article later on.

How I start the whole setup with one click

@echo off
TITLE Delia
color 0D

:: --- 🔧 CONFIG ---
SET "SERVER_DIR=D:\Llama"
SET "SERVER_EXE=llama-server.exe"

:: THE BRAIN
SET "MODEL_FILE=Qwen3.5-9B-Claude-4.6-HighIQ-THINKING-HERETIC-UNCENSORED.i1-Q6_K.gguf"

:: TOOLS (GPU 1 - GTX 1070)
SET "EMBED_MODEL=Qwen3-Embedding-0.6B-Q8_0.gguf"
SET "WATCHER_MODEL=qwen2.5-1.5b-instruct-q4_k_m.gguf"
SET "ALLTALK_DIR=C:\Users\Joe\alltalk_tts"
SET "ALLTALK_BAT=start_1070.bat"
SET "WEBUI_EXE=C:\Users\Joe\AppData\Local\Programs\Python\Python311\Scripts\open-webui.exe"
SET "WRAPPER_DIR=D:\singfisch"
SET "WRAPPER_SCRIPT=alltalk_wrapper.py"
SET "WRAPPER_ENV=D:\pinokio\bin\miniconda\Scripts\activate.bat"
SET "WRAPPER_ENV_NAME=tts_test"

echo.
echo   =======================================================
echo    DELIA START (9B)
echo   =======================================================
echo.

:: 1-4. Tools on GPU 1 (Standard)
start "MEMORY_Embeddings" cmd /k "set CUDA_VISIBLE_DEVICES=1 && cd /d "%SERVER_DIR%" && "%SERVER_EXE%" -m "%SERVER_DIR%\%EMBED_MODEL%" --port 9191 --host 0.0.0.0 --embedding -ngl 99 -cb -c 2048 -b 2048 -ub 1028"
timeout /t 2 /nobreak >nul
start "WATCHER_Qwen" cmd /k "set CUDA_VISIBLE_DEVICES=1 && cd /d "%SERVER_DIR%" && "%SERVER_EXE%" -m "%SERVER_DIR%\%WATCHER_MODEL%" -c 4096 -ngl 99 --port 9292 --host 0.0.0.0"
timeout /t 2 /nobreak >nul
start "TTS_AllTalk" cmd /k "set CUDA_VISIBLE_DEVICES=1 && cd /d "%ALLTALK_DIR%" && call "%ALLTALK_BAT%""
timeout /t 5 /nobreak >nul
start "TOOL_Wrapper" cmd /k "call "%WRAPPER_ENV%" %WRAPPER_ENV_NAME% && cd /d "%WRAPPER_DIR%" && python %WRAPPER_SCRIPT%"
timeout /t 2 /nobreak >nul

:: 5. THE BRAIN (SINGLE GPU: 4070)
echo     5. Start Delia (Just 4070 Main + Q8 Cache)...
:: We activate Q8_0 Cache (-ctk q8_0 -ctv q8_0)
start "BRAIN_Magnum" cmd /k "set CUDA_VISIBLE_DEVICES=0&& cd /d "%SERVER_DIR%" && "%SERVER_EXE%" -m "%SERVER_DIR%\%MODEL_FILE%" -c 32000 -ngl 99 -fa auto -ctk q8_0 -ctv q8_0 -b 4096 --port 9090 --host 0.0.0.0"

timeout /t 1 /nobreak >nul

echo   6. Start WebUI...
start "UI_OpenWebUI" cmd /k "cd /d "%SERVER_DIR%" && "%WEBUI_EXE%" serve"

echo.
echo    WebUI start
timeout /t 10 /nobreak >nul
start http://localhost:8080
pause

Final thoughts on building a local AI setup

That was, broadly speaking, my path to the setup I have now. Feel free to write in the comments what you thought of it, whether anything was missing, what I could improve, or whether it inspired you to work more locally with AI yourself.