My own LLM system on one small computer: what runs, how fast it is, and what went wrong
One AMD computer with 128 GB of shared memory, a gateway for the Anthropic and OpenAI APIs, open models from the Hub, and ten days of lessons. Most lessons were not about models.
Why I built it
I use Claude Code, an editor assistant and my own scripts every day. I wanted them to work with models that I run myself. My prompts stay private. There is no usage limit and no price per token. I still use the large cloud models for hard tasks. Much of my daily work does not need them.
The computer
It is a GMKtec EVO-X2 with an AMD Ryzen AI MAX+ 395. It has 128 GB of memory, and the CPU and the built-in GPU share this memory. The system is Ubuntu 24.04 with ROCm, the AMD software for GPU computing. Half of the memory is reserved for the GPU. For this reason a model of 52 GB fits completely on a built-in GPU.
The design
Claude Code, editor, scripts
|
my gateway (FastAPI)
checks access, chooses the model, controls prompt length, keeps a queue
|
the model server on the GPU (one large model in memory at a time)
plus smaller servers for speech, transcription and images, started when needed
A gateway is a small service that receives every request first. Mine understands both the Anthropic API and the OpenAI API. One decision kept it small: the gateway forwards requests and does not rebuild them. It changes the model name and passes the data through. It does not interpret streaming or tool calls, because it could get them wrong.
Names work in two steps. A client asks for a name it already knows, such as claude-sonnet-4-5 or gpt-4o. The gateway maps the name to a role. The role points to a local model. When I change a model, no client needs a change. A client can also ask for auto. Then the gateway chooses by content: images go to the vision or OCR model, audio goes to the audio model, and everything else goes to the coding model.
| Role | Model |
|---|---|
| coding (always in memory) | Qwen3-Coder-Next, 80B-A3B, Q4_K_M, 52 GB |
| reasoning | gpt-oss 120B |
| small tasks | gpt-oss 20B |
| judging and vision | Qwen3.6 35B |
| OCR | DeepSeek-OCR |
| embeddings | BAAI/bge-m3, on the CPU, so it never removes the coding model from the GPU |
| audio input | Voxtral Mini 3B, and whisper.cpp for other languages |
| speech output | Kokoro |
Most of these models come from the Hub as GGUF files.
How fast it is
I measured on 19 September 2026 with inference-benchmarker. The tool ran as a Kubernetes job on the same computer, without supervision. It tested the coding model through the gateway.
| Measurement | Result |
|---|---|
| Requests | 132, with 0 failures |
| Time to the first token, 200-token prompt, model already loaded | 0.64 s (90 % of requests: under 0.73 s) |
| Generation speed | 50 tokens per second, and 45 with an 8,000-token context |
| Reading an 8,000-token prompt | 13.7 s, about 590 tokens per second |
| Three requests at the same time | the first token takes 13 times longer; total speed rises by only 6 % |
The last row matters. This server handles one request at a time. More parallel requests do not make it faster. They only wait. So the queue in my gateway does not remove waiting. It makes waiting visible.
One practical note about the tool. All of its ready-made profiles first load the server to its maximum. On a server that handles one request at a time, this only measures the queue. I set the options by hand and used one virtual user.
I also compared my system with the Claude models on my own test set of 16 graded tasks. The Claude models scored 28 to 30 of 32. My system scored clearly lower. It failed on special cases, for example quoting rules in CSV files, and once it returned incomplete JSON. So I use it for the work it does well: first drafts, routine code, summaries, and tests for a function that I paste in. I make the decisions myself.
What went wrong
This is the main part of the article.
1. A long prompt returned another request's answer. I sent a prompt that was longer than the model's limit. The server cut the prompt to half of the limit. It returned status 200, which means success. But the answer was the answer to the previous request, word for word. On a shared server, one user could see another user's answer. I reproduced the error without my gateway. I reported it to the project with exact steps. I now run a fixed version. My gateway also counts tokens and rejects a prompt that is too long, with the normal API error. My first version of this check had its own error: it counted an image as text, and the vision and OCR models failed for one day. A check like this needs tests on both sides of the limit.
2. Shared memory hides GPU memory. When the GPU memory is full, the AMD driver moves data into normal system memory. This movement ignores the configured limit. The memory is not assigned to any program, and it does not appear in the normal memory statistics. On one bad day, 35 to 42 GiB of memory was missing from every report. The system then stopped other important programs again and again to free memory. One setting did not solve this. I needed four things together:
- A fixed upper limit for this kind of memory.
- The Linux
dmemcontrol group. It protects the GPU memory of each workload. - A rule: every heavy job must ask for permission before it uses the GPU.
- A small monitor. It checks the GPU memory of each program every two seconds.
If you run language models and image generation on one such computer, check this early.
3. The computer stopped for 8 hours and 22 minutes, and nobody was informed. No log explains why it stopped. But I know exactly why it took 8 hours. A hardware watchdog should have restarted the computer. A watchdog is a timer inside the computer's chipset. The system must reset this timer every few seconds. If the system stops responding, nobody resets the timer, and the timer restarts the computer. A program cannot do this job, because a program stops together with the computer. I had configured it, and my script reported that it was active. After the next restart it was not active any more. The Linux distribution blocks this driver by default, and the system skips a blocked driver without any message. Also, my alerts ran on the same computer, so they stopped too. During the investigation I found more problems that nobody had noticed. The system backup had failed every night for three days. The second copy of the backups was four days old. Later I found that my GPU memory log had recorded the value 0.0 for four days, because it read the wrong device. In every case a setting said "yes" while the real state said "no". I published the tools from this work: silent-failures.
4. Local models cannot review code yet. I connected serge to my gateway. A code review then ran on my own computer with my own model, and no code left my house. serge worked correctly on the first attempt. The models did not. I ran four reviews of the same change. The first review was a short approval. The second was twelve comments, and each one only said that the code was good. The third did not finish in time. The fourth used the 120B reasoning model. It reported several errors, such as a missing import. None of them existed: the same code passes the linter, the type checker and 413 tests. The coding model never used the tools that let it read files. The reasoning model tried to use tools that did not exist. I removed the review workflow. A reviewer that only praises looks like a quality check, but it is not one. I also made a similar mistake myself. I reported an error after I had only read the change. Then I wrote a test for it, and the test showed that the error did not exist. A finding is only an assumption until a test or a run confirms it.
How it was built
An AI coding agent did much of the building. It worked through a written plan on the computer itself. The plan has numbered steps. Every step has an automatic check that the agent cannot influence. There is a spending limit, and every step writes a report. The agent has no administrator rights. When a step needs such rights, it is a script that first prints what it will do, and I run it myself. Last night's speed test was one of these steps. I started it, and then I switched off my laptop.
What I use from the Hub
First, the models. I also use inference-benchmarker to get numbers that I can compare over time. I also tested text-embeddings-inference against my current embedding service, on the same texts. It was 1.3 to 3.3 times faster, and it returned the same vectors. I keep my current service for now. Nobody waits for embeddings on this computer, and the dedicated server would use about 4.5 GB of memory all the time. The test also showed a free improvement: on a CPU, eight separate requests are faster than one request with eight long texts. For a side project I use the Belarusian part of FineWeb-2. My Belarusian speech recognition work has a Space: https://huggingface.co/spaces/YauhenBichel/belarusian-asr.
My advice to someone who starts
- Plan one part of your time for the models and nine parts for operations.
- Put one protected entrance in front of everything, and close everything else.
- Reject a prompt that is too long. Never cut it silently.
- Make every check read the real state, not the configuration.
- Keep a log of what you do, with times, and include your own mistakes. My log allowed me to rebuild a fix that I had lost.
My next project is an open-source tool for computers like this one. It will show GPU memory, loaded models, jobs, requests and hidden failures on one timeline.