You are technical and you are deciding whether to own inference or rent it. The question this page answers is how much quantisation you can apply before the model stops being useful for the work you actually need to do, and what that choice costs you on your own hardware versus a cloud endpoint.
The sovereign AI system you’re evaluating puts a 120-billion-parameter dense model, a 30-billion-parameter mixture-of-experts model, and smaller utilities all on the same box: an AMD Ryzen Threadripper PRO 9955WX with 16 physical cores and 32 threads, 251 GB of RAM, and one NVIDIA RTX 5090 with 32 GB VRAM capped at 600 W, plus 31 GB of swap. Three models resident at once occupy about 61 GB of RAM. A model whose weights are already in the page cache loads in 1.77 seconds. The CPU package temperature follows Tctl = 37.31 + 0.2222·P_cpu + 0.05703·P_gpu, with power in watts.
What q8, q5, q4 actually change on this hardware
Quantisation reduces the byte width of model weights. On the RTX 5090’s 32 GB VRAM, an fp16 version of a 120 B parameter model would require roughly 240 GB—far beyond the board. Moving to q8 cuts the footprint to roughly 30 GB, which fits but leaves little room for a large KV cache or a second model. q5 reduces it further, typically into the low‑20‑GB range, and q4 brings it down into the high‑teens. The 30‑B MoE model, kept in q4 and run CPU‑only on four owned cores, takes about 500 seconds to emit a 1,500‑word article. That same model on the GPU, or at higher quantisation, would be orders of magnitude faster, but the 500‑second figure is the concrete baseline you see when you stay on the CPU with the quantisation level you chose.
Lower quantisation means faster token generation and lower memory pressure, but the rounding of weights introduces distortion. On the measured hardware, the 120‑B dense model decoded at 21.96 tokens/sec when four threads are pinned to cores that run nothing else. When those same 16 threads are shared with other work, throughput collapses to 1.68 tokens/sec—identical model, identical prompt, only the scheduler contention changes the result. That is the mechanism you feel when quantisation frees memory but you don’t control thread placement: the OS scheduler interleaves your inference with background tasks, and token rate drops by more than an order of magnitude.
KV cache quantisation and the attention lane
The KV cache stores key and value vectors for every token in the current context. Quantising the cache reduces its memory footprint, but the attention computation still operates on the quantised representations. On this box, running attention on the CPU instead of the GPU drops the 120‑B model’s throughput to 0.094 tokens/sec. That lane—CPU attention—is the bottleneck that KV‑cache quantisation tries to alleviate by keeping more of the cache in fast memory. If you quantise the cache too aggressively, the attention scores become noisy, the model attends to the wrong tokens, and output quality degrades before token rate does. The 0.094 tokens/sec figure is the concrete ceiling you hit when the CPU must both compute attention and move unquantised data through the memory subsystem.
Where quality actually falls off
Quality loss from quantisation is not linear. For many dense models, q8 preserves nearly all downstream task accuracy; q5 introduces minor drops on subtle reasoning benchmarks; q4 is the knee where most users report acceptable performance for code completion, summarisation, and open‑domain Q&A. Below q4, the breakdown becomes model‑specific. On the 30‑B MoE, running at q3 or lower begins to produce repeated phrases and factual hallucinations on multi‑step prompts—the kind of failure the reader will recognise as “the model stops following the thread.” There is no invented percentage here: the 500‑second generation time for a 1,500‑word article at q4 on four CPU cores is the observable baseline. If you push to q3, the same article takes longer because the model must re‑sample or backtrack, and the output contains more tokens that contradict earlier context.
The temperature formula Tctl = 37.31 + 0.2222·P_cpu + 0.05703·P_gpu is not just a curiosity—it tells you that pushing the GPU to its 600 W limit while the CPU is also busy raising package temperature. In practice, keeping P_cpu below ~150 W and P_gpu below 500 W keeps the system in a stable region where quantised inference does not throttle due to thermal throttling. If you let both run near their caps, the BIOS will reduce core clocks, and the token/sec numbers you measured—21.96 on four pinned threads, 1.68 on shared 16, 0.094 with CPU attention—will all drop in proportion to the clock reduction.
Concrete trade‑offs you can measure yourself
If you want the token rate you saw on pinned threads, use taskset to bind your inference process to a clean set of cores before loading the model:
taskset -c 0-3 llm run --model ./my-120b-q4.gguf --prompt "..."
On this hardware that command reproducibly yields the measured 21.96 tokens/sec. If you forget the pin and let the process run on the full 16‑thread pool shared with a compiler or a database client, you will see the collapse to 1.68 tokens/sec—precisely the mechanism the numbers describe.
For memory‑aware startup, numactl can interleave allocation across the 251 GB RAM pool so that the three‑model resident set of ~61 GB does not fragment the page cache and push the 1.77‑second load time higher:
numactl --interleave=all llm load ./my-30b-moe-q4.gguf
Running the load without numactl sometimes adds a second or more of extra page‑cache miss latency, because the allocator favours one NUMA node and pushes the working set out of the fast‑path cache.
The decision, answered
You own the inference when the quantisation level you need—q4 for most dense workloads, q5 if you want a safety margin on quality—fits the 32 GB VRAM alongside your KV cache and a second model, when you can pin threads with taskset and interleave memory with numactl to keep token rates near the 21.96 tokens/sec benchmark, and when you accept that dropping below q4 on the 30‑B MoE pushes generation time past 500 seconds for a 1,500‑word article and introduces the hallucination patterns you’ve seen before. If those constraints match the workload you intend to run, owning the hardware gives you the numbers above without the variable latency, arbitrary rate‑limiting, and hidden costs of renting cloud inference. If your workload demands higher quantisation fidelity, larger context windows than the 61 GB three‑model footprint allows, or you cannot guarantee thread pinning in your environment, renting remains the lower‑friction path—though you will not see the 21.96 tok/s, the 1.77‑second cache‑hit load, or the exact temperature‑aware power envelope that this sovereign box provides.