You are deciding whether to own your inference stack or rent it. Renting moves the OOM kernel, the thread scheduler, and the context-window boundary onto someone else’s host; you trade responsibility for visibility. Owning your hardware—an AMD Ryzen Threadripper PRO 9955WX, 251 GB RAM, 31 GB swap, and one NVIDIA RTX 5090 with 32 GB VRAM under a 600 W limit—puts those boundaries in your lap. The measurements below were taken on that machine. They illustrate what can go wrong when you are the one holding the lease.
OOM killer choosing your model server
This machine provides 251 GB of RAM and 31 GB of swap. Three models at once occupy about 61 GB of RAM. That leaves ample address space for a single large model and its KV cache, but the working set grows as decoding proceeds. If the sum of model weights, active context, and OS overhead exceeds the 251 GB ceiling, the kernel’s OOM reaper may target the inference process for termination, or the swap daemon may begin paging weight files out to the 31 GB swap area. In either case the operator sees the generation halt abruptly—a SIGKILL in the logs, or a progressive slowdown as the system resorts to swap. The reader who has never managed a sovereign stack may mistake the kill for a software bug or a model limitation; the true cause is the kernel’s pressure algorithm acting on the 251 GB / 31 GB resource boundary that you now control.
A lane that reports healthy while generating nothing
Thread placement dominates CPU inference. The same 120‑billion‑parameter dense model decoded at 21.96 tokens/sec on four threads pinned to cores used for nothing else. When those same sixteen threads are shared with other work, the model produces 1.68 tokens/sec. The process still runs, the CPU still reports utilization, but useful decoding cycles have been fragmented across scheduler contexts, interrupt handlers, and background daemons. The reader monitoring top or htop will see the process listed as active, perhaps even consuming significant CPU time, yet the token rate has collapsed. The gap between 21.96 and 1.68 is not a statistic I compute; it is the observable difference between a thread placement that grants exclusive core time and one that shares cores with contending workloads. Proper pinning is achieved with taskset -c 0-3 or numactl --cpunodebind 0 --membind 0, which restricts the decoding thread to a specific core and memory node. Without it, the scheduler may park decoding work on any available thread, including those handling network stacks or power management, and the reader sees the generation crawl while the process appears healthy.
A context limit that truncates silently mid‑answer
The sovereign stack gives you direct access to the context window configuration, but the effective limit is bounded by physical RAM and swap. If the active context—model weights plus KV cache for the attended tokens—grows toward the 251 GB ceiling, the kernel may begin reclaiming pages. With only 31 GB of swap available, the system cannot indefinitely offload weight pages; it may instead invoke the OOM killer, or the inference engine may silently drop the oldest tokens to stay within the allocated memory budget. The reader may observe a generation that stops mid‑sentence or returns a response that cuts off without error, and because no crash signal is raised, the operator may attribute it to a model quirk rather than a resource boundary. The measurements of 251 GB RAM, 31 GB swap, and the 1.77‑second load time for a model already in the page cache are the only numbers I cite here; they mark the edges of the space within which the context window must fit.
# Pin decoding to four specific cores, excluding background work
taskset -c 0-3 llama-cli --model ~/models/120B.gguf --threads 4
Owning your inference stack means you are the one who decides where the OOM killer looks, which cores decode your tokens, and where the context window ends. The measurements above are the only hard numbers on this machine; the failure modes they reveal are the price of that control. If you are not prepared to manage thread placement, memory headroom, and kernel boundaries, renting may still be the safer path. But if you choose to own, these are the failure modes you will face, day after day.