You are choosing hardware for a sovereign AI system and deciding how much RAM to dedicate to keeping model weights resident, versus how much to leave for the operating system, page cache, and concurrent processes. That decision is the boundary between an inference server that stays in the fast lane and one that slides off a swap cliff the moment a second model is loaded.
Measured on our own hardware, loading a model whose weights were already in the page cache took 1.77 seconds—this is the lower‑bound time from process start to first token when all weight pages are hot in kernel memory. If the page cache is cold, the kernel must fetch pages from the 31 GB swap area configured on the Threadripper PRO 9955WX rig; each page fault adds latency that scales with the number of models competing for the same RAM, and in practice this can add several seconds per model load and stall every subsequent token until the fault is resolved.
Measured on our own hardware, a 120‑billion‑parameter dense model decoded at 21.96 tokens/sec when four threads were pinned to cores that had no other work running. When those same 16 threads were shared with other processes, throughput collapsed to 1.68 tokens/sec, identical model, identical prompt. The mechanism is cache‑line interference, memory‑controller request‑queue saturation, and TLB misses: when multiple processes occupy the same physical cores, the L1/L2 caches and the memory subsystem become a shared bottleneck, and the token‑per‑second rate drops by more than an order of magnitude. If you pack too many model‑generation threads onto the same socket, you will see the same collapse, even if all weights reside in RAM.
Measured on our own hardware, with attention computed on the CPU instead of the GPU, that lane fell to 0.094 tokens/sec. That is a dramatic reduction compared to the GPU‑accelerated 21.96 tokens/sec, and it occurs because the CPU must materialize attention matrices in system RAM, traverse the memory subsystem, and serialize the result. On a sovereign box you typically keep attention on the GPU and only fall back to CPU when VRAM is exhausted; the temperature formula for this board, Tctl = 37.31 + 0.2222 P_cpu + 0.05703 P_gpu with P in watts, shows that sustained CPU‑only attention raises package temperature quickly, and the 600 W limit on the RTX 5090 means you cannot simply increase GPU power to compensate—you are constrained by the total envelope.
Measured on our own hardware, a 30‑billion‑parameter mixture‑of‑experts model, CPU‑only on four owned cores, takes about 500 seconds to write a 1,500‑word article. If those weights are forced into the 31 GB swap, the write time balloons because each token requires a page fault that retrieves expert parameters from disk. The measurement “three models resident at once occupied about 61 GB of RAM” is a practical upper bound: it is the sum of weights for three models that can stay entirely in the resident RAM pool without touching swap, leaving the remainder of the 251 GB for the OS, page cache, and a fourth smaller model. Attempting a fourth model whose weights push the total past 251 GB triggers the kernel to swap out pages from the least‑recently‑used models, and you will observe the same throughput collapse the thread‑placement numbers warn about.
Sizing RAM for a multi‑model sovereign rig therefore breaks into three concrete steps. First, sum the weights of every model you want simultaneously resident, including KV‑cache space, attention buffers, and OS overhead. Second, verify the total stays below your RAM; on this platform with 251 GB, keeping about 61 GB free for the kernel and page cache gives a stable zone where mlock can lock all model pages without exceeding physical RAM. Third, pin generation threads to uncrowded cores using taskset -c 0-3 or numactl --interleave=all at launch; the 21.96 versus 1.68 token/sec measurement is the direct evidence of what happens when you don’t. If you are running CPU‑only or offloading attention to the CPU, remember the 0.094 token/sec ceiling and the temperature coefficient that will throttle your package as P_cpu rises.
The performance cliff you are guarding against is this: once any model’s weights are evicted from the page cache and into swap, every token generation incurs a page fault. The kernel must locate a free page, write back any dirty pages, read the model weight from the swap store, and only then can generation continue. On NVMe‑backed swap the fault latency is tens of milliseconds; on HDD it is hundreds. Because token generation is a tight loop, even one fault per dozens of tokens can cut effective throughput substantially. With mlockall or mlock you can lock model pages into physical RAM and prevent the kernel from paging them out, but the call will fail with ENOMEM if the total locked region plus OS overhead exceeds available RAM, and the process must then fall back to un‑pinned operation—exactly the moment the cliff appears.
Size RAM to hold all resident model weights plus overhead for the page cache, kernel structures, and KV‑cache growth during generation. On the Threadripper PRO 9955WX with 251 GB, keeping about 61 GB free for the kernel and page cache accommodates the “three models resident at once ~61 GB” benchmark and leaves room for larger configurations. Use taskset or numactl when launching model workers to keep threads on dedicated cores, and treat the 1.77‑second page‑cache load time as your target lower bound for model initialization. If you exceed the RAM budget, the 500‑second/1,500‑word MoE write time is your first indicator that swapping has engaged; the token‑per‑second numbers from the thread‑placement and attention‑placement measurements are the downstream symptoms you will see in logs and latency profiles.
The decision you are facing is whether to over‑provision RAM so every model stays resident, uses the page cache, and generates tokens at the rates the hardware can deliver, or to under‑provision and accept the swap cliff that drags latency through the floor and collapses throughput to a fraction of what the silicon is capable of. There is no middle ground: the measurements show that once weights leave RAM, the drop is not gradual—it is a collapse. Size for residency, pin your threads, and keep the page cache hot, and the sovereign inference stack will deliver the tokens‑per‑second numbers the hardware was built for.