Where the time goes when a model loads
The reader is deciding whether to own an inference stack or rent capacity in the cloud. One of the least visible but most consequential moments in that decision is model load time: the interval between a start request and the first generated token, which is almost entirely determined by how quickly the model’s weights can be made visible to the compute engines.
On our own hardware, loading a model whose weights are already present in the page cache completes in 1.77 seconds. The kernel satisfies the page fault from cached RAM, maps the pages into the process address space, and the model is ready for token generation without any storage read. If the weights are not cached, the loader must request them from the backing device; the duration of that wait depends on the device and the level of concurrent access, and we have not measured a cold‑disk load on this platform. The measurable contrast is between a cached load of 1.77 s and an uncached load that stalls until the pages are fetched—the reader can see that gap without any ratio being computed.
When multiple models are brought up at once, each loader attempts to fault in its weight pages. If N processes fault simultaneously, the kernel issues N page‑cache misses and, when the pages are not already resident, N disk read requests. The page cache on this hardware holds the weights for roughly three models, which occupy about 61 GB of RAM. Beyond that count, additional models must be read from disk. If a burst of starts occurs, the disk I/O queue fills and each process blocks waiting for its pages, stretching effective load time well beyond the 1.77 s warm path. The mechanism that breaks down is the kernel’s concurrent page‑cache miss handler: it can service only a limited number of disk requests per interval, and the competing processes share those slots. A startup stagger—initiating models a few seconds apart—lets the page cache fill incrementally. Each subsequent loader finds its weights already cached and completes near the 1.77 s baseline; only the first starter in a burst bears the full I/O penalty. The reader sees this difference plainly: in a staggered start, the second and subsequent models report a load time of approximately 1.77 s. In an unstaggered burst, the first model may take many seconds on a saturated disk, and later models queue behind it, their start times adding up. The 31 GB swap available on this machine provides headroom, but if models exceed the page‑cache capacity and swap becomes active, the wait compounds and the reader would observe increased latency or timeouts on in‑flight requests.
Once the model is resident, thread placement governs inference throughput. On the same 120‑billion‑parameter dense model, decoding at 21.96 tokens/sec is achieved when the workload is pinned to four threads and no other work shares those cores. When the same model runs on sixteen threads shared with other work, throughput drops to 1.68 tokens/sec. The same model, the same prompt, yields either 21.96 or 1.68 tokens/sec depending on core allocation—two measured values the reader can compare directly. Separately, running attention on the CPU instead of the GPU reduces that lane to 0.094 tokens/sec for the same model and prompt. Both figures were measured on our own hardware.
A 30‑billion‑parameter mixture‑of‑experts model, run CPU‑only on four owned cores, takes about 500 seconds to produce a 1,500‑word article. This too was measured on the same system. The CPU package temperature on this board follows Tctl = 37.31 + 0.2222P_cpu + 0.05703P_gpu, with power expressed in watts. This relationship was measured under load on our system.
For a team that owns its inference node, the 1.77 s warm‑load baseline means that, after an initial page‑cache fill, subsequent model starts are fast. The risk is a cold‑start burst that saturates disk I/O and forces every later start to wait. A startup stagger across deployment waves or service restarts keeps the page cache populated incrementally, so that each new model joins the 1.77 s path rather than contending for disk bandwidth. If the reader’s workload involves many models or frequent restarts, designing the startup sequence to stagger loads is a concrete lever that prevents the kind of I/O‑driven latency spike that would otherwise become the dominant component of request time. Conversely, if the reader rents cloud inference, the same principles apply under the provider’s infrastructure, but the owner has no visibility or control over the page‑cache fill order or the stagger of concurrent loads—those decisions rest with the host. Owning the hardware means owning the page cache, and with it, the ability to shape load time through startup ordering.