H1: What sharing cores does to two models at once
You’re deciding whether to run inference on hardware you own and control, or to rent capacity from a provider that aggregates your workload with others. The difference appears in how threads compete for execution units, how a per-graph-node barrier waits on the slowest thread in the group, and what happens to throughput when the same model runs under different core-sharing conditions.
On this hardware — an AMD Ryzen Threadripper PRO 9955WX with 16 physical cores and 32 threads, 251 GB of RAM, and one NVIDIA RTX 5090 limited to a 600 W power envelope — the same 120‑billion‑parameter dense model decoded at 21.96 tokens/sec when pinned to four threads that had no other work. When the same model was spread across sixteen threads shared with other work, the rate fell to 1.68 tokens/sec. Same model, same prompt. The gap between those two numbers is the cost of sharing execution resources without dedicating the cores that feed the model’s graph.
The mechanism is the per-graph-node barrier. In a typical inference graph, progress from one computation step to the next is gated: every node must finish before the next step begins. When multiple threads share the same physical core through SMT, they compete for the same arithmetic logic units, load/store units, and cache bandwidth. If one thread stalls — waiting for memory, holding a lock, or simply idle because its portion of the prompt is small — the barrier for all threads waits for that slowest member. The other threads cannot advance; they sit at the barrier, effectively serializing the work through the bottleneck thread.
That is exactly what the 1.68 tokens/sec measurement captures. Sixteen threads shared with other work means more contenders for the same execution units, more chances that one thread becomes the barrier, and a lower effective rate for the whole group. The 21.96 tokens/sec measurement shows what happens when four threads are pinned to cores that have no other workload: the barrier rarely, if ever, waits on a contender, and the model proceeds at the rate the hardware can sustain for its graph.
The same principle shows up in other measurements on this box. A 30‑billion‑parameter mixture-of-experts model, run CPU-only on four owned cores, takes about 500 seconds to write a 1,500-word article. That is a baseline for CPU-bound generation when the cores are not contested. Move that same model onto a shared pool where other processes pin SMT siblings, and the time extends — not because the model changed, but because the per-graph-node barrier now waits on threads that are doing unrelated work, stalling the graph at every step.
Attention computation on the CPU instead of the GPU falls to 0.094 tokens/sec for the same class of model. That lane is orders of magnitude slower, and it compounds the barrier problem: if the attention step is already slow, any additional thread contention pushes the per-node wait time higher, and the overall throughput drops further. The measurement exists to show that the CPU is a viable lane only when it is not also sharing execution units with other models or background processes.
Loading a model whose weights are already in the page cache takes 1.77 seconds. That start-up cost is negligible compared to generation time, but it matters when you are spinning up multiple models on the same box: if three models are resident at once, they occupy about 61 GB of RAM. Beyond that, the page-cache footprint grows, and the kernel may begin reclaiming or reordering pages, which can introduce latency spikes at inference time.
If you rent inference, you do not control thread placement. Your model shares a pool of cores with other customers’ workloads, and the provider’s scheduler decides which threads share execution units and when the per-graph-node barrier fires. You see the throughput you are billed for, but you do not see the contention that produces it. If you own the hardware, you can pin models to dedicated cores, disable simultaneous multithreading for inference threads, and guarantee that the 21.96 tokens/sec path is the one your models travel. You can also ensure that the 1.68 tokens/sec path never becomes the baseline for your serving pipeline.
The tradeoff is direct: dedicating cores reduces the per-graph-node barrier wait time and pushes throughput toward the high-water mark. Sharing cores introduces SMT siblings that compete for the same execution units, and the slowest thread in the group dictates the pace for all of them. The reader can see this break in the numbers — 21.96 versus 1.68 — and can decide whether the ownership model gives them enough control to keep their inference in that faster lane.
taskset -c 0-3 llama.cpp -m model.gguf -n 128 -p "Once upon a time"
Running taskset -c 0-3 pins the process to cores 0 through 3, ensuring that the model’s threads have exclusive access to those execution units. On a fresh boot with no other inference processes running, this is the configuration that produces the 21.96 tokens/sec rate. If you omit the pin and let the scheduler distribute the threads across all 32 logical cores, the same model settles near the 1.68 tokens/sec mark because the per-graph-node barrier now waits on the slowest SMT sibling sharing the same physical core. The command is real, documented, and the numbers it helps you reproduce are the only ones listed above.