We build custom Garnet boxes for teams and individuals who need a reliable, efficient, and scalable machine to handle their machine learning and data work. The right box isn’t just about raw power—it’s about matching the compute to the work. Here’s how we decide what to put inside.
Understand the Workload
Before picking any components, we start by asking: What are you doing? Is it training large models from scratch? Fine-tuning existing models on a dataset? Running inference at scale? Or a mix of all three?
Each task has different demands. Training requires heavy compute and memory, especially for model weights and gradients. Fine-tuning needs consistent access to data and model updates. Inference demands low latency and high throughput, often with real-time or batch processing. We look at the frequency, size, and complexity of the work to shape the entire system.
Choose the Model Architecture
The model type is the first major driver. A small transformer like BERT or DistilBERT needs less RAM and VRAM than a large one like Llama-3-70B or Mixtral-8x7B. We assess the model’s parameter count, sequence length, and activation patterns.
For models with over 10 billion parameters, we usually require at least 32GB of VRAM per GPU, often more. For models beyond 50 billion parameters, we consider multi-GPU setups. The choice of model also influences whether we prioritize throughput (many small models in parallel) or latency (few large models with deep pipelines).
Size the RAM and VRAM
RAM and VRAM are the two pillars of performance. We start with the model size and memory footprint.
If the model fits in a single GPU’s VRAM, we pick a GPU with enough memory—typically 48GB or higher. For models that don’t fit, we consider either model parallelism (splitting the model across multiple GPUs) or offloading parts to system RAM.
We also account for the batch size and sequence length. Larger batches require more VRAM for activations and gradients. We estimate memory usage across all layers, including attention matrices and hidden states, and add 20–30% overhead for system buffers and data pipelines.
For training, we often double the VRAM requirement to account for gradients and optimizer states (like Adam). For inference, we focus on peak memory usage during a forward pass.
Select Storage for Weights and Data
Model weights are the largest single component. A 70B-parameter model in 16-bit precision takes about 140GB of storage. That’s just the model. Add data, checkpoints, logs, and temporary files, and storage becomes critical.
We start with SSDs—preferably NVMe—because of their speed and reliability. For models that are frequently accessed, we place the model weights on fast storage. If a model is used across multiple jobs, we keep it on a shared, high-throughput drive.
We also consider how often models are loaded and cached. If a model is loaded frequently but not continuously, we use tiered storage: fast SSDs for hot models, slower but larger drives for cold models. For real-time inference, we often use a cache layer—RAM or an SSD-based cache—to keep the most-used models ready.
We size the storage based on:
- Total model weight size
- Number of models and versions
- Data input and output volume
- Checkpoint frequency
- Temporary workspace needs
Balance Throughput and Latency
Throughput and latency are often in tension. High throughput means handling many tasks at once—ideal for batch inference or continuous training. Low latency means fast response times—critical for interactive applications.
We assess the workload’s timing needs. If models are used in a web service, we prioritize low latency. If data is processed in large batches overnight, throughput is key.
For high throughput, we use multiple GPUs, fast interconnects (like NVLink or PCIe), and efficient data pipelines. We also consider CPU cores, memory bandwidth, and I/O throughput.
For low latency, we focus on minimizing data movement. This means optimizing the model’s input pipeline, using efficient batching, and placing frequently accessed data close to the GPU.
Final System Integration
Once RAM, VRAM, storage, and throughput are aligned, we integrate the pieces.
We pick a motherboard and CPU that can handle the GPU count and memory bandwidth. We ensure the power supply is sufficient and cooling is adequate, especially for multi-GPU systems.
We also consider the operating system, containerization, and orchestration tools. The system must be easy to deploy, monitor, and scale.
We test the full stack: loading a model, running inference, training a batch, and monitoring memory and CPU usage. We adjust the configuration based on real behavior.
The Result
A custom Garnet box isn’t just a machine—it’s a tailored solution. It runs the right models, handles the data, and meets the performance expectations. It scales with the workload, adapts to new tasks, and reduces friction across the pipeline.
We don’t just spec components—we build a system that works.