Infrastructure should exist to meet a latency, throughput, training-time or research objective. Without that target, every subsystem tends to be oversized independently. To address “Begin with a service-level objective”, define models, users, concurrency, completion-time targets and expected growth before selecting a hardware tier. Test that result on NVIDIA AI Infrastructure Planner.

Different precisions, model architectures and parallelism strategies produce different accelerator requirements. To address “Convert workload into compute demand”, use benchmark or profiling data from the intended software stack to estimate GPU count and keep utilization assumptions visible. Test that result on NVIDIA AI Infrastructure Planner.

GPU HBM, CPU RAM, local NVMe and remote storage can all hold parts of the working set, but with very different cost and performance. To address “Treat memory as several pools”, assign model weights, runtime cache, preprocessing and offload data to explicit tiers instead of assuming aggregate capacity is interchangeable. Test that result on NVIDIA AI Infrastructure Planner.

Storage must load models, deliver datasets and absorb checkpoints or outputs fast enough to keep expensive compute productive. To address “Give storage a feed-rate target”, specify sustained and burst bandwidth plus capacity, then validate the complete client-to-storage path. Test that result on NVIDIA AI Infrastructure Planner.

Node count is not enough to size the fabric. Collective communication, storage I/O, service ingress and management must be considered separately. To address “Give networking a traffic matrix”, build a traffic matrix and calculate oversubscription and port counts for the topology actually being proposed. Test that result on NVIDIA AI Infrastructure Planner.

Even a well-sized GPU/NIC/NVMe list can perform poorly when components share constrained upstream links or cross NUMA boundaries. To address “Map hardware onto PCIe and NUMA”, use the server block diagram and slot map to validate locality before freezing the bill of materials. Test that result on NVIDIA AI Infrastructure Planner.

Server power and heat accumulate at the rack, where feeds, PDUs, UPS architecture and cooling systems impose their own limits. To address “Translate IT load into rack requirements”, calculate normal, peak and failure-state rack loads and reserve capacity before adding another node. Test that result on NVIDIA AI Infrastructure Planner.

Air cooling may be sufficient for lower-density systems, while rack-scale platforms can require direct liquid cooling or facility-water integration. To address “Select cooling from sustained density”, use OEM thermal requirements and a qualified mechanical review rather than extrapolating from a desktop GPU build. Test that result on NVIDIA AI Infrastructure Planner.

Infrastructure is reliable when the service can survive the failures that matter, not simply when components have redundant labels. To address “Design the failure model”, identify single points of failure across power, switching, storage, management and scheduler capacity and decide the recovery method for each. Test that result on NVIDIA AI Infrastructure Planner.

GPU utilization alone cannot explain storage stalls, network congestion, thermal throttling or PDU limits. To address “Build observability into the architecture”, collect cross-layer telemetry and use common timestamps so infrastructure events can be correlated with job behavior. Test that result on NVIDIA AI Infrastructure Planner.