AI Infrastructure
AI Infrastructure Basics for GPU Clusters and Model-Serving Systems
A practical map of the layers that shape AI behavior in production: workload shape, GPUs, topology, schedulers, storage, networks, cooling, serving patterns, batching, and validation evidence.
Context
AI infrastructure is no longer just a place to run training jobs. Production AI now lives in serving paths, batch systems, retrieval pipelines, agentic workflows, fine-tuning loops, and customer-facing products where latency, cost, availability, and operator control all matter at the same time.
The public direction from NVIDIA, Kubernetes, and specialist AI clouds points to the same conclusion: useful AI infrastructure is full-stack. GPUs matter, but so do CPU and memory balance, PCIe and NVLink topology, NIC locality, fabric behavior, storage class, cooling, scheduling policy, model-server behavior, and the evidence that ties all of those layers to the workload.
Kubernetes gives platform teams lifecycle, policy, service discovery, storage attachment, rollout control, and a place for device plugins and GPU operators to automate node-level software. Slurm gives researchers and training teams queues, partitions, accounting, reservations, and topology-aware job semantics. Production platforms often need both operating models, but they need a clear contract for which layer owns scheduling, quota, logs, failure recovery, and customer-visible evidence.
Inference deserves special attention because it is where AI products meet users. A demo endpoint can be slow once in a while; a production inference path needs predictable tail latency, warmup behavior, batching policy, retry behavior, model loading, cache behavior, cost visibility, and rollback. Batch inference, streaming inference, online inference, and human-reviewed automation are different operating models, not minor variations of the same service.
The most reliable architecture reviews start from workload shape rather than vendor nouns. A team should be able to explain the request path, the hot data path, the scheduler contract, the storage behavior, the failure domains, and the acceptance test before trusting a cluster with customer traffic or expensive training runs.
Decision Guide
Frame the decision before choosing the architecture.
Decision
Which serving, GPU, scheduler, storage, and network choices are actually required for the product promise?
Who It Helps
Teams planning model-serving systems, GPU clusters, private AI platforms, or infrastructure-backed AI products.
Proof to Look For
Representative workload tests, tail latency, batching behavior, topology evidence, storage pressure, capacity model, and operator runbooks.
01
Start with the Workload, Not the GPU
The same accelerator can behave very differently under multi-node training, fine-tuning, batch inference, low-latency serving, retrieval, simulation, or an agent workflow that makes many calls per user action. The architecture should begin by naming the workload shape, traffic pattern, latency budget, state movement, and failure tolerance.
A useful review separates proof-of-concept success from production behavior. The early question is whether the system can produce an answer. The production question is whether it can keep producing answers with predictable latency, cost, capacity, rollout behavior, and operator visibility when demand shifts.
02
The Control Plane Is Part of the Product
Kubernetes, Slurm, VMs, bare metal, model-serving frameworks, and managed inference providers all create different ownership boundaries. Kubernetes is strong for declarative lifecycle, policy, service routing, and elastic platform operation. Slurm is strong for finite allocations, queues, partitions, accounting, reservations, and familiar research workflows. VMs and bare metal may be better when isolation, direct device ownership, predictable topology, or simpler failure domains matter more than elastic abstraction.
The wrong control-plane choice usually shows up as operational confusion: nobody knows which layer owns placement, drains, quota, logs, failed jobs, network attachments, storage mounts, image lifecycle, or customer evidence. The architecture should make those ownership lines visible before the first incident.
03
Networking and Storage Decide Real Performance
GPU workloads move more than compute. They move tensors, gradients, checkpoints, embeddings, model artifacts, cache, metadata, logs, and inference outputs through specific storage systems and network paths. Normal pod networking may be fine for many services, while multi-node training or storage-heavy serving can require SR-IOV, RDMA, Multus, tuned MTU, correct GID selection, and NIC-to-GPU locality.
Storage also needs workload-specific language. Local NVMe, object storage, shared filesystems, VAST, WEKA, Ceph, and Rook/Ceph inside Kubernetes solve different problems. Model loading, checkpoint write pressure, metadata rates, recovery behavior, and durability requirements should decide the storage path, not brand preference or diagram neatness.
04
Production Inference Needs an Operating Model
CoreWeave's production-inference framing is useful because it treats inference as reliability, cost, and control rather than only model serving. That is the right mental model: every request consumes capacity, exposes latency, creates cost, and depends on observability that can explain what happened when users notice drift.
The serving plan should say where batching happens, how warmup works, how long requests can wait, which failures are retried, how streaming differs from batch, what happens during model rollout, and how operators connect a customer symptom back to infrastructure, model-server, storage, network, or application evidence.
What to Understand
- A GPU cluster is a product system when customers depend on it. Capacity, reliability, observability, support, metering, and upgrade paths matter as much as raw accelerator inventory.
- The useful unit of design is the workload path: request, route, schedule, load model or data, execute on accelerator, write outputs, observe behavior, and recover from failure.
- Kubernetes device plugins and GPU operators can automate access to hardware, drivers, container runtimes, node labels, monitoring, MIG, GPUDirect RDMA, and related components, but they do not remove the need to validate the real data path.
- Topology is operational evidence. GPU-to-GPU, GPU-to-NIC, NUMA, PCIe generation, NVLink or fabric path, storage locality, MTU, and congestion behavior need to be visible enough for operators to explain performance.
- The serving architecture should separate batch, online, streaming, and human-reviewed workflows when their latency budgets, retry behavior, queue policy, and rollback expectations differ.
Common Failure Modes
- Teams buy the right accelerators but validate the wrong workload. Training benchmarks, single-node tests, and vendor line cards do not prove production inference, multi-node behavior, storage pressure, or customer-facing tail latency.
- The platform hides too much. When routing, batching, model loading, GPU memory behavior, cache misses, network path, and storage latency are opaque, operators can only react to symptoms after users notice them.
- Kubernetes, Slurm, VMs, and bare metal are treated as interchangeable deployment targets. They are not. Each changes scheduling, isolation, device visibility, rollout behavior, failure recovery, and evidence collection.
- Storage and networking are treated as background services. In practice, they often decide whether expensive GPUs are busy with useful work or waiting on checkpoints, artifacts, metadata, routing, congestion, or cold starts.
- The team optimizes utilization but not value. A dashboard can show busy GPUs while users wait on queues, retries, model warmup, slow storage, or a serving path that cannot meet the product promise.
What Good Looks Like
- The architecture explains the workload path in plain language: user or job entry, queue, scheduler, model server, cache, storage, accelerator, telemetry, fallback, and rollback.
- Acceptance tests are tied to the product promise. They measure throughput, latency distribution, cost behavior, failure behavior, upgrade behavior, and operator evidence for the specific workload class.
- Infrastructure choices are explicit. Kubernetes, Slurm, VMs, bare metal, managed inference, local NVMe, shared filesystems, object stores, SR-IOV, RDMA, and standard pod networking are selected because their tradeoffs match the workload.
- Operators can explain incidents from evidence: topology, driver state, device plugin state, scheduler events, storage behavior, network counters, model-server telemetry, request traces, and customer-visible symptoms.
- Capacity planning connects utilization to useful output: tokens, requests, jobs, checkpoints, user-facing latency, revenue, and support load, not only GPU hours consumed.
Quick Diagnostic
- What workload is this for: training, fine-tuning, batch inference, online inference, retrieval, simulation, or mixed use?
- Does this path need real-time latency, async batch throughput, streaming responses, offline enrichment, or human-reviewed automation?
- Is the serving path running on Kubernetes, VMs, or bare metal, and what does that change about scheduling, isolation, device access, rollout, and recovery?
- Which network layer owns the request path: ingress, gateway, service mesh, CNI, SDN policy, RDMA or Ethernet fabric, DNS, or external load balancer?
5 more in private context
Evidence to Look For
- Serving-path diagram covering gateway, router, queue, scheduler, model server, cache, accelerator, storage, telemetry, and fallback behavior.
- Deployment-substrate decision record comparing Kubernetes, VMs, and bare metal for hardware access, isolation, lifecycle, networking, storage, and operator workflow.
- Network-path evidence for ingress, service routing, CNI or SDN policy, east-west traffic, MTU, congestion, DNS, and failure isolation.
- SR-IOV or RDMA evidence covering NetworkAttachmentDefinitions, advertised device resources, NIC-to-GPU locality, MTU, GID selection, and representative workload tests.
6 more in private context
Protected Preview
- Provider-specific topology notes and benchmark logs.
- Model-serving architecture reviews for batch, real-time, and streaming workloads.
- Kubernetes, VM, and bare-metal deployment tradeoff templates for AI serving platforms.
- SDN/CNI and storage architecture review templates for GPU serving platforms.
5 more in private context
Further Resources
Apply to a Decision
Apply this to a product, infrastructure, or diligence decision.
If this resource matches a decision you need to make, these services turn the framework into a review, roadmap, validation plan, or risk assessment for a specific environment.
Private Resources
Provider-specific topology notes, benchmark logs, diagrams, and operational runbooks stay in the protected area.
View Private Resources