SRE

How to make Kubernetes run AI workloads without turning your infrastructure budget into a forensic accounting investigation.

At 9:17 on a Tuesday morning, your company’s flagship AI chatbot begins taking twelve seconds to answer questions that used to take two. Panic quietly ripples through Slack. You open the monitoring dashboard, expecting to see silicon melting through the floor of a Virginia data center, only to find average GPU utilization sitting at a leisurely 24%. Yet Kubernetes insists, with the cold bureaucratic certainty of a municipal parking enforcement officer, that every single GPU in the cluster is 100% allocated.

Down the hall, someone on the machine learning team has just kicked off an unscheduled fine-tuning experiment that apparently requires the electrical output of a small Baltic nation. Meanwhile, your cloud bill is performing beautifully. There are zero latency issues there.

Faced with angry product managers and a wheezing chatbot, the infrastructure team reaches the most natural human conclusion available. We need to buy more GPUs.

It is a comforting instinct. GPUs are shiny, brutally expensive, and widely treated as the universal ointment for any corporate rash involving artificial intelligence. If the application is slow, add another GPU. If it is still slow, perhaps the GPU was not sufficiently aristocratic, and you need an H100 instead of an L40S. Unfortunately, Kubernetes does not automatically become a discerning AI platform just because someone installed the NVIDIA device plugin and granted a pod access to “nvidia.com/gpu”.

Running AI workloads without bankrupting your employer requires something far less glamorous than hardware procurement. It requires teaching your cluster the difference between a customer waiting for an answer and a data scientist testing a hunch. The good news is that Kubernetes already owns the machinery to enforce these manners. We just need to stop letting it run our most expensive hardware like an unsupervised toddler buffet.

Inference and training should not be roommates

The original sin of most AI platforms is treating every workload that touches a GPU as morally equivalent. Putting production inference and model training into the same unrestricted GPU pool is like forcing a polite emergency-room receptionist to share a studio apartment with a man who tests jet engines in the living room at three in the morning. They have fundamentally incompatible personalities.

Inference is an anxious people-pleaser. A human types a prompt into a chat window, and the inference engine has roughly two hundred milliseconds to begin sounding intelligent before the user loses interest and wanders off to open another browser tab. Latency matters. Predictability matters. Availability is non-negotiable.

Training, on the other hand, has the social awareness of a glacier. It moves into the cluster for four days, inhales terabytes of training data over the network, locks up eight GPUs simultaneously, and periodically pauses to dump a four-hundred-gigabyte checkpoint onto the living room carpet. It does not care about two-hundred-millisecond response times. It just wants to chew through matrix multiplications in peace until Friday.

When you toss both of these creatures into an open cluster, the outcome is mathematically inevitable. Eventually, an experimental training job will camp out on the exact hardware needed by your customer-facing API. Kubernetes will not consider this a scandal. As far as the default scheduler is concerned, a pod asked for a GPU, a GPU was sitting there, and the paperwork was stamped in triplicate. The humans simply forgot to mention which pod actually pays everyone’s salaries.

Give your GPUs some boundaries

Before debating which NVIDIA architecture belongs on your roadmap, you have to establish basic zoning laws. Production inference needs protected capacity where opportunistic workloads are legally barred from entry. Training deserves its own playground, ideally wired up to cheaper, interruptible cloud instances. Experimentation and Jupyter notebooks belong in a tightly fenced sandbox where a forgotten loop cannot accidentally annex the company’s quarterly profits.

You do not need three sprawling, physically separate clusters on day one. You just need explicit boundaries using mechanisms Kubernetes has had for a decade. Node labels identify the hardware, taints act as a velvet rope to keep uninvited pods off the premises, and tolerations serve as the VIP pass for approved workloads.

However, many platform teams learn the hard way that a toleration is merely permission to enter a node, not a reservation. Without node affinity to steer pods toward their designated pools and “ResourceQuota” objects to cap total consumption, a Friday afternoon experiment can still wander into the wrong neighborhood. And while you are fencing off the silicon, do not neglect the unglamorous supporting cast. A top-tier GPU starved of CPU threads, system RAM, or storage bandwidth to load its model weights is nothing more than a thirty-thousand-dollar space heater with driver issues.

In practice, the thin line between a responsive Tuesday morning API and an all-hands incident call usually looks like twenty lines of YAML that nobody got around to writing:

apiVersion: v1
kind: Pod
metadata:
  name: customer-chatbot-vllm
  namespace: ai-production
spec:
  priorityClassName: production-inference-critical
  nodeSelector:
    workload-class: gpu-inference
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "inference-only"
      effect: "NoSchedule" # Keeps experimental training jobs off the lawn
  containers:
    - name: vllm-engine
      image: vllm/vllm-openai:latest
      resources:
        limits:
          nvidia.com/gpu: "1" # A hard ceiling, not a polite suggestion
          memory: "48Gi"
        requests:
          cpu: "8" # Prevents our expensive GPU from starving for data
          memory: "48Gi"
          nvidia.com/gpu: "1"

Stop autoscaling on CPU and hoping for the best

Once the workloads are separated, the next self-inflicted wound is usually autoscaling. Traditional Kubernetes autoscaling revolves around CPU utilization. For a stateless web server, watching the CPU is sensible. For an LLM inference server, scaling on CPU utilization is like trying to guess how many people are waiting outside a restaurant by measuring the chef’s shoe size.

During inference, the heavy lifting happens inside the GPU’s tensor cores and high-bandwidth memory, while the host CPU mostly orchestrates tokenization and network plumbing. Your chatbot pod can display a serene 15% CPU utilization while forty angry users sit backed up in the inference queue waiting for a slot in the KV cache.

If you want autoscaling to reflect reality, you have to scale on the backlog of actual work. Modern serving engines like vLLM expose metrics such as “vllm:num_requests_waiting” and token cache usage directly to Prometheus. Pairing those signals with KEDA (Kubernetes Event-driven Autoscaling) lets your cluster react to the line forming outside the door rather than the temperature of the hallway.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-inference-scaler
  namespace: ai-production
spec:
  scaleTargetRef:
    name: customer-chatbot-vllm
  minReplicaCount: 1 # Set to 0 only if an external proxy is watching the door
  maxReplicaCount: 6 # Also known as the CFO's blood pressure limit
  cooldownPeriod: 300 # Wait 5 minutes before killing a warm, expensive GPU
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-operated.monitoring.svc:9090
        metricName: vllm_num_requests_waiting
        query: sum(vllm:num_requests_waiting{service="customer-chatbot"})
        threshold: "5" # Wake up another replica when 5 prompts are stuck in the lobby

Even with KEDA, there are two classic traps waiting for the optimistic engineer. First, when HPA asks for another pod and the cluster autoscaler asks your cloud provider for another GPU node, that node only materializes if the cloud provider actually has spare GPUs in that availability zone. Kubernetes cannot manufacture silicon out of enthusiasm.

Second, there is the existential comedy of scaling to zero. Everyone loves the idea of scaling idle GPU deployments down to zero pods overnight to save money. Right up until they realize that if the autoscaler relies on a metric exported by the inference pod itself, scaling to zero means you have just euthanized the only witness capable of reporting that customers have arrived. The pod is gone, the Prometheus metric vanishes with it, and Kubernetes sits in blissful silence while early-morning users stare at a spinning wheel. Unless an external gateway or message queue stays awake outside the deployment to poke KEDA with a stick, your service has achieved permanent financial efficiency by ceasing to exist.

The cheapest GPU is the one you do not have to panic-start

Autoscaling is usually sold in conference talks as a performance feature, but in an AI cluster, it is primarily a defense mechanism against bankruptcy. Setting “minReplicaCount” and “maxReplicaCount” gives you guardrails, provided you understand the physics of what happens between those two numbers.

Spinning up a fresh GPU pod is not like launching a twenty-megabyte Go microservice. A new node can take several minutes to provision and attach drivers, followed by the geological epoch required to pull a container image and haul seventy gigabytes of model weights across the network into VRAM. If your scale-down policy is too twitchy, your cluster enters an expensive cardiovascular workout: traffic dips for forty seconds, Kubernetes terminates a GPU replica, three users show up, Kubernetes panics and provisions a fresh GPU node, the model spends four minutes loading, and by the time it is ready, the users have left.

Give your inference workloads generous cooldown periods and keep warm replicas alive for anything facing a real human. Save aggressive scale-to-zero policies for staging environments, internal batch summaries, and those pet projects that executives insist are mission-critical but only visit once a quarter.

Batching beats a procurement meeting

Let us return to our Tuesday morning crime scene. Our chatbot GPU was only at 24% utilization, yet latency was terrible. Before signing a purchase order for more hardware, it is worth asking why three-quarters of that GPU was essentially daydreaming.

Often, the culprit is naive request handling. Processing LLM prompts one by one in strict isolation is like running a city bus system where the driver refuses to let a second passenger board until the first passenger has been driven all the way across town and dropped off at their front porch. The bus is technically occupied the entire time, so Kubernetes marks it as allocated, even though fifty-eight seats are empty.

Inference engines solve this through batching. NVIDIA Triton uses dynamic batching to group incoming requests over a tiny time window, while LLM engines like vLLM rely on continuous batching, which slips new prompts into the GPU mid-flight as older prompts finish generating tokens. Done right, continuous batching can multiply throughput on your existing hardware without spending an extra dime.

Naturally, there is a catch. Batching is a game of chicken with user patience. Pack too many long prompts together, and you exhaust the GPU’s KV cache memory or stretch out the time-per-output-token until the text trickles onto the user’s screen like cold molasses. A massive batch size that looks heroic on a throughput benchmark while obliterating your p95 latency target is not an optimization. It is just a service outage with very impressive statistics.

Priorities, evictions, and the myth of the untested checkpoint

Sooner or later, despite your best architectural diagrams, resource contention will happen. A researcher will submit a legitimate distributed training job requesting eight GPUs at the exact moment your cluster only has six to spare.

This is where governance tools like Kubernetes “PriorityClass” and batch queuing systems like Kueue earn their keep. Instead of letting eight training pods schedule six of their siblings and deadlock the cluster while waiting for the missing two (the distributed computing equivalent of chartering three-quarters of an airplane), Kueue holds the entire job in a polite waiting room until the full block of quota is available. Meanwhile, priority classes ensure that if production inference suddenly needs to scale up, Kubernetes knows which experimental pods can be unceremoniously thrown out the airlock to make room.

Of course, evicting training jobs (or running them on cheap cloud Spot instances that can vanish with a two-minute warning) only saves money if those jobs are built to survive sudden death. A nine-hour training run that loses all its progress at hour eight because nobody configured periodic checkpoints is not a victory for FinOps. It is just an discounted way to heat the atmosphere. And remember to actually test restoring from those checkpoints; a recovery script nobody has ever run is not a disaster strategy, it is a bedtime story.

GPU sharing is great until the neighbors move in

When management discovers that several teams are requesting entire GPUs only to use 15% of their compute, someone inevitably suggests GPU sharing. On paper, it sounds like urban density planning at its finest. In practice, it depends entirely on whether you build walls between the apartments.

NVIDIA time-slicing is the technological equivalent of an airline selling four tickets for the exact same middle seat and hoping the passengers agree to take turns sitting down every few milliseconds. It works surprisingly well for lightweight notebooks and polite internal tools. However, time-slicing provides zero memory isolation. All those pods share the same pool of physical VRAM. The moment one developer’s script decides to load an oversized model or leaks memory, the resulting out-of-memory panic does not just crash the offending pod. It ejects the entire row of passengers without a parachute.

If you need predictable isolation on shared hardware, NVIDIA Multi-Instance GPU (MIG) is the adult alternative. Supported on higher-end cards like the A100 and H100, MIG physically partitions the GPU into independent hardware slices, each with its own dedicated compute cores and walled-off memory. Your workloads get smaller GPUs, but at least their neighbors cannot burn the house down.

How to keep the Finance Department out of your Slack channels

You do not need to build a miniature hyperscaler to stop your cluster from haemorrhaging cash, nor do you need a hundred-page governance manifesto. You just need a handful of enforceable habits before Monday morning traffic arrives:

• Segregate the creatures: Never let long-running training jobs and latency-sensitive inference pods compete for the same unguarded node pool.
• Retire CPU-based scaling for AI: Wire KEDA up to actual inference queue depth or token wait times, and make sure an external proxy is watching the door if you ever scale to zero.
• Tune the engine before buying a second car: Benchmark continuous batching, concurrency limits, and KV cache utilization before assuming you are out of raw compute.
• Make batch jobs wait their turn: Use Kueue and “PriorityClass” so distributed jobs launch all together or not at all, and ensure training code checkpoints often enough to survive preemption.
• Partition with hardware, not hope: Use time-slicing for disposable dev sandboxes, but demand MIG whenever shared workloads actually matter.
• Attach a name tag to every watt: Combine NVIDIA DCGM telemetry, inference metrics, and namespace cost allocation so you can measure cost per thousand tokens rather than staring blindly at a monthly lump sum. A namespace without quotas is a blank check, and a model without cost attribution is just an accounting mystery that happens to use CUDA.

The expensive part was never just the silicon

Let us check back in on our struggling chatbot from Tuesday at 9:17 AM.

Once you put proper observability on the cluster, the mystery usually evaporates in about four minutes. You discover that the chatbot did not run out of raw GPU horsepower at all. Instead, an unthrottled training experiment wandered onto the shared node pool, saturated the node’s network interface pulling a massive dataset, and left a single inference replica choking on an unbatched queue of seventeen concurrent users while HPA stared calmly at a 20% CPU graph and did nothing.

Buying four more GPUs would have temporarily masked the symptoms, right up until the next unscheduled experiment expanded to consume those too.

Kubernetes is phenomenally good at packing containers onto servers. It is terrible at reading your company’s balance sheet or guessing which workloads deserve thirty-thousand-dollar accelerators. Before you approve the next hardware requisition, force your platform to answer three simple questions: what is waiting in line, who owns the pod blocking the aisle, and how much does each answer actually cost us?

If nobody in the room can answer all three, your cluster does not need another GPU. It needs a chaperone.