October 2026

Why your Kubernetes GPUs are 100% allocated and 76% bored

How to make Kubernetes run AI workloads without turning your infrastructure budget into a forensic accounting investigation.

At 9:17 on a Tuesday morning, your company’s flagship AI chatbot begins taking twelve seconds to answer questions that used to take two. Panic quietly ripples through Slack. You open the monitoring dashboard, expecting to see silicon melting through the floor of a Virginia data center, only to find average GPU utilization sitting at a leisurely 24%. Yet Kubernetes insists, with the cold bureaucratic certainty of a municipal parking enforcement officer, that every single GPU in the cluster is 100% allocated.

Down the hall, someone on the machine learning team has just kicked off an unscheduled fine-tuning experiment that apparently requires the electrical output of a small Baltic nation. Meanwhile, your cloud bill is performing beautifully. There are zero latency issues there.

Faced with angry product managers and a wheezing chatbot, the infrastructure team reaches the most natural human conclusion available. We need to buy more GPUs.

It is a comforting instinct. GPUs are shiny, brutally expensive, and widely treated as the universal ointment for any corporate rash involving artificial intelligence. If the application is slow, add another GPU. If it is still slow, perhaps the GPU was not sufficiently aristocratic, and you need an H100 instead of an L40S. Unfortunately, Kubernetes does not automatically become a discerning AI platform just because someone installed the NVIDIA device plugin and granted a pod access to “nvidia.com/gpu”.

Running AI workloads without bankrupting your employer requires something far less glamorous than hardware procurement. It requires teaching your cluster the difference between a customer waiting for an answer and a data scientist testing a hunch. The good news is that Kubernetes already owns the machinery to enforce these manners. We just need to stop letting it run our most expensive hardware like an unsupervised toddler buffet.

Inference and training should not be roommates

The original sin of most AI platforms is treating every workload that touches a GPU as morally equivalent. Putting production inference and model training into the same unrestricted GPU pool is like forcing a polite emergency-room receptionist to share a studio apartment with a man who tests jet engines in the living room at three in the morning. They have fundamentally incompatible personalities.

Inference is an anxious people-pleaser. A human types a prompt into a chat window, and the inference engine has roughly two hundred milliseconds to begin sounding intelligent before the user loses interest and wanders off to open another browser tab. Latency matters. Predictability matters. Availability is non-negotiable.

Training, on the other hand, has the social awareness of a glacier. It moves into the cluster for four days, inhales terabytes of training data over the network, locks up eight GPUs simultaneously, and periodically pauses to dump a four-hundred-gigabyte checkpoint onto the living room carpet. It does not care about two-hundred-millisecond response times. It just wants to chew through matrix multiplications in peace until Friday.

When you toss both of these creatures into an open cluster, the outcome is mathematically inevitable. Eventually, an experimental training job will camp out on the exact hardware needed by your customer-facing API. Kubernetes will not consider this a scandal. As far as the default scheduler is concerned, a pod asked for a GPU, a GPU was sitting there, and the paperwork was stamped in triplicate. The humans simply forgot to mention which pod actually pays everyone’s salaries.

Give your GPUs some boundaries

Before debating which NVIDIA architecture belongs on your roadmap, you have to establish basic zoning laws. Production inference needs protected capacity where opportunistic workloads are legally barred from entry. Training deserves its own playground, ideally wired up to cheaper, interruptible cloud instances. Experimentation and Jupyter notebooks belong in a tightly fenced sandbox where a forgotten loop cannot accidentally annex the company’s quarterly profits.

You do not need three sprawling, physically separate clusters on day one. You just need explicit boundaries using mechanisms Kubernetes has had for a decade. Node labels identify the hardware, taints act as a velvet rope to keep uninvited pods off the premises, and tolerations serve as the VIP pass for approved workloads.

However, many platform teams learn the hard way that a toleration is merely permission to enter a node, not a reservation. Without node affinity to steer pods toward their designated pools and “ResourceQuota” objects to cap total consumption, a Friday afternoon experiment can still wander into the wrong neighborhood. And while you are fencing off the silicon, do not neglect the unglamorous supporting cast. A top-tier GPU starved of CPU threads, system RAM, or storage bandwidth to load its model weights is nothing more than a thirty-thousand-dollar space heater with driver issues.

In practice, the thin line between a responsive Tuesday morning API and an all-hands incident call usually looks like twenty lines of YAML that nobody got around to writing:

apiVersion: v1
kind: Pod
metadata:
  name: customer-chatbot-vllm
  namespace: ai-production
spec:
  priorityClassName: production-inference-critical
  nodeSelector:
    workload-class: gpu-inference
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "inference-only"
      effect: "NoSchedule" # Keeps experimental training jobs off the lawn
  containers:
    - name: vllm-engine
      image: vllm/vllm-openai:latest
      resources:
        limits:
          nvidia.com/gpu: "1" # A hard ceiling, not a polite suggestion
          memory: "48Gi"
        requests:
          cpu: "8" # Prevents our expensive GPU from starving for data
          memory: "48Gi"
          nvidia.com/gpu: "1"

Stop autoscaling on CPU and hoping for the best

Once the workloads are separated, the next self-inflicted wound is usually autoscaling. Traditional Kubernetes autoscaling revolves around CPU utilization. For a stateless web server, watching the CPU is sensible. For an LLM inference server, scaling on CPU utilization is like trying to guess how many people are waiting outside a restaurant by measuring the chef’s shoe size.

During inference, the heavy lifting happens inside the GPU’s tensor cores and high-bandwidth memory, while the host CPU mostly orchestrates tokenization and network plumbing. Your chatbot pod can display a serene 15% CPU utilization while forty angry users sit backed up in the inference queue waiting for a slot in the KV cache.

If you want autoscaling to reflect reality, you have to scale on the backlog of actual work. Modern serving engines like vLLM expose metrics such as “vllm:num_requests_waiting” and token cache usage directly to Prometheus. Pairing those signals with KEDA (Kubernetes Event-driven Autoscaling) lets your cluster react to the line forming outside the door rather than the temperature of the hallway.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-inference-scaler
  namespace: ai-production
spec:
  scaleTargetRef:
    name: customer-chatbot-vllm
  minReplicaCount: 1 # Set to 0 only if an external proxy is watching the door
  maxReplicaCount: 6 # Also known as the CFO's blood pressure limit
  cooldownPeriod: 300 # Wait 5 minutes before killing a warm, expensive GPU
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-operated.monitoring.svc:9090
        metricName: vllm_num_requests_waiting
        query: sum(vllm:num_requests_waiting{service="customer-chatbot"})
        threshold: "5" # Wake up another replica when 5 prompts are stuck in the lobby

Even with KEDA, there are two classic traps waiting for the optimistic engineer. First, when HPA asks for another pod and the cluster autoscaler asks your cloud provider for another GPU node, that node only materializes if the cloud provider actually has spare GPUs in that availability zone. Kubernetes cannot manufacture silicon out of enthusiasm.

Second, there is the existential comedy of scaling to zero. Everyone loves the idea of scaling idle GPU deployments down to zero pods overnight to save money. Right up until they realize that if the autoscaler relies on a metric exported by the inference pod itself, scaling to zero means you have just euthanized the only witness capable of reporting that customers have arrived. The pod is gone, the Prometheus metric vanishes with it, and Kubernetes sits in blissful silence while early-morning users stare at a spinning wheel. Unless an external gateway or message queue stays awake outside the deployment to poke KEDA with a stick, your service has achieved permanent financial efficiency by ceasing to exist.

The cheapest GPU is the one you do not have to panic-start

Autoscaling is usually sold in conference talks as a performance feature, but in an AI cluster, it is primarily a defense mechanism against bankruptcy. Setting “minReplicaCount” and “maxReplicaCount” gives you guardrails, provided you understand the physics of what happens between those two numbers.

Spinning up a fresh GPU pod is not like launching a twenty-megabyte Go microservice. A new node can take several minutes to provision and attach drivers, followed by the geological epoch required to pull a container image and haul seventy gigabytes of model weights across the network into VRAM. If your scale-down policy is too twitchy, your cluster enters an expensive cardiovascular workout: traffic dips for forty seconds, Kubernetes terminates a GPU replica, three users show up, Kubernetes panics and provisions a fresh GPU node, the model spends four minutes loading, and by the time it is ready, the users have left.

Give your inference workloads generous cooldown periods and keep warm replicas alive for anything facing a real human. Save aggressive scale-to-zero policies for staging environments, internal batch summaries, and those pet projects that executives insist are mission-critical but only visit once a quarter.

Batching beats a procurement meeting

Let us return to our Tuesday morning crime scene. Our chatbot GPU was only at 24% utilization, yet latency was terrible. Before signing a purchase order for more hardware, it is worth asking why three-quarters of that GPU was essentially daydreaming.

Often, the culprit is naive request handling. Processing LLM prompts one by one in strict isolation is like running a city bus system where the driver refuses to let a second passenger board until the first passenger has been driven all the way across town and dropped off at their front porch. The bus is technically occupied the entire time, so Kubernetes marks it as allocated, even though fifty-eight seats are empty.

Inference engines solve this through batching. NVIDIA Triton uses dynamic batching to group incoming requests over a tiny time window, while LLM engines like vLLM rely on continuous batching, which slips new prompts into the GPU mid-flight as older prompts finish generating tokens. Done right, continuous batching can multiply throughput on your existing hardware without spending an extra dime.

Naturally, there is a catch. Batching is a game of chicken with user patience. Pack too many long prompts together, and you exhaust the GPU’s KV cache memory or stretch out the time-per-output-token until the text trickles onto the user’s screen like cold molasses. A massive batch size that looks heroic on a throughput benchmark while obliterating your p95 latency target is not an optimization. It is just a service outage with very impressive statistics.

Priorities, evictions, and the myth of the untested checkpoint

Sooner or later, despite your best architectural diagrams, resource contention will happen. A researcher will submit a legitimate distributed training job requesting eight GPUs at the exact moment your cluster only has six to spare.

This is where governance tools like Kubernetes “PriorityClass” and batch queuing systems like Kueue earn their keep. Instead of letting eight training pods schedule six of their siblings and deadlock the cluster while waiting for the missing two (the distributed computing equivalent of chartering three-quarters of an airplane), Kueue holds the entire job in a polite waiting room until the full block of quota is available. Meanwhile, priority classes ensure that if production inference suddenly needs to scale up, Kubernetes knows which experimental pods can be unceremoniously thrown out the airlock to make room.

Of course, evicting training jobs (or running them on cheap cloud Spot instances that can vanish with a two-minute warning) only saves money if those jobs are built to survive sudden death. A nine-hour training run that loses all its progress at hour eight because nobody configured periodic checkpoints is not a victory for FinOps. It is just an discounted way to heat the atmosphere. And remember to actually test restoring from those checkpoints; a recovery script nobody has ever run is not a disaster strategy, it is a bedtime story.

GPU sharing is great until the neighbors move in

When management discovers that several teams are requesting entire GPUs only to use 15% of their compute, someone inevitably suggests GPU sharing. On paper, it sounds like urban density planning at its finest. In practice, it depends entirely on whether you build walls between the apartments.

NVIDIA time-slicing is the technological equivalent of an airline selling four tickets for the exact same middle seat and hoping the passengers agree to take turns sitting down every few milliseconds. It works surprisingly well for lightweight notebooks and polite internal tools. However, time-slicing provides zero memory isolation. All those pods share the same pool of physical VRAM. The moment one developer’s script decides to load an oversized model or leaks memory, the resulting out-of-memory panic does not just crash the offending pod. It ejects the entire row of passengers without a parachute.

If you need predictable isolation on shared hardware, NVIDIA Multi-Instance GPU (MIG) is the adult alternative. Supported on higher-end cards like the A100 and H100, MIG physically partitions the GPU into independent hardware slices, each with its own dedicated compute cores and walled-off memory. Your workloads get smaller GPUs, but at least their neighbors cannot burn the house down.

How to keep the Finance Department out of your Slack channels

You do not need to build a miniature hyperscaler to stop your cluster from haemorrhaging cash, nor do you need a hundred-page governance manifesto. You just need a handful of enforceable habits before Monday morning traffic arrives:

Segregate the creatures: Never let long-running training jobs and latency-sensitive inference pods compete for the same unguarded node pool.
Retire CPU-based scaling for AI: Wire KEDA up to actual inference queue depth or token wait times, and make sure an external proxy is watching the door if you ever scale to zero.
Tune the engine before buying a second car: Benchmark continuous batching, concurrency limits, and KV cache utilization before assuming you are out of raw compute.
Make batch jobs wait their turn: Use Kueue and “PriorityClass” so distributed jobs launch all together or not at all, and ensure training code checkpoints often enough to survive preemption.
Partition with hardware, not hope: Use time-slicing for disposable dev sandboxes, but demand MIG whenever shared workloads actually matter.
Attach a name tag to every watt: Combine NVIDIA DCGM telemetry, inference metrics, and namespace cost allocation so you can measure cost per thousand tokens rather than staring blindly at a monthly lump sum. A namespace without quotas is a blank check, and a model without cost attribution is just an accounting mystery that happens to use CUDA.

The expensive part was never just the silicon

Let us check back in on our struggling chatbot from Tuesday at 9:17 AM.

Once you put proper observability on the cluster, the mystery usually evaporates in about four minutes. You discover that the chatbot did not run out of raw GPU horsepower at all. Instead, an unthrottled training experiment wandered onto the shared node pool, saturated the node’s network interface pulling a massive dataset, and left a single inference replica choking on an unbatched queue of seventeen concurrent users while HPA stared calmly at a 20% CPU graph and did nothing.

Buying four more GPUs would have temporarily masked the symptoms, right up until the next unscheduled experiment expanded to consume those too.

Kubernetes is phenomenally good at packing containers onto servers. It is terrible at reading your company’s balance sheet or guessing which workloads deserve thirty-thousand-dollar accelerators. Before you approve the next hardware requisition, force your platform to answer three simple questions: what is waiting in line, who owns the pod blocking the aisle, and how much does each answer actually cost us?

If nobody in the room can answer all three, your cluster does not need another GPU. It needs a chaperone.

AWS Lambda Web Functions let your server be a server

You write a perfectly capable Express application. It hums along beautifully on your laptop, accepting requests and handing back responses like a well-trained retriever fetching a tennis ball. Then, you decide to deploy it to AWS Lambda.

Historically, this is the exact moment your straightforward web server is forced into a bureaucratic nightmare. Suddenly, it has to work through an intermediary, a tiny administrative layer whose sole purpose in life is to mangle perfectly good HTTP requests into bizarre JSON events, hand them to your code, and then frantically repackage the output back into HTTP. It is exhausting just thinking about it.

Enter Lambda Web Functions. Instead of relying on specific adapters to translate events, AWS now simply gives your application a host and a port via the AWS_LAMBDA_HTTP_ENDPOINT environment variable. Your server gets to be a server again, listening for HTTP traffic directly. It sounds entirely logical, which, in the world of cloud architecture, makes it mildly suspicious.

Not Exactly a First Rodeo

To be fair, running websites on Lambda is not a brand new concept. The AWS Lambda Web Adapter has been letting people run standard web frameworks for a while, and response streaming has been knocking around since 2023. If anyone tries to sell you this update as the invention of serverless web hosting, feel free to raise an eyebrow.

The actual novelty lies in a specific, highly potent combination. AWS has taken a native HTTP model, added concurrency per environment, and sprinkled in a billing model based on active CPU time. It is a concoction explicitly designed for web applications. The magic isn’t that Lambda can serve a web page. The real magic is how it manages the resources while doing so.

The Perils of a Crowded Waiting Room

Let us talk about what it means for one warm environment to handle multiple simultaneous requests.

If your application spends most of its life waiting on a sluggish database or a third-party API, letting one environment juggle multiple conversations makes tremendous sense. It is highly efficient. However, sharing an environment means sharing problems. It is the architectural equivalent of putting multiple strangers in a single taxi.

If one of those requests suddenly decides to perform an intensive cryptographic calculation, it hogs the CPU and ruins the ride for everyone else. Connection pools can be exhausted in the blink of an eye if limits aren’t strictly enforced. And global mutable state, that old programming villain, returns with a vengeance. AWS gives you the tools to dynamically adjust and limit this concurrency to protect your backend dependencies, but the responsibility of policing the environment is entirely yours.

The CPU Clocks Out but Memory Still Pays Rent

This brings us to the financial heart of the matter, and arguably the most fascinating shift in the paradigm.

Consider a hypothetical web request that takes a full second to complete. For 950 milliseconds of that second, your code is doing absolutely nothing but staring out the window, waiting for an external API to return a payload. Previously, you paid for the whole second.

With Web Functions, the billing model tracks requests, data transfer, memory, and active CPU time. When your application is waiting for I/O, the CPU essentially clocks out. AWS stops charging you for processor time while your code twiddles its thumbs.

However, you should not summarize this as “you only pay when your code works.” Your code might be taking a break, but your memory footprint is not. Memory is billed like a hotel room. You pay for it as long as your luggage is inside, even if you are out for the day. If your application relies on heavy I/O waits, you will see a benefit. If it is purely CPU-bound, your math will look very different. The humor here is subtle but real. You can stop the clock on the processor, but the memory meter keeps running as long as the environment exists.

Code Travels Light but Data Needs a Moving Van

The allure of a global endpoint and multi-region deployment is undeniable. AWS will happily replicate your code and configuration across regions, using latency-based routing to send users to the closest available environment. It sounds like a geographical utopia until you read the fine print, which casually mentions that this system does not perform health-based failover.

Then comes the physics problem. Your user in Tokyo is routed to a lightning-fast Web Function in the Tokyo region. Excellent. But that function needs to query a relational database that resides in Ohio. Suddenly, your code’s incredible deployment speed is entirely negated by the stubborn inflexibility of the speed of light.

Distributing code across the globe simplifies exactly one half of the equation. Your data architecture still requires hard decisions about residency, regional dependencies, and failover strategies. Code travels light, but state is incredibly heavy.

Fewer Adapters and the Same Old Architectural Nightmares

The reality of deploying this setup hits you the moment you push a new revision.

Web Functions rely on immutable revisions. You can publish a new version, send a fraction of your traffic to it, watch the error rates and tail latencies closely, and decide whether to promote it or roll it back. It is a fantastic workflow, provided you remember one delightful quirk. When you create a new revision, omitted configuration variables do not magically carry over from the previous one. They revert to defaults. If you aren’t managing your configuration declaratively, you will inevitably deploy a heavily optimized piece of code with the memory allocation of a pocket calculator.

Furthermore, if you want a custom domain or web application firewall protection, the current documentation gently points you toward putting CloudFront in front of your Web Functions. The more things change, the more layers you add.

If you are considering adoption, start small. Build an endpoint with controlled I/O waits and measure the difference between total latency and consumed CPU. Try a CPU-bound calculation to see what happens when the processor becomes the bottleneck.

At launch, this feature is restricted to Node.js 24 running on Arm64, available in a handful of regions like Virginia, Ohio, Oregon, Tokyo, and Ireland. These constraints matter. They dictate whether your application can even participate in this new model today.

Ultimately, Web Functions allow your web server to act like a web server again. The awkward administrative translators are gone. But while the infrastructure gets out of your way to deliver HTTP, someone still has to sit down and worry about connection pools, database gravity, and deployment pipelines. Your server finally gets to be a server, which means you still have to be an architect.

October 13, 2026 by Fernando SRE Cloud stuff DevOps stuff

Jev and the architectural relief of an AI that refuses to write

There is a profoundly silly habit currently sweeping the software engineering world. We have a tiny, binary decision to make, so we summon a massive neural network. Is this support ticket urgent? Should the digital agent retry a failed search? Which tool should handle this query? To answer these deeply pedestrian questions, we take a model trained on the entire sum of human knowledge, capable of discussing quantum mechanics and composing a surprisingly competent resignation letter, and we force it to choose between the billing and technical support departments.

It works, in the same way that hiring a Michelin star chef to sort nuts and bolts in the back room of a hardware store works. It gets the job done, but it is a tragic and baffling waste of potential.

At a few requests per minute, nobody notices. At one hundred thousand requests, the architecture begins sending little postcards from reality. Latency rears its ugly head. Token bills pile up like parking tickets. Down in the server racks, the load balancers begin to sweat and hyperventilate because the model is spending vital milliseconds trying to decide which synonym for “frustrated” feels most empathetic, only to stuff that emotional labor into a sterile, soulless JSON output. Then somebody has to create an entirely new dashboard just to monitor the system that was supposed to make everything simpler.

Jev, a model released by TypeSafe AI in September 2026, starts from a refreshingly mundane assumption. Maybe the machine does not need to talk. Maybe it just needs to point.

Words are an expensive luxury for a server

Large language models are fundamentally generative machines. They receive tokens and produce more tokens, one after another, sequentially, until they have constructed an answer. That flexibility is exactly why they are so useful. It is also wildly extravagant when all you actually need is a “true” or “false” boolean.

Picture an exceptionally talented Victorian scholar acting as a modern management consultant. You ask him a simple question about whether a minor software incident should be escalated. The scholar clears his desk, meticulously examines the evidence, writes a beautifully structured three-page memorandum on the nature of urgency, and eventually concludes with a polite “Yes.” Software did not want the memorandum. Software just wanted the yes.

LLMs increasingly support structured output, constrained decoding, and JSON schemas. These tools make them much easier to wedge into modern applications. But underneath the hood, the architecture is still stubbornly built around generating sequences of text. For tasks involving writing, complex reasoning, generating code, or patient explanation, a sequence of text is exactly what we want. For millions of repetitive, strictly bounded decisions, it is a catastrophic overkill.

TypeSafe calls Jev a System One model, borrowing the psychological terminology made famous by Daniel Kahneman. Instead of generating sweeping prose, Jev is designed specifically for fast, blunt decisions that can be swallowed directly by software without chewing. The architectural distinction is far more important than the product itself.

The joyless clerk of the artificial intelligence world

Jev’s interface is almost aggressively uninterested in conversation. You provide some state representing what your program currently knows, and then you ask one or more strictly typed questions about it. The answers come back as typed values attached to probabilities.

There are currently three main primitives. Choice selects from a predefined set of options. Score evaluates something along a defined mathematical scale. Noul handles yes-or-no judgments and returns a probability. Noul is just TypeSafe’s quirky terminology for a Boolean, but we can forgive them for trying to brand it. The API allows several of these dry questions to be submitted in the same request, with the answers neatly mapped back to their question names.

This creates a rather different programming model. An LLM might receive an angry customer message and proudly produce a sentence explaining that the customer appears frustrated, the issue seems technical, and therefore it recommends routing the request to the technical support team with elevated priority. This is wonderfully useful if you are a human being reading a screen.

Software, however, prefers a world that looks like this:

department = technical

frustration = high

urgent = true

Plus, software wants probabilities telling it exactly how much blind faith to place in those decisions. Jev is deliberately built for this second, colder world.

TypeSafe describes the underlying training approach as Reinforcement Learning for Calibrated Decisions. The objective is to produce mathematical probabilities that reflect genuine uncertainty, rather than producing the kind of confident prose that humans find pleasing to read. Questions in the same request are evaluated in parallel rather than being generated one agonizing token after another.

The result is less “conversational artificial intelligence” and more “intelligent conditional logic.” It is a fuzzy if statement. This sounds considerably less exciting than artificial general intelligence, but it turns out to be immensely more useful inside a production system trying to keep the lights on.

Pulling your hand off the hot stove

Kahneman’s System One and System Two distinction provides a brilliant mental model for what is going wrong in our server farms. System Two handles deliberate, exhausting thought. It is the individual who sits rubbing their chin in front of a chessboard, evaluating seventeen possible moves and their downstream consequences.

System One is the primal instinct that makes you violently yank your hand away from a hot stove because you smell burning hair.

Modern reasoning models are spectacular System Two machines. Give them a complicated architectural diagram, a highly ambiguous security incident, or a tricky programming task, and their ability to reason through it can be extraordinary. But production software contains an ocean of System One questions.

Is this credit card transaction suspicious? Which queue gets this mundane ticket? Does this generated response contradict the source material? Should this digital agent search the web? Is this request safe enough to process automatically without calling a lawyer?

We have spent the last few years applying increasingly powerful System Two chess players to a surprising number of System One hot stoves. Jev’s entire existence is an argument that we have been using a massive reasoning hammer on tiny, decision-shaped nails. It is not trying to replace an LLM. It is trying to stop us from calling one when we never needed a paragraph in the first place.

Benchmark trophies and the inevitable vendor asterisk

TypeSafe advertises Jev as reaching roughly 70 to 500 milliseconds end to end. They report gains ranging from tens to hundreds of times in speed and cost on workloads designed around structured decisions. Their headline benchmark currently boasts that Jev is 193.6 times faster and 444.6 times cheaper in their workflow evaluations.

Those are undeniably impressive numbers. They are also vendor numbers, which means they should be treated with the same suspicion you apply to a real estate agent describing a house as “cozy.”

TypeSafe rightfully acknowledges several caveats. Their workflows were created internally, the reference answers come from frontier models rather than objective ground truth, and some comparisons use configurations that are particularly favorable to Jev’s execution model.

Fortunately, more interesting evidence is beginning to appear out in the wild. MotherDuck tested Jev on one hundred thousand articles from a news classification dataset. Jev chewed through them in around forty seconds for fifty cents and achieved an 89 percent accuracy rate.

GPT-4o mini took almost twenty minutes, cost nearly two dollars, and hit 80 percent accuracy. GPT-5.6 Terra achieved 88 percent accuracy but spent nearly thirty-two minutes doing it and cost a staggering thirty-seven dollars.

Now the comparison becomes genuinely useful. Not because Jev is magically four hundred times better than an LLM. It isn’t. The useful conclusion here is that bounded classification may not require generation at all. That is an architectural observation, not a shiny benchmark trophy. And as usual, the only benchmark that will eventually matter is the one running on your own servers.

Putting the rubber stamp before the eccentric artist

The absolute best use of Jev is not replacing an LLM. It is standing directly in front of one like a bouncer at a nightclub.

Consider an AI agent trying to decide which tool to invoke. Perhaps its options are answering directly, searching the web, querying a database, running code, or asking the user for help. There is absolutely no reason the routing decision itself needs a beautifully written paragraph of justification.

Jev can make the routing decision and hand back its confidence score. If the confidence is sufficiently high, the workflow proceeds immediately. If the confidence falls below a set threshold, the system escalates the problem to a more capable, expensive reasoning model. If the consequences are particularly dire, it escalates to an actual human being.

This gives us a beautifully practical architecture. Cheap decisions happen first, and expensive reasoning is reserved for when it is strictly necessary.

The same pattern works flawlessly for support triage, content moderation, incident classification, lead scoring, and agent evaluation. It also creates a fascinating verification layer. Let the expensive, eccentric genius LLM generate an answer. Then, let Jev act as the joyless clerk with a rubber stamp, checking whether that answer actually addresses the question, matches company policy, or is supported by the supplied context. The expensive model creates. The inexpensive model checks. Humans only ever see the weird, uncomfortable edge cases.

That last part matters because the probability score may ultimately be more useful than the decision itself. Automation rarely fails because software cannot choose between option A and option B. Automation fails because nobody knows when the machine should be trusted to make that choice without adult supervision.

Please do not let conference demos dictate your infrastructure

There is one extremely obvious trap here. If your entire architecture depends on confidence thresholds, those mathematical probabilities need to actually mean something.

An independent evaluation published recently tested Jev across thirty-seven datasets and more than three hundred thousand requests. The researchers found Jev’s choice probabilities to be generally well calibrated and highly useful for selective prediction. Binary probabilities, however, were more troublesome when treated with a rigid 0.5 threshold. Performance only improved significantly when those thresholds were tuned using task-specific data.

That is probably the single most useful lesson for a software architect. Do not write a rule that says “if confidence is greater than 0.8” just because somebody used 0.8 on a slide during a flashy conference demo. You have to measure it.

Maybe 0.74 is perfectly safe for routing a low-priority IT ticket. Maybe 0.97 is absolutely necessary before an automated security system blocks a user. Maybe no threshold on earth is acceptable for autonomously deleting customer data, which would be reassuring evidence that human common sense remains a commercially viable trait. Confidence only becomes useful infrastructure when you calibrate it against the consequences of being wrong.

The jobs a joyless clerk should never get

The limitations of this new architecture are unusually easy to explain. Jev cannot write.

If the output needs to be read by a human, explained, rewritten, summarized, or turned into functional code, you still want a traditional language model. Furthermore, the possible answers need to be known completely in advance. The Choice primitive supports a finite set of alternatives, currently with cardinality limits that make it totally unsuitable for arbitrary open-ended generation. (TypeSafe documents Choice cardinality up to 255 options, which is plenty for routing but useless for brainstorming).

It is also currently designed strictly around structured and textual state, rather than being a magical, all-seeing multimodal model.

And sometimes, an explanation matters far more than the blunt decision. A security system shouting “BLOCK 0.96” might be operationally useful in the heat of the moment. But the next morning, an auditor is still going to ask why the block happened. At which point, System Two gets another reluctant invitation to the meeting to explain the mess.

Building an oracle just to empty the digital trash

It is very tempting to treat Jev as just another minor model launch and ask whether it beats GPT-this or Claude-that. Doing so completely misses the impending architectural shift.

For several years, the AI stack has been dominated by one wonderfully convenient but bloated primitive. Send text to an LLM. Receive text from an LLM. Repeat the process until the venture capital funding improves.

Decision models suggest that the software stack is finally beginning to separate its responsibilities. Generative models will be kept around to reason, explain, code, and communicate. Decision models will step in to classify, route, score, verify, and trigger. Normal, boring software will handle everything deterministic that happens in between them.

That is a vastly healthier architecture, because intelligence stops being a single, enormous, expensive API call and becomes just another component that can be composed according to cost, latency, and risk. Jev might dominate this new category, or a competitor might crush them in six months. The category itself is what matters.

The name Jev is a subtle nod to the 19th-century economist William Stanley Jevons and the paradox forever associated with him. The Jevons paradox states that making a resource more efficient does not necessarily reduce our consumption of it. In fact, making it cheaper and more efficient usually makes us use vastly, absurdly more of it.

That might turn out to be the most important part of this whole story. When an intelligent decision costs a dollar, we reserve artificial intelligence for highly important decisions. When it costs fractions of a cent and arrives in milliseconds, we start cramming intelligence into places where nobody in their right mind would have ever considered paying an LLM to look.

We will suddenly find our software making millions of tiny, mundane judgments that nobody previously thought were worth making. Not because AI finally learned how to write better. But because we finally taught it how to shut up and point. And so, we arrive at the ultimate punchline of human engineering. We have successfully invented the most sophisticated reasoning engine in the history of the universe, and we are going to use it to decide whether an email offering a discount on sneakers belongs in the spam folder.

October 4, 2026 by Fernando SRE Cloud stuff Computer Science stuff

Making Kubernetes upgrades boring on purpose

The email always arrives in the same tone, the calm and faintly apologetic voice of a notary reading a will. Your Kubernetes version, it says, will soon reach the end of standard support. Nobody has died yet. But the cluster has started looking at you the way an elderly Labrador looks at the car when it suspects the destination is the vet.

Kubernetes ships three minor releases a year, and managed providers support each one for roughly fourteen months. That gives your cluster about the shelf life of a yogurt with ambitions. On EKS, ignoring the expiry date does not break anything, it just gets expensive. Clusters roll into extended support by default, and the control plane goes from $0.10 to $0.60 per hour, six times the price for the privilege of postponing a decision you will have to make anyway. Procrastination, it turns out, has an hourly rate.

Large companies handle this ritual with platform teams the size of a small village. You have three people, a backlog with its own gravitational field, and a production environment that nobody fully remembers configuring. Upgrading means touching the control plane, the nodes, the controllers, and the workloads, all while the patient is awake and serving traffic.

So let us be clear about the goal. This is not about heroism. In operations, heroism is a polite word for someone losing sleep. The goal is to make upgrades so boring, so predictable and so reversible that nobody mentions them at lunch the following week. Think of it as elective surgery, scheduled, rehearsed, and performed by people who have read the patient’s chart.

Kubernetes rarely kills anyone directly

When an upgrade goes wrong, Kubernetes itself is almost never the murderer. It is more like the host of a dinner party where something in the soup disagreed with the guests. The real culprits are the things you bolted onto it over the years.

The classic case is the ingress controller that flatlines the moment the upgrade finishes. It was quietly relying on networking.k8s.io/v1beta1, an API version Kubernetes deprecated years earlier and finally removed in 1.22. Nobody noticed because nothing broke, until everything did. PodSecurityPolicy pulled the same trick in 1.25, leaving like a houseguest who sneaks out before breakfast and takes your security model with them.

Then there is Helm, which keeps the rendered manifests of every release the way some people keep old love letters. If those manifests reference an API that no longer exists, your next helm upgrade fails, even when the new chart is perfectly modern. The helm-mapkubeapis plugin exists precisely to clean up this kind of sentimental clutter.

Add a service mesh with its own compatibility matrix (Istio publishes the Kubernetes versions each release supports, and it means it), stateful workloads that panic when a node vanishes, and version skew between the control plane and the nodes, and the pattern becomes obvious. Most upgrade incidents are dependency failures wearing a Kubernetes trench coat.

Taking the patient’s medical history

No surgeon operates on a stranger. Before you touch anything, write down what is really inside the cluster, which is rarely what the architecture diagram claims.

The list does not need to be elegant. It needs your current and target versions, the OS images on every node pool, and every add-on, CRD, and operator, including the one someone installed during an incident in 2022 and never mentioned again. Note the critical workloads and the humans who own them, because at 3 a.m. “the payments team” is not a phone number. Record the external dependencies too, from databases to DNS to the identity provider whose webhook will suddenly matter a great deal if it stops answering.

Keep this inventory in Git, next to the code. It becomes the baseline for the next upgrade and the one after that. The second upgrade should be cheaper than the first. If it is not, the inventory was a diary entry rather than a medical chart.

Hunting for the APIs nobody admits to using

Production should never be the first place you test a theory. Before the upgrade, scan for deprecated and removed APIs in your manifests, your Kustomize overlays, your Helm releases, and the resources your CI pipeline generates, which nobody has read since the day they were written. A few commands go a long way.

# Deprecated APIs in manifests and Helm releases
pluto detect-files -d ./manifests
pluto detect-helm -owide

# What is running in the cluster right now
kubent

# Which clients are still calling deprecated APIs
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

The last one is the most honest of the group. It asks the API server who is still calling deprecated endpoints, which catches things that live outside Git, like a forgotten CronJob or a vendor operator with outdated opinions.

The cloud providers will help too, each with its own temperament. EKS upgrade insights flags readiness problems before you press the button. GKE surfaces deprecation insights and tells you which versions your release channel offers. AKS is strict about the version skew it allows between the control plane and node pools, and its rules are worth reading before you plan the sequence rather than halfway through it.

A backup you have never restored is a bedtime story

Backups belong before the surgery, not after it. A backup you have never restored is a story you tell yourself so you can fall asleep, and like most bedtime stories, it contains no useful information about what happens when the lights go out.

Think in three layers. The first is your GitOps repositories and infrastructure as code, which describe what the cluster should look like. The second is the cluster’s own configuration (RBAC, network policies, operator settings, and the custom resources that never quite made it into Git). The third is the application data, persistent volumes included, which is the only layer your customers care about.

Tools like Velero cover the second and third layers well, but owning a tool is not the test. The test is restoring into a scratch cluster and timing it. Do it every quarter and before any upgrade that touches stateful workloads. “We have snapshots” is a feeling. “We can restore the orders database in 40 minutes” is a plan.

The operation, in five uneventful acts

Upgrading everything at once is a reliable way to end up on a 3 a.m. call with a dozen people, a shared screen, and no idea which change started the fire. Doing it in phases keeps the blast radius small and the call list short.

Pre-op paperwork

Check your quotas and your IP address capacity. Surge upgrades and new node pools need room, and a subnet with no free addresses will stall an upgrade more efficiently than any bug. Freeze nonessential deployments. Then write down, before you start, what success looks like and which specific signal will make you stop. Abort criteria decided in advance are engineering. Abort criteria decided during the incident are a negotiation.

Practicing on the cadaver

Medical students learn on cadavers because cadavers do not file complaints. Your staging cluster plays the same role. Upgrade it first, run the integration tests, and let it soak for a defined period. Every manual tweak you needed is a bug in your process, so write it down and automate it before production. If staging is so different from production that a clean run proves nothing, congratulations, you have found next quarter’s real project.

Brain surgery, performed by someone else

The control plane upgrade is the one part of the operation where the cloud provider holds the scalpel, and you hold the patient’s hand. You move one minor version at a time, and on the major managed services you rarely get a say in the matter, since they will not let you skip ahead. It is one of the few occasions when a vendor protects you from your own optimism.

Leave the node pools on the old version while the control plane changes. The upstream skew policy allows kubelets to run up to three minor versions behind the API server, so a patient can live for a while with a new brain and old limbs. The opposite arrangement, nodes newer than the control plane, is not supported at all.

The organ transplant

Node upgrades are where availability really gets tested, because you are dismantling the machines your code is running on. For simple stateless workloads, a surge upgrade is fine: add a new node, drain an old one, repeat. For anything critical, create a fresh node pool on the target version and move workloads across deliberately. GKE offers blue-green node pool upgrades out of the box, and AKS recommends going control plane first, then system node pools, then user node pools.

Do the transplant in stages. Move your least important background jobs to the new pool first and watch them like a nervous parent at a school play. Are error rates stable? Is scheduling behaving? Are the pods starting at all? Only then move critical workloads, in batches.

None of this works without PodDisruptionBudgets, readiness probes, and graceful shutdown. PDBs, however, have a dark side. A budget that allows zero disruptions does not protect your service, it takes the upgrade hostage, and each provider handles hostages differently. GKE respects PDBs for up to an hour during a surge upgrade and then evicts the remaining pods by force. EKS managed node groups wait 15 minutes and then fail the upgrade with a PodEvictionFailure error unless you pass the force flag. Neither ending is what you had in mind when you wrote that PDB.

Physical therapy for the add-ons

Once the core is done, move on to CoreDNS, the CNI, the ingress controller, cert-manager and the service mesh, one at a time, each checked against its own compatibility notes. Ship application changes separately from infrastructure changes. If two things change at once and something breaks, you will spend the afternoon running a paternity test instead of fixing the problem.

The seven-day return policy

For years, rolling back a managed control plane was mostly wishful thinking. Upstream Kubernetes does not support it natively, so teams paid for reversibility with duplicate clusters or elaborate snapshots of cluster state.

That changed this summer. In July 2026, AWS launched EKS Version Rollback, which lets you return the control plane to the previous minor version within seven days of an upgrade. It goes back one minor version at a time, and before starting it runs rollback readiness insights that check API compatibility, version skew, add-ons, and cluster health. GKE has offered control plane minor version rollback since 1.33, while AKS limits rollback to node pools.

Two caveats keep this from being a get-out-of-jail-free card. The first is the clock. If your plan is to watch production for ten days before declaring victory, the safety net disappears on day seven, so fit your observation window inside it. The second is scope. Only the control plane goes back. Your database migrations, the CRDs an operator already converted, and the data written in the meantime all stay exactly where they are. A rollback undoes the surgery, not the meals the patient has eaten since.

Let the robots drain, let the humans sign

Small teams survive by automating the boring parts, such as version compatibility checks, health polling, node draining and post-upgrade smoke tests. Machines are excellent at doing the same tedious thing the same way every time, which is more than anyone can say for an engineer on their fourth coffee.

What should not be automated is judgment. A human approves the production control plane upgrade. A human approves deleting the old node pools, which is the exact moment your fallback stops existing. A human decides any rollback that involves stateful data. Automate the labor, but keep a person on the consent form.

A runbook you can read while panicking

Upgrade documentation should be short enough to read with your heart rate at 140. If it reads like a novel, nobody will open it during the incident. Here is a one-page template you are welcome to steal.

The best upgrade is the one nobody remembers

Upgrading Kubernetes does not have to be traumatic. Move one minor version at a time, take your dependencies more seriously than Kubernetes itself, rehearse on staging, and treat any backup you have not restored with the suspicion it deserves.

Do that a few times and something strange happens. The notary’s email arrives, someone opens a ticket, the runbook gets filled in, and a week later nobody can quite remember when the upgrade happened. In operations, that kind of amnesia is the highest compliment there is.

October 3, 2026 by Fernando SRE Cloud stuff DevOps stuff Kubernetes SRE stuff