SiteReliabilityEngineering

Your Kubernetes rollback is lying

The PagerDuty alert screams. The new release, born just minutes ago with such promising release notes, is coughing up blood in production. The team’s Slack channel is a frantic mess of flashing red emojis. Someone, summoning the voice of a panicked adult, yells the magic word: “ROLLBACK!”

And so, Helm, our trusty tow-truck operator, rides in with a smile, waving its friendly green check marks. The dashboards, those silent accomplices, beam with the serene glow of healthy metrics. Kubernetes probes, ever so polite, confirm that the resurrected pods are, in fact, “breathing.”

Then, production face-plants. Hard.

The feeling is like putting a cartoon-themed bandage on a burst water pipe and then wondering, with genuine surprise, why the living room has become a swimming pool. This article is the autopsy of those “perfect” rollbacks. We’re going to uncover why your monitoring is a pathological liar, how network traffic becomes a double agent, and what to do so that the next time Helm gives you a thumbs-up, you can actually believe it.

A state that refuses to time-travel

The first, most brutal lie a rollback tells you is that it can turn back time. A helm rollback is like the “rewind” button on an old VCR remote; it diligently rewinds the tape (your YAML manifests), but it has absolutely no power to make the actors on screen younger.

Your application’s state is one of those stubborn actors.

While your ConfigMaps and Secrets might dutifully revert to their previous versions, your data lives firmly in the present. If your new release included a database migration that added a column, rolling back the application code doesn’t magically make that column disappear. Now your old code is staring at a database schema from the future, utterly confused, like a medieval blacksmith being handed an iPad.

The same goes for PersistentVolumeClaims, external caches like Redis, or messages sitting in a Kafka queue. The rollback command whispers sweet nothings about returning to a “known good state,” but it’s only talking about itself. The rest of your universe has moved on, and it refuses to travel back with you.

The overly polite doorman

The second culprit in our investigation is the Kubernetes probe. Think of the readinessProbe as an overly polite doorman at a fancy party. Its job is to check if a guest (your pod) is ready to enter. But its definition of “ready” can be dangerously optimistic.

Many applications, especially those running on the JVM, have what we’ll call a “warming up” period. When a pod starts, the process is running, the HTTP port is open, and it will happily respond to a simple /health check. The doorman sees a guest in a tuxedo and says, “Looks good to me!” and opens the door.

What the doorman doesn’t see is that this guest is still stretching, yawning, and trying to remember where they are. The application’s caches are cold, its connection pools are empty, and its JIT compiler is just beginning to think about maybe, possibly, optimizing some code. The first few dozen requests it receives will be painfully slow or, worse, time out completely.

So while your readinessProbe is giving you a green light, your first wave of users is getting a face full of errors. For these sleepy applications, you need a more rigorous bouncer.

A startupProbe is that bouncer. It gives the app a generous amount of time to get its act together before even letting the doorman (readiness and liveness probes) start their shift.

# This probe gives our sleepy JVM app up to 5 minutes to wake up.
livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 20
readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
startupProbe:
  httpGet:
    path: /health/ready
    port: 8080
  # Kubelet will try 30 times with a 10-second interval (300 seconds).
  # If the app isn't ready by then, the pod will be restarted.
  failureThreshold: 30
  periodSeconds: 10

Without it, your rollback creates a fleet of pods that are technically alive but functionally useless, and Kubernetes happily sends them a flood of unsuspecting users.

Traffic, the double agent

And that brings us to our final suspect: the network traffic itself. In a modern setup using a service mesh like Istio or Linkerd, traffic routing is a sophisticated dance. But even the most graceful dancer can trip.

When you roll back, a new ReplicaSet is created with the old pod specification. The service mesh sees these new pods starting up, asks the doorman (readinessProbe) if they’re good to go, gets an enthusiastic “yes!”, and immediately starts sending them a percentage of live production traffic.

This is where all our problems converge. Your service mesh, in its infinite efficiency, has just routed 50% of your user traffic to a platoon of sleepy, confused pods that are trying to talk to a database from the future.

Let’s look at the evidence. This VirtualService, which we now call “The 50/50 Disaster Splitter,” was routing traffic with criminal optimism.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: checkout-api-vs
  namespace: prod-eu-central
spec:
  hosts:
    - "checkout.api.internal"
  http:
    - route:
        - destination:
            host: checkout-api-svc
            subset: v1-stable
          weight: 50 # 50% to the (theoretically) working pods
        - destination:
            host: checkout-api-svc
            subset: v1-rollback
          weight: 50 # 50% to the pods we just dragged from the past

The service mesh isn’t malicious. It’s just an incredibly efficient tool that is very good at following bad instructions. It sees a green light and hits the accelerator.

A survival guide that won’t betray you

So, you’re in the middle of a fire, and the “break glass in case of emergency” button is a lie. What do you do? You need a playbook that acknowledges reality.

Step 0: Breathe and isolate the blast radius

Before you even think about rolling back, stop the bleeding. The fastest way to do that is often at the traffic level. Use your service mesh or ingress controller to immediately shift 100% of traffic back to the last known good version. Don’t wait for new pods to start. This is a surgical move that takes seconds and gives you breathing room.

Step 1: Declare an incident and gather the detectives

Get the right people on a call. Announce that this is not a “quick rollback” but an incident investigation. Your goal is to understand why the release failed, not just to hit the undo button.

Step 2: Perform the autopsy (while the system is stable)

With traffic safely routed away from the wreckage, you can now investigate. Check the logs of the failed pods. Look at the database. Is there a schema mismatch? A bad configuration? This is where you find the real killer.

Step 3: Plan the counter-offensive (which might not be a rollback)

Sometimes, the safest path forward is a roll forward. A small hotfix that corrects the issue might be faster and less risky than trying to force the old code to work with a new state. A rollback should be a deliberate, planned action, not a panic reflex. If you must roll back, do it with the knowledge you’ve gained from your investigation.

Step 4: The deliberate, cautious rollback

If you’ve determined a rollback is the only way, do it methodically.

Scale down the broken deployment:
kubectl scale deployment/checkout-api –replicas=0
Execute the Helm rollback:
helm rollback checkout-api 1 -n prod-eu-central
Watch the new pods like a hawk: Monitor their logs and key metrics as they come up. Don’t trust the green check marks.
Perform a Canary Release: Once the new pods look genuinely healthy, use your service mesh to send them 1% of the traffic. Then 10%. Then 50%. Then 100%. You are now in control, not the blind optimism of the automation.

The truth will set you free

A Kubernetes rollback isn’t a time machine. It’s a YAML editor with a fancy title. It doesn’t understand your data, it doesn’t appreciate your app’s need for a morning coffee, and it certainly doesn’t grasp the nuances of traffic routing under pressure.

Treating a rollback as a simple, safe undo button is the fastest way to turn a small incident into a full-blown outage. By understanding the lies it tells, you can build a process that trusts human investigation over deceptive green lights. So the next time a deployment goes sideways, don’t just reach for the rollback lever. Reach for your detective’s hat instead.

The ugly truth about SRE Dashboards

Every engineer loves a good dashboard. The vibrant graphs, the neat panels, the comforting glow of a wall of green lights. It’s the digital equivalent of a clean garage; it feels productive, organized, and ready for anything.

But let’s be honest: your dashboards are probably lying to you. They’re like a well-intentioned friend who tells you everything’s fine when you’ve got a smudge of chocolate on your nose and a bird nesting in your hair. They show you the surface, but hide the messy, inconvenient truth.

I learned this the hard way, at 2 a.m., as all the best lessons are learned. We were on-call when production latency went absolutely bonkers. I stared at four massive dashboards, each with a dozen panels of metrics swirling on my screen: CPU, memory, queue depth, disk I/O, HPA stats, all the usual suspects. I was a detective with a thousand clues but no insights, scrolling through what felt like a colorful, confusing kaleidoscope.

An hour of this high-octane confusion later, we discovered the culprit: a single, rogue DNS misconfiguration in a downstream service. The dashboards, those beautiful, useless liars, had all been glowing green.

This isn’t just bad luck. It’s a design flaw.

Designed for reports, not for war

Most dashboards are built for managers who need to glance at high-level metrics during a meeting, not for engineers trying to solve a full-blown crisis. We obsess over the shiny vanity metrics: request counts and 99th percentile latency, while the real demons, the retry storms and misbehaving clients, hide in the shadows.

Think of it like this: your dashboard is a doctor who only checks your height and weight. You might look great on paper, but your appendix could be about to explode. The surface looks fine, but the guts are in chaos.

The graveyard of abandoned dashboards

Have you ever wondered where old dashboards go to die? The answer is: nowhere. They simply get abandoned, like a pet you can no longer care for. Metrics get deprecated, panels start showing N/A, and alerts get muted permanently. They become relics of a bygone era, cluttering your screens with useless data and false promises. It’s the digital equivalent of that one junk drawer in your kitchen; it feels organized at a glance, but you know deep down it’s a monument to things you’ll never use again.

Too much signal, too much noise

Adding more panels doesn’t automatically give you better visibility. At scale, dashboards become a cacophony of white noise. You spend 30 minutes scanning, 5 minutes guessing, and 10 minutes restarting pods just to see if the blinking stops. That’s not observability; that’s panic dressed up as process.

Imagine trying to find your house key on a keychain with 500 different keys on it. You can see all of them, but you can’t find the one you need when you’re standing in the rain.

So, how do you fix it? You stop making art and start getting answers.

From Metrics to Methods

We stopped dumping metrics onto giant boards and created what we called “Runbooks with Graphs.” Instead of a hundred metrics per service, we had a handful per failure mode. It’s a fundamental shift in perspective.

Here’s an example of what that looked like:

failure_mode: API_response_slowdown
title: "API Latency Exceeding SLO"
hypothesis: "Is the database overloaded?"
metrics:
  - name: "database_connections_count"
    query: "sum(database_connections_total)"
  - name: "database_query_latency_p99"
    query: "histogram_quantile(0.99, rate(database_query_latency_seconds_bucket[5m]))"
runbook_link: "https://your-wiki.com/runbooks/api_latency_troubleshooting"

This simple shift grouped our metrics by the why, not just the what.

Slaying Alert Fatigue

We took a good, hard look at our alerts and deleted 40% of them. Then, we rebuilt them from the ground up, basing them on symptoms, not raw metrics. This meant getting rid of things like this:

# BEFORE: A useless alert
- alert: HighCPULoad
  expr: avg(cpu_usage_rate) > 0.8
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High CPU on instance {{ $labels.instance }}"

And replacing it with something like this:

# AFTER: A meaningful, symptom-based alert
- alert: CustomerFacingSLOViolation
  expr: rate(http_requests_total{status_code!~"2.."}) / rate(http_requests_total) > 0.1
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Too many failed API requests - SLO violated"
    description: "The percentage of failed requests is over 10%."

Suddenly, the team trusted the alerts again. When the pager went off, it actually meant something was wrong for the customers, not just a server having a bad day.

Blackhole checks and truth bombs

If dashboards can lie, you need tools that don’t. We added synthetic tests and end-to-end user simulations. These act like a secret shopper for your service, proving something is broken, whether your metrics look good or not.

Here’s a simple example of a synthetic check:

const axios = require('axios');
async function checkAPIMetrics() {
  try {
    const response = await axios.get('https://api.yourcompany.com/v1/health');
    if (response.status !== 200) {
      throw new Error(`Health check failed with status: ${response.status}`);
    }
    console.log('API is healthy.');
  } catch (error) {
    console.error('API health check failed:', error.message);
    // Send alert to PagerDuty or Slack
  }
}
checkAPIMetrics();

Your internal metrics may say “OK,” but a synthetic user never lies about the customer’s experience.

The hard truth

Dashboards don’t solve outages. People do. They’re useful, but only if they’re maintained, contextual, and grounded in real-world operations. If your dashboards don’t reflect how failures actually unfold, they’re not observability, they’re art. And in the middle of a P1 incident, you don’t need art. You need answers.

This is the part where I’m supposed to give you a tidy, inspirational conclusion. Something about how we can all be better, more vigilant SREs. But let’s be realistic. The truth is, the world is full of dashboards that are just digital wallpaper, beautiful to look at, utterly useless in a crisis. They’re a collective delusion that makes us feel like we have everything under control, when in reality, we’re just scrolling through colorful confusion, hoping something will catch our eye.

So, before you build another massive, 50-panel dashboard, stop and ask yourself: is this going to help me at 2 a.m., with my coffee pot empty and a panic-stricken developer on the other end of the line? Or is it just another pretty lie to add to the collection?

How many of your dashboards are truly battle-ready? And which ones are just decorative?

August 28, 2025 by Fernando SRE Cloud stuff DevOps stuff SRE stuff

Synthetic Monitoring with Amazon CloudWatch

Downtime is unacceptable. In today’s hyper-connected world, your users expect your website and applications to be available, always. There are no excuses. But maintaining that uptime is a constant challenge, a battle against the forces of digital entropy. Luckily, you don’t have to fight this battle alone. Amazon CloudWatch Synthetics provides a powerful arsenal of tools to proactively monitor your digital assets, giving you the edge to stay ahead of the game. Let’s explore how these canaries can be your secret weapon for achieving bulletproof uptime.

Why should you care?

Let’s face it: In today’s digital world, downtime is a cardinal sin. Your website or application is your storefront, your lifeline to your customers. Every second it’s unavailable is a lost opportunity, a frustrated user, and a potential blow to your reputation. Think about the last time you tried to access a website and it was down. Frustrating, right? Now imagine being on the other side, responsible for that frustration. It is a feeling of overwhelm.

But it’s not just about websites. APIs, the invisible threads connecting the digital world, are just as crucial. A broken API can bring an entire ecosystem grinding to a halt. And what about those pesky broken links or unexpected changes to your website’s appearance? They might seem small, but they can chip away at user trust and make your site look unprofessional.

Enter the canaries

This is where CloudWatch Synthetics steps in, your proactive problem-solving sidekick. It lets you create “canaries”, not the feathered kind, but automated scripts that mimic your users’ actions. These canaries are like those brave little birds miners used to take into coal mines. If the canary stopped singing, you knew there was a problem with the air. Similarly, if your digital canary trips an alarm, you know something’s up with your application, even before the users come complaining.

Recipes for success the blueprints

Now, you might be thinking, “Writing scripts? That sounds complicated!” But fear not, AWS provides us with what they call “blueprints”, think of them as ready-made recipes for your canaries. These templates cover the most common monitoring scenarios, so you don’t have to start from scratch. Let’s explore a few:

Heartbeat Monitoring. Imagine that you have a hypochondriac friend who calls you every hour to make sure you are still alive. The Heartbeat Monitor is something like that but for your website. It will check if your URL is alive and kicking.
API Canary. This is like a food taster for your APIs, making sure each endpoint is serving up fresh and accurate data, and testing basic read and write operations. A must-have for any API-driven application.
Broken Link Checker. Think of this as a digital detective, meticulously combing through your website for any broken links, those pesky 404 errors that lead users down a dead end.
Visual Monitoring. This canary is like a security guard, comparing snapshots of your website over time to a baseline image. Any unexpected changes raise the alarm. Useful for detecting visual regressions or unauthorized modifications.
Canary Recorder. This is pure magic. You can record your actions on a website, and it automatically generates a canary script based on that recording. It’s like having a digital parrot that mimics your every move.
GUI Workflow Builder. This blueprint is perfect for testing complex user interactions, like logging into a web form or completing a multi-step process. It ensures that your users can navigate your application without hitting any roadblocks.

The power of proactive monitoring

So, why are these canaries so important? It’s all about being proactive instead of reactive. Instead of waiting for users to report problems, you’re finding and fixing them before they even impact anyone.

Availability and Latency Monitoring. You can measure how fast your pages are loading, and how quickly your APIs are responding. Slow and steady doesn’t win the race in the digital world.
Early Problem Detection. Identify issues before they escalate into major outages. Catch those bugs before they bite.
CloudWatch Alarms Integration. Configure your canaries to trigger alarms in CloudWatch, so you can get notified immediately when things go wrong.
Customizable Scripts. You have the flexibility to write your own scripts in Node.js or Python, giving you full control over your monitoring.
Headless Browser Usage. The canaries use a headless Google Chrome browser, which means they can simulate real user interactions with your website without needing a visible browser window.
Configurable Run Schedules. Run your canaries once or on a recurring schedule, providing continuous monitoring.

A real-world example

Imagine you have an e-commerce website. You can use Route 53 for DNS, and a canary to constantly monitor your website’s URL. If the canary detects that your website is down, a CloudWatch Alarm is triggered. You can even have a Lambda function automatically redirect traffic to a backup server in another region, ensuring that your customers can still shop even if your primary server is having issues. This is the kind of automation that can save your bacon.

Beyond the basics

CloudWatch Synthetics isn’t just about monitoring; it’s about optimizing. By simulating user behavior, you can ensure that your application works as expected under various conditions. And because it’s integrated with other AWS services, you can automate incident response and minimize downtime.

So, should you use it?

If you’re serious about the uptime and performance of your applications, the answer is a resounding yes! CloudWatch Synthetics provides a robust, flexible, and proactive way to monitor your digital assets. It’s an essential tool for any AWS Architect or DevOps Engineer looking to build resilient and reliable systems.

Amazon CloudWatch Synthetics is more than just a monitoring tool; it’s a peace-of-mind provider. By letting these digital canaries do the hard work, you can focus on what you do best: building amazing applications. So, unleash the canaries, and keep your apps singing! And remember, don’t just react to problems, prevent them.

January 10, 2025 by Fernando SRE Cloud stuff DevOps stuff

Managing Incidents While Fostering Blameless Postmortems in DevOps

Let’s jump into a topic that is gaining importance in the world of DevOps and Site Reliability Engineering (SRE): incident management and blameless postmortems. Now, I know these terms might seem a bit intimidating at first, but don’t worry, we’re going to break them down in a way that’s easy to grasp. So, grab a cup of coffee (or your favorite beverage), and let’s explore these critical skills together.

1. Introduction. Why Is Incident Management Such a Big Deal?

Imagine you’re piloting a spaceship through uncharted territory. Suddenly, a red warning light starts flashing. What do you do? Panic? Start pressing random buttons? Of course not! You want a well-rehearsed plan, right? That’s essentially what incident management is all about in the tech world.

Unexpected issues might arise in today’s rapid digital environment, much like that red light on your spaceship’s dashboard. Users become irate when websites crash and services are unavailable. The methodical approach known as incident management enables teams to respond to these issues promptly and effectively, reducing downtime and expediting the restoration of service.

But what does this have to do with DevOps and SRE? Well, if DevOps and SRE professionals are the astronauts of the tech world, then incident management is their emergency survival training. And it’s becoming more and more essential as companies recognize how critical it is to keep their services running smoothly.

2. Incident Management. Keeping the Digital Spaceship Afloat

Sticking with our spaceship analogy, a small issue in space can quickly spiral out of control if not managed properly. Similarly, a minor glitch in a digital service can escalate into a major outage if the response isn’t swift and effective. That’s where incident management shines in DevOps and SRE.

Effective incident management is like having a well-practiced, automatic response when things go wrong. It’s the difference between panicking and pressing all the wrong buttons, or calmly addressing the issue while minimizing damage. Here’s how the process generally unfolds:

Incident Detection and Alerting: Think of this as your spaceship’s radar. It constantly scans for anomalies and sounds the alarm when something isn’t right.
Incident Response and Triage: Once the alert goes off, it’s time for action! This step is like diagnosing a patient in the ER – figuring out the severity of the situation and the best course of action.
Incident Resolution and Communication: Now it’s time to fix the problem. But equally important is keeping everyone informed – from your team to your customers, about what’s happening.
Post-Incident Analysis and Documentation: After things calm down, it’s time to analyze what happened, why it happened, and how to prevent it from happening again. This is where blameless postmortems come into play.

3. Blameless Postmortems. Learning from Mistakes Without the Blame Game

Now, let’s talk about blameless postmortems. The idea might sound strange at first, but “postmortem” usually refers to an examination after death, right? In this context, however, a postmortem is simply an analysis of what went wrong during an incident.

The key here is the word “blameless.” Instead of pointing fingers and assigning blame, the goal of a blameless postmortem is to learn from mistakes and figure out how to improve in the future. It’s like a sports team reviewing a lost game, instead of blaming the goalkeeper for missing a save, the entire team looks at how they can play better together next time.

So, why is this approach so effective?

Encourages open communication: When people don’t fear blame, they’re more willing to be honest about what happened.
Promotes continuous learning: By focusing on improvement rather than punishment, teams grow and become stronger over time.
Prevents repeat incidents: The deeper you understand what went wrong, the better you can prevent similar incidents in the future.
Builds trust and psychological safety: When team members know they won’t be scapegoated, they’re more willing to take risks and innovate.

4. How to Conduct a Blameless Postmortem.

So, how exactly do you conduct a blameless postmortem?

Gather all the facts: First, collect all relevant data about the incident. Think of yourself as a detective gathering clues to solve a mystery.
Assemble a diverse team: Get input from different parts of the organization. The more perspectives, the better your understanding of what went wrong.
Create a safe environment: Make it clear that this is a blame-free zone. The focus is on learning, not blaming.
Identify the root cause: Don’t stop at what happened. Keep asking “why” until you get to the core of the issue.
Brainstorm improvements: Once the root cause is identified, think about ways to prevent the problem from recurring. Encourage creative solutions.
Document and share: Write everything down and share it with your team. Knowledge is most valuable when it’s shared.

5. Best Practices for Incident Management and Blameless Postmortems

Now that you understand the basics, let’s look at some tips to take your incident management and postmortems to the next level:

Invest in automation: Use tools that can detect and respond to incidents quickly. It’s like giving your spaceship an AI co-pilot to help monitor the systems in real-time.
Define clear roles: During an incident, everyone should know their specific responsibility. This prevents chaos and ensures a more coordinated response.
Foster transparency: Be honest about incidents, both internally and with your customers. Transparency builds trust, and trust is key to customer satisfaction.
Regularly review and refine: The tech landscape is always changing, so your incident management processes should evolve too. Keep reviewing and improving them.
Celebrate successes: When your team handles an incident well, take the time to recognize their effort. Celebrating successes reinforces positive behavior and keeps morale high.

6. Embracing a Journey of Continuous Improvement

We have taken a journey through the fascinating world of incident management and blameless postmortems. It’s more than just a skill for the job, it’s a mindset that fosters continuous improvement.

Mastering these practices is key to becoming an exceptional DevOps or SRE professional. But more importantly, it’s about adopting a philosophy of learning from every incident, evolving from every mistake, and pushing our digital spaceships to fly higher and higher.

So, the next time something goes wrong, remember: it’s not just an incident, it’s an opportunity to learn, grow, and get even better. After all, isn’t that what continuous improvement is all about?

September 24, 2024 by Fernando SRE DevOps stuff SRE stuff