SRE

Confessions of a recovering GitOps addict

There’s a moment in every tech trend’s lifecycle when the magic starts to wear off. It’s like realizing the artisanal, organic, free-range coffee you’ve been paying eight dollars for just tastes like… coffee. For me, and many others in the DevOps trenches, that moment has arrived for GitOps.

We once hailed it as the silver bullet, the grand unifier, the one true way. Now, I’m here to tell you that the romance is over. And something much more practical is taking its place.

The alluring promise of a perfect world

Let’s be honest, we all fell hard for GitOps. The promise was intoxicating. A single source of truth for our entire infrastructure, nestled right in the warm, familiar embrace of Git. Pull Requests became the sacred gates through which all changes must pass. CI/CD pipelines were our holy scrolls, and tools like ArgoCD and Flux were the messiahs delivering us from the chaos of manual deployments.

It was a world of perfect order. Every change was audited, every state was declared, and every rollback was just a git revert away. It felt clean. It felt right. It felt… professional. For a while, it was the hero we desperately needed.

The tyranny of the pull request

But paradise had a dark side, and it was paved with endless YAML files. The first sign of trouble wasn’t a catastrophic failure, but a slow, creeping bureaucracy that we had built for ourselves.

Need to update a single, tiny secret? Prepare for the ritual. First, the offering: a Pull Request. Then, the prayer for the high priests (your colleagues) to grant their blessing (the approval). Then, the sacrifice (the merge). And finally, the tense vigil, watching ArgoCD’s sync status like it’s a heart monitor, praying it doesn’t flatline.

The lag became a running joke. Your change is merged… but has it landed in production? Who knows! The sync bot seems to be having a bad day. When everything is on fire at 2 AM, Git is like that friend who proudly tells you, “Well, according to my notes, the plan was for there not to be a fire.” Thanks, Git. Your record of intent is fascinating, but I need a fire hose, not a historian.

We hit our wall during what should have been a routine update.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: auth-service
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: auth-service-container
        image: our-app:v1.12.4
        envFrom:
        - secretRef:
            name: production-credentials

A simple change to the production-credentials secret required updating an encrypted file, PR-ing it, and then explaining in the commit message something like, “bumping secret hash for reasons”. Nobody understood it. Infrastructure changes started to require therapy sessions just to get merged.

And then, the tools fought back

When a system creates more friction than it removes, a rebellion is inevitable. And the rebels have arrived, not with pitchforks, but with smarter, more flexible tools.

First, the idea that developers should be fluent in YAML began to die. Internal Developer Platforms (IDPs) like Backstage and Port started giving developers what they always wanted: self-service with guardrails. Instead of wrestling with YAML syntax, they click a button in a portal to provision a database or spin up a new environment. Git becomes a log of what happened, not a bottleneck to make things happen.

Second, we remembered that pushing things can be good. The pull-based model was trendy, but let’s face it: push is immediate. Push is observable. We’ve gone back to CI pipelines pushing manifests directly into clusters, but this time they’re wearing body armor.

# This isn't your old wild-west kubectl apply
# It's a command wrapped in an approval system, with observability baked in.
deploy-cli --service auth-service --env production --approve

The change is triggered precisely when we want it, not when a bot feels like syncing. Finally, we started asking a radical question: why are we describing infrastructure in a static markup language when we could be programming it? Tools like Pulumi and Crossplane entered the scene. Instead of hundreds of lines of YAML, we’re writing code that feels alive.

import * as aws from "@pulumi/aws";

// Create an S3 bucket with versioning enabled.
const bucket = new aws.s3.Bucket("user-uploads-bucket", {
    versioning: {
        enabled: true,
    },
    acl: "private",
});

Infrastructure can now react to events, be composed into reusable modules, and be written in a language with types and logic. YAML simply can’t compete with that.

A new role for the abdicated king

So, is GitOps dead? No, that’s just clickbait. But it has been demoted. It’s no longer the king ruling every action; it’s more like a constitutional monarch, a respected elder statesman.

It’s fantastic for auditing, for keeping a high-level record of intended state, and for infrastructure teams that thrive on rigid discipline. But for high-velocity product teams, it’s become a beautifully crafted anchor when what we need is a motor.

We’ve moved from “Let’s define everything in Git” to “Let’s ship faster, safer, and saner with the right tools for the job.”

Our current stack is a hybrid, a practical mix of the old and new:

Backstage to abstract away complexity for developers.
Push-based pipelines with strong guardrails for immediate, observable deployments.
Pulumi for typed, programmable, and composable infrastructure.
Minimal GitOps for what it does best: providing a clear, auditable trail of our intentions.

GitOps wasn’t a mistake; it was the strict but well-meaning grandparent of infrastructure management. It taught us discipline and the importance of getting approval before touching anything important. But now that we’re grown up, that level of supervision feels less like helpful guidance and more like having someone watch over your shoulder while you type, constantly asking, “Are you sure you want to save that file?” The world is moving on to flexibility, developer-first platforms, and code you can read without a decoder ring. If you’re still spending your nights appeasing the YAML gods with Pull Request sacrifices for trivial changes… you’re not just living in the past, you’re practically a fossil.

That awkward moment when On-Prem is cheaper

Let’s be honest. For the better part of a decade, the public cloud has been the charismatic, free-spending friend who gets you out of any jam. Need to throw a last-minute party for a million users? They’ve got the hardware. Need to scale an app overnight? They’re already warming up the car. It was fast, it was elastic, and it saved you from the tedious, greasy work of racking your own servers. The only price was a casual, “You can pay me back later.”

Well, it’s later. The bill has arrived, and it has more cryptic line items than a forgotten ancient language. The finance department is calling, and they don’t sound happy.

This isn’t an angry stampede for the exits. Nobody is burning their AWS credits in protest. It’s more of a pragmatic reshuffle, a collective moment of clarity. Teams are looking at their sprawling digital estates and asking a simple question: Does everything really need to live in this expensive, all-inclusive resort? The result is a new normal where the cloud is still essential, just not universal.

The financial hangover

The cloud is wonderfully elastic. Elastic things, by their nature, bounce. So do monthly statements. Teams that scaled at lightning speed are now waking up to a familiar financial hangover with four distinct symptoms. First, there’s the billing complexity. Your monthly invoice isn’t a bill; it’s a mystery novel written by a sadist. Thousands of line items, tiered pricing, and egress charges transform the simple act of “moving data” into a budget-devouring monster.

-- A query that looks innocent but costs a fortune in data egress
SELECT
    event_id,
    user_id,
    payload
FROM
    user_events_production.events_archive
WHERE
    event_date BETWEEN '2025-07-01' AND '2025-07-31'
    AND region != 'eu-central-1'; -- Oh, you wanted to move 5TB across continents? That'll be extra.

Second is the unpredictable demand. A few busy weeks, a successful marketing campaign, or a minor viral event can undo months of careful savings plans. You budget for a quiet month, and suddenly you’re hosting the Super Bowl.

Then come the hidden multipliers. These are the gremlins of your infrastructure. Tiny, seemingly insignificant charges for cross-AZ traffic, managed service premiums, and per-request pricing that quietly multiply in the dark, feasting on your budget.

Finally, there’s the convenience tax. You paid a premium to turn the pain of operations into someone else’s problem. But for workloads that are steady, predictable, and bandwidth-heavy, that convenience starts to look suspiciously like setting money on fire. Those workloads are starting to look much cheaper on hardware you own or lease, where capital expenditure and depreciation replace the tyranny of per-hour-everything.

The gilded cage of convenience

Cloud providers don’t lock you in with malice. They seduce you with helpfulness. They offer a proprietary database so powerful, an event bus so seamless, an identity layer so integrated that before you know it, your application is woven into the very fabric of their ecosystem.

Leaving isn’t a migration; it’s a full-scale renovation project. It’s like living in a luxury hotel. They don’t forbid you from leaving, but once you’re used to the 24/7 room service, are you really going to go back to cooking for yourself?

Faced with this gilded cage, smart teams are now insisting on a kind of technological prenuptial agreement. It’s not about a lack of trust; it’s about preserving future freedom. Where practical, they prefer:

Open databases or engines with compatible wire protocols.
Kubernetes with portable controllers over platform-specific orchestration.
OpenTelemetry for metrics and traces that can travel.
Terraform or Crossplane to describe infrastructure in a way that isn’t tied to one vendor.

This isn’t purity theater. It simply reduces the penalty for changing your mind later.

# A portable infrastructure module
# It can be pointed at AWS, GCP, or even an on-prem vSphere cluster
# with the right provider.

resource "kubernetes_namespace" "app_namespace" {
  metadata {
    name = "my-awesome-app"
  }
}

resource "helm_release" "app_database" {
  name       = "app-postgres"
  repository = "https://charts.bitnami.com/bitnami"
  chart      = "postgresql"
  namespace  = kubernetes_namespace.app_namespace.metadata[0].name

  values = [
    "${file("values/postgres-prod.yaml")}"
  ]
}

A new menu of choices

The choice is no longer just between a hyperscaler and a dusty server cupboard under the stairs. The menu has expanded:

Private cloud: Using platforms like OpenStack or Kubernetes on bare metal in a modern colocation facility.
Alternative clouds: A growing number of providers are offering simpler pricing and less lock-in.
Hybrid models: Keeping sensitive data close to home while bursting to public regions for peak demand.
Edge locations: For workloads that need to be physically close to users and hate round-trip latency.

The point isn’t to flee the public cloud. The point is workload fitness. You wouldn’t wear hiking boots to a wedding, so why run a predictable, data-heavy analytics pipeline on a platform optimized for spiky, uncertain web traffic?

A personality test for your workload

So, how do you decide what stays and what goes? You don’t need a crystal ball. You just need to give each workload a quick personality test. Ask these six questions:

Is its demand mostly steady or mostly spiky? Is it a predictable workhorse or a temperamental rock star?
Is its data large and chatty or small and quiet?
Is latency critical? Does it need instant responses or is a few dozen milliseconds acceptable?
Are there strict data residency or compliance rules?
Does it rely on a proprietary managed service that would be a nightmare to replace?
Can we measure its unit economics? Do we know the cost per request, per user, or per gigabyte processed?

Steady and heavy often wins on owned or leased hardware. Spiky and uncertain still loves the elasticity of the hyperscalers. Regulated and locality-bound prefer the control of a private or hybrid setup. And if a workload gets its superpowers from a proprietary managed service, you either keep it where its powers live or make peace with a less super version of your app.

What does this mean for you, Architect

If you’re a DevOps engineer or a Cloud Architect, congratulations. Your job description just grew a new wing. You are no longer just a builder of digital infrastructure; you are now part financial planner, part supply chain expert, and part treaty negotiator.

Your playbook now includes:

FinOps literacy: The ability to connect design choices to money in a way the business understands and trusts.
Portability patterns: Designing services that can move without a complete rewrite.
Hybrid networking: Weaving together different environments without creating a haunted house of routing tables and DNS entries.
Observability without borders: Using vendor-neutral signals to see what’s happening from end to end.
Procurement fluency: The skill to make apples-to-apples comparisons between amortized hardware, managed services, and colocation contracts.

Yes, it’s time to carry a pocket calculator again, at least metaphorically.

The unsexy path to freedom

The journey back from the cloud is paved with unglamorous but essential work. It’s not a heroic epic; it’s a series of small, carefully planned steps. The risks are real. You have to account for the people cost of patching and maintaining private platforms, the lead times for hardware, and the shadow dependencies on convenient features you forgot you were using.

The antidote is small steps, honest metrics, and boringly detailed runbooks. Start with a proof-of-concept, create a migration plan that moves slices, not the whole cake, and have rollback criteria that a non-engineer can understand.

This is just a course correction

The Great Cloud Exit is less a rebellion and more a rationalization. Think of it as finally cleaning out your closet after a decade-long shopping spree. The public cloud gave us a phenomenal decade of speed, and we bought one of everything. Now, we’re sorting through the pile. That spiky, unpredictable web service? It still looks great in the elastic fabric of a hyperscaler. That massive, steady-state analytics database? It’s like a heavy wool coat that was never meant for the tropics; it’s time to move it to a more suitable climate, like your own data center. And that experimental service you spun up in 2019 and forgot about? That’s the impulse buy sequin jacket you’re never going to wear. Time to donate it.

Treating workload placement as a design problem instead of a loyalty test is liberating. It’s admitting you don’t need a Swiss Army knife when all you have to do is turn a single screw. Choosing the right environment for the job results in a system that costs less and complains less. It performs better because it’s not being forced to do something it was never designed for.

This leads to the most important outcome: options. In a landscape that changes faster than we can update our résumés, flexibility is the only superpower that truly matters. The ability to move, to adapt, and to choose without facing a punishing exit fee or a six-month rewrite, that’s the real prize. The cloud isn’t the destination anymore; it’s just one very useful stop on the map.

August 11, 2025 by Fernando SRE Cloud stuff DevOps stuff SRE stuff

What is AWS Nucleus, and why Is it poised to replace EC2?

It all started with a coffee and a bill. My usual morning routine. But this particular Tuesday, the AWS bill had an extra kick that my espresso lacked. The cost for a handful of m5.large instances had jumped nearly 40% over the past year. I almost spat out my coffee.

I did what any self-respecting Cloud Architect does: I blamed myself. Did I forget to terminate a dev environment? Did I leave a data transfer running to another continent? But no. After digging through the labyrinth of Cost Explorer, the truth was simpler and far more sinister: EC2 was quietly getting more expensive. Spot instances had become as predictable as a cat on a hot tin plate, and my “burstable” CPUs seemed to run out of breath if they had to do more than jog for a few minutes.

EC2, our old, reliable friend. The bedrock of the cloud. It felt like watching your trusty old car suddenly start demanding premium fuel and imported spare parts just to get to the grocery store. Something was off.

And then, it happened. A slip-up in a public Reddit forum. A senior AWS engineer accidentally posted a file named ec2-phaseout-q4–2027.pdf. It was deleted in minutes, but the internet, as we know, has the memory of an elephant with a grudge.

(Disclaimer for the nervous: This PDF is my narrative device. A ghost in the machine. A convenient plot twist. But the trends it points to? The rising costs, the architectural creaks? Those are very, very real. Now, where were we?)

The document was a bombshell. It laid out a plan to deprecate over 80% of current EC2 instance families by the end of 2027, paving the way for a “next-gen compute platform.” Was this real? I made some calls. The first partner laughed it off. The second went quiet, a little too quiet. The third, after I promised to buy them beers for a month, whispered: “We’re already planning the transition for our enterprise clients.”

Bingo.

Why our beloved EC2 is becoming a museum piece

My lead engineer summed it up beautifully last week. “Running real-time ML on today’s EC2,” he sighed, “feels like asking a 2010 laptop to edit 4K video. It’ll do it, but it’ll scream in agony the whole time, and you’d better have a fire extinguisher handy.”

He’s not wrong. For general-purpose apps, EC2 is still a trusty workhorse. But for the demanding, high-performance workloads that are becoming the norm? You can practically see the gray hairs and hear the joints creaking.

This isn’t just about cost. It’s about architecture. EC2 was built for a different era, an era before serverless was cool, before WebAssembly (WASM) was a thing, and before your toaster needed to run a Kubernetes cluster. The cracks are starting to show.

Meet AWS Nucleus, the secret successor

No press release. No re:Invent keynote. But if you’re connected to AWS insiders, you’ve probably heard whispers of a project internally codenamed “Nucleus.” We got access to this stealth-mode compute platform, and it’s unlike anything we’ve used before.

What does it feel like? Think of it this way: if Lambda and Fargate had a baby, and that baby was raised by a bare-metal server with a PhD in performance, you’d get Nucleus. It has the speed and direct hardware access of a dedicated machine, but with the auto-scaling magic of serverless.

Here are some of the early capabilities we’ve observed:

No more cold starts. Unlike Lambda, which can sometimes feel like it’s waking up from a deep nap.
Direct hardware access. Full control over GPU and SSD resources without the usual virtualization overhead.
Predictive autoscaling. It analyzes traffic patterns and scales before the spike hits, not during.
WASM-native runtime. Support for Node.js, Python, Go, and Rust is baked in from the ground up.

It’s not generally available yet, but internal teams and a select few partners are already building on it.

A 30-day head-to-head test

Yes, we triple checked those cost figures. Even if AWS adjusts the pricing after the preview, the efficiency gap is too massive to ignore.

Your survival guide for the coming shift

Let’s be clear, there’s no need to panic and delete all your EC2 instances. But if this memo is even half-right, you don’t want to be caught flat-footed in a few years. Here’s what we’re doing, and what you might want to start experimenting with.

Step 1: Become a cloud whisperer

Start by pinging your AWS Solutions Architect, not directly about “Nucleus,” but something softer:

“Hey, we’re exploring options for more performant, cost-effective compute. Are there any next-gen runtimes or private betas AWS is piloting that we could look into?”

You’ll be surprised what folks share if you ask the right way.

Step 2: test on the shadow platform

Some partners already have early access CLI builds. If you get your hands on one, you’ll notice some familiar patterns.

# Initialize a new service from a template
nucleus init my-api --template=fastapi

# Deploy with a single command
nucleus deploy --env=staging --free-tier

Disclaimer: Not officially available. Use in isolated test environments only. Do not run your production database on this.

Step 3: Run a hybrid setup

If you get preview access, try bridging the old with the new. Here’s a hypothetical Terraform snippet of what that might look like:

# Our legacy EC2 instance for the old monolith
resource "aws_instance" "legacy_worker" {
  ami           = "ami-0b5eea76982371e9" # An old Amazon Linux 2 AMI
  instance_type = "t3.medium"
}

# The new Nucleus service for a microservice
resource "aws_nucleus_service" "new_api" {
  runtime       = "go1.19"
  source_path   = "./app/api"
  
  # This is the magic part: linking to the old world
  vpc_ec2_links = [aws_instance.legacy_worker.id]
}

We ran a few test loads between legacy workers and the new compute, no regressions, and latency even dropped.

Step 4: Estimate the savings yourself

Even with preview pricing, the gap is noticeable. A simple Python script can give you a rough idea.

# Fictional library to estimate costs
import aws_nucleus_estimator

# Your current monthly bill for a specific workload
current_ec2_cost = 4200 

# Estimate based on vCPU hours and memory
# (These numbers are for illustration only)
estimated_nucleus_cost = aws_nucleus_estimator.estimate(
    vcpu_hours=1200, 
    memory_gb_hours=2400
)

print(f"Rough monthly savings: ${current_ec2_cost - estimated_nucleus_cost}")

This is bigger than just EC2

Let’s be honest. This shift isn’t just about cutting costs or shrinking cold start times. It’s about redefining what “compute” even means. EC2 isn’t being deprecated because it’s broken. It’s being phased out because modern workloads have evolved, and the old abstractions are starting to feel like training wheels we forgot to take off.

A broader pattern is emerging across the industry. What AWS is allegedly doing with Nucleus mirrors a larger movement:

Google Cloud is reportedly piloting a Cloud Run variant that uses a WASM-based runtime.
Microsoft Azure is quietly testing a system to blur the line between containers and functions.
Oracle, surprisingly, has been sponsoring development tools optimized for WASM-native environments.

The foundational idea is clear: cloud platforms are moving toward fast-boot, auto-scaling, WASM-capable compute that sits somewhere between Lambda and Kubernetes, but without the overhead of either.

Is EC2 the new legacy?

It’s strange to say, but EC2 is starting to feel like “bare metal” did a decade ago: powerful, essential, but something you try to abstract away.

One of our SREs shared this gem the other day:

“A couple of our junior engineers thought EC2 was some kind of disaster recovery tool for Kubernetes.”

That’s from a Fortune 100 company. When your flagship infrastructure service starts raising eyebrows from fresh grads, you know a generational shift is underway.

The cloud is evolving, again. But this isn’t a gentle, planned succession. It’s a Cambrian explosion in real-time. New, bizarre forms of compute are crawling out of the digital ooze, and the old titans, once thought invincible, are starting to look slow and clumsy. They don’t get a gold watch and a retirement party. They become fossils, their skeletons propping up the new world.

EC2 isn’t dying tomorrow. It’s becoming a geological layer. It’s the bedrock, the sturdy but unglamorous foundation upon which nimbler, more specialized predators will hunt. The future isn’t about killing the virtual machine; it’s about making it an invisible implementation detail. In the same way, most of us stopped thinking about the physical server racks in a data center, we’ll soon stop thinking about the VM. We’ll just care about the work that needs doing.

So no, EC2 isn’t dying. It’s becoming a legend. And in the fast-moving world of technology, legends belong in museums, admired from a safe distance.

August 11, 2025 by Fernando SRE Cloud stuff DevOps stuff SRE stuff

When docker compose stopped being magic

There was a time, not so long ago, when docker-compose up felt like performing a magic trick. You’d scribble a few arcane incantations into a YAML file and, poof, your entire development stack would spring to life. The database, the cache, your API, the frontend… all humming along obediently on localhost. Docker Compose wasn’t just a tool; it was the trusty Swiss Army knife in every developer’s pocket, the reliable friend who always had your back.

Until it didn’t.

Our breakup wasn’t a single, dramatic event. It was a slow fade, the kind of awkward drifting apart that happens when one friend grows and the other… well, the other is perfectly happy staying exactly where they are. It began with small annoyances, then grew into full-blown arguments. We eventually realized we were spending more time trying to fix our relationship with YAML than actually building things.

So, with a heavy heart and a sigh of relief, we finally said goodbye.

The cracks begin to show

As our team and infrastructure matured, our reliable friend started showing some deeply annoying habits. The magic tricks became frustratingly predictable failures.

Our services started giving each other the silent treatment. The networking between containers became as fragile and unpredictable as a Wi-Fi connection on a cross-country train. One moment they were chatting happily, the next they wouldn’t be caught dead in the same virtual network.
It was worse at keeping secrets than a gossip columnist. The lack of native, secure secret handling was, to put it mildly, a joke. We were practically writing passwords on sticky notes and hoping for the best.
It developed a severe case of multiple personality disorder. The same docker-compose.yml file would behave like a well-mannered gentleman on one developer’s machine, a rebellious teenager in staging, and a complete, raving lunatic in production. Consistency was not its strong suit.
The phrase “It works on my machine” became a ritualistic chant. We’d repeat it, hoping to appease the demo gods, but they are a fickle bunch and rarely listened. We needed reliability, not superstition.

We had to face the truth. Our old friend just couldn’t keep up.

Moving on to greener pastures

The final straw was the realization that we had become full-time YAML therapists. It was time to stop fixing and start building again. We didn’t just dump Compose; we replaced it, piece by piece, with tools that were actually designed for the world we live in now.

For real infrastructure, we chose real code

For our production and staging environments, we needed a serious, long-term commitment. We found it in the AWS Cloud Development Kit (CDK). Instead of vaguely describing our needs in YAML and hoping for the best, we started declaring our infrastructure with the full power and grace of TypeScript.

We went from a hopeful plea like this:

# docker-compose.yml
services:
  api:
    build: .
    ports:
      - "8080:8080"
    depends_on:
      - database
  database:
    image: "postgres:14-alpine"

To a confident, explicit declaration like this:

// lib/api-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as ecs_patterns from 'aws-cdk-lib/aws-ecs-patterns';

// ... inside your Stack class
const vpc = /* your existing VPC */;
const cluster = new ecs.Cluster(this, 'ApiCluster', { vpc });

// Create a load-balanced Fargate service and make it public
new ecs_patterns.ApplicationLoadBalancedFargateService(this, 'ApiService', {
  cluster: cluster,
  cpu: 256,
  memoryLimitMiB: 512,
  desiredCount: 2, // Let's have some redundancy
  taskImageOptions: {
    image: ecs.ContainerImage.fromRegistry("your-org/your-awesome-api"),
    containerPort: 8080,
  },
  publicLoadBalancer: true,
});

It’s reusable, it’s testable, and it’s cloud-native by default. No more crossed fingers.

For local development, we found a better roommate

Onboarding new developers had become a nightmare of outdated README files and environment-specific quirks. For local development, we needed something that just worked, every time, on every machine. We found our perfect new roommate in Dev Containers.

Now, we ship a pre-configured development environment right inside the repository. A developer opens the project in VS Code, it spins up the container, and they’re ready to go.

Here’s the simple recipe in .devcontainer/devcontainer.json:

{
  "name": "Node.js & PostgreSQL",
  "dockerComposeFile": "docker-compose.yml", // Yes, we still use it here, but just for this!
  "service": "app",
  "workspaceFolder": "/workspace",

  // Forward the ports you need
  "forwardPorts": [3000, 5432],

  // Run commands after the container is created
  "postCreateCommand": "npm install",

  // Add VS Code extensions
  "extensions": [
    "dbaeumer.vscode-eslint",
    "esbenp.prettier-vscode"
  ]
}

It’s fast, it’s reproducible, and our onboarding docs have been reduced to: “1. Install Docker. 2. Open in VS Code.”

To speak every Cloud language, we hired a translator

As our ambitions grew, we needed to manage resources across different cloud providers without learning a new dialect for each one. Crossplane became our universal translator. It lets us manage our infrastructure, whether it’s on AWS, GCP, or Azure, using the language we already speak fluently: the Kubernetes API.

Want a managed database in AWS? You don’t write Terraform. You write a Kubernetes manifest.

# rds-instance.yaml
apiVersion: database.aws.upbound.io/v1beta1
kind: RDSInstance
metadata:
  name: my-production-db
spec:
  forProvider:
    region: eu-west-1
    instanceClass: db.t3.small
    masterUsername: admin
    allocatedStorage: 20
    engine: postgres
    engineVersion: "14.5"
    skipFinalSnapshot: true
    # Reference to a secret for the password
    masterPasswordSecretRef:
      namespace: crossplane-system
      name: my-db-password
      key: password
  providerConfigRef:
    name: aws-provider-config

It’s declarative, auditable, and fits perfectly into a GitOps workflow.

For the creative grind, we got a better workflow

The constant cycle of code, build, push, deploy, test, repeat for our microservices was soul-crushing. Docker Compose never did this well. We needed something that could keep up with our creative flow. Skaffold gave us the instant gratification we craved.

One command, skaffold dev, and suddenly we had:

Live code syncing to our development cluster.
Automatic container rebuilds and redeployments when files change.
A unified configuration for both development and production pipelines.

No more editing three different files and praying. Just code.

The slow fade was inevitable

Docker Compose was a fantastic tool for a simpler time. It was perfect when our team was small, our application was a monolith, and “production” was just a slightly more powerful laptop.

But the world of software development has moved on. We now live in an era of distributed systems, cloud-native architecture, and relentless automation. We didn’t just stop using Docker Compose. We outgrew it. And we replaced it with tools that weren’t just built for the present, but are ready for the future.

August 6, 2025 by Fernando SRE Cloud stuff DevOps stuff SRE stuff

If your Kubernetes YAML looks Like hieroglyphics, this post is for you

It all started, as most tech disasters do, with a seductive whisper. “Just describe your infrastructure with YAML,” Kubernetes cooed. “It’ll be easy,” it said. And we, like fools in a love story, believed it.

At first, it was a beautiful romance. A few files, a handful of lines. It was elegant. It was declarative. It was… manageable. But entropy, the nosy neighbor of every DevOps team, had other plans. Our neat little garden of YAML files soon mutated into a sprawling, untamed jungle of configuration.

We had 12 microservices jostling for position, spread across 4 distinct environments, each with its own personality quirks and dark secrets. Before we knew it, we weren’t writing infrastructure anymore; we were co-authoring a Byzantine epic in a language seemingly designed by bureaucrats with a fetish for whitespace.

The question that broke the camel’s back

The day of reckoning didn’t arrive with a server explosion or a database crash. It came with a question. A question that landed in our team’s Slack channel with the subtlety of a dropped anvil, courtesy of a junior engineer who hadn’t yet learned to fear the YAML gods.

“Hey, why does our staging pod have a different CPU limit than prod?”

Silence. A deep, heavy, digital silence. The kind of silence that screams, “Nobody has a clue.”

What followed was an archaeological dig into the fossil record of our own repository. We unearthed layers of abstractions we had so cleverly built, peeling them back one by one. The trail led us through a hellish labyrinth:

We started at deployment.yaml, the supposed source of all truth.
That led us to values.yaml, the theoretical source of all truth.
From there, we spelunked into values.staging.yaml, where truth began to feel… relative.
We stumbled upon a dusty patch-cpu-emergency.yaml, a fossil from a long-forgotten crisis.
Then we navigated the dark forest of custom/kustomize/base/deployment-overlay.yaml.
And finally, we reached the Rosetta Stone of our chaos: an argocd-app-of-apps.yaml.

The revelation was as horrifying as finding a pineapple on a pizza: we had declared the same damn value six times, in three different formats, using two tools that secretly despised each other. We weren’t managing the configuration. We were performing a strange, elaborate ritual and hoping the servers would be pleased.

That’s when we knew. This wasn’t a configuration problem. It was an existential crisis. We were, without a doubt, deep in YAML Hell.

The tools that promised heaven and delivered purgatory

Let’s talk about the “friends” who were supposed to help. These tools promised to be our saviors, but without discipline, they just dug our hole deeper.

Helm, the chaotic magician

Helm is like a powerful but slightly drunk magician. When it works, it pulls a rabbit out of a hat. When it doesn’t, it sets the hat on fire, and the rabbit runs off with your wallet.

The Promise: Templating! Variables! A whole ecosystem of charts!

The Reality: Debugging becomes a form of self-torment that involves piping helm template into grep and praying. You end up with conditionals inside your templates that look like this:

image:
  repository: {{ .Values.image.repository | quote }}
  tag: {{ .Values.image.tag | default .Chart.AppVersion }}
  pullPolicy: {{ .Values.image.pullPolicy | default "IfNotPresent" }}

This looks innocent enough. But then someone forgets to pass image.tag for a specific environment, and you silently deploy :latest to production on a Friday afternoon. Beautiful.

Kustomize the master of patches

Kustomize is the “sensible” one. It’s built into kubectl. It promises clean, layered configurations. It’s like organizing your Tupperware drawer with labels.

The Promise: A clean base and tidy overlays for each environment.

The Reality: Your patch files quickly become a mystery box. You see this in your kustomization.yaml:

patchesStrategicMerge:
  - increase-replica-count.yaml
  - add-resource-limits.yaml
  - disable-service-monitor.yaml

Where are these files? What do they change? Why does disable-service-monitor.yaml only apply to the dev environment? Good luck, detective. You’ll need it.

ArgoCD, the all-seeing eye (that sometimes blinks)

GitOps is the dream. Your Git repo is the single source of truth. No more clicking around in a UI. ArgoCD or Flux will make it so.

The Promise: Declarative, automated sync from Git to cluster. Rollbacks are just a git revert away.

The Reality: If your Git repo is a dumpster fire of conflicting YAML, ArgoCD will happily, dutifully, and relentlessly sync that dumpster fire to production. It won’t stop you. One bad merge, and you’ve automated a catastrophe.

Our escape from YAML hell was a five-step sanity plan

We knew we couldn’t burn it all down. We had to tame the beast. So, we gathered the team, drew a line in the sand, and created five commandments for configuration sanity.

1. We built a sane repo structure

The first step was to stop the guesswork. We enforced a simple, predictable layout for every single service.

├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── configmap.yaml
└── overlays/
    ├── dev/
    │   ├── kustomization.yaml
    │   └── values.yaml
    ├── staging/
    │   ├── kustomization.yaml
    │   └── values.yaml
    └── prod/
        ├── kustomization.yaml
        └── values.yaml

This simple change eliminated 80% of the “wait, which file do I edit?” conversations.

2. One source of truth for values

This was a sacred vow. Each environment gets one values.yaml file. That’s it. We purged the heretics:

values-prod-final.v2.override.yaml
backup-of-values.yaml
donotdelete-temp-config.yaml

If a value wasn’t in the designated values.yaml for that environment, it didn’t exist. Period.

3. We stopped mixing Helm and Kustomize

You have to pick a side. We made a rule: if a service requires complex templating logic, use Helm. If it primarily needs simple overlays (like changing replica counts or image tags per environment), use Kustomize. Using both on the same service is like trying to write a sentence in two languages at once. It’s a recipe for suffering.

4. We render everything before deploying

Trust, but verify. We added a mandatory step in our CI pipeline to render the final YAML before it ever touches the cluster.

# For Helm + Kustomize setups
helm template . --values overlays/prod/values.yaml | \
kustomize build | \
kubeval -

This simple script does three magical things:

It validates that the output is syntactically correct YAML.
It lets us see exactly what is about to be applied.
It has completely eliminated the “well, that’s not what I expected” class of production incidents.

5. We built a simple config CLI

To make the right way the easy way, we built a small internal CLI tool. Now, instead of navigating the YAML jungle, an engineer simply runs:

$ ops-cli config generate --app=user-service --env=prod

This tool:

Pulls the correct base templates and overlay values.
Renders the final, glorious YAML.
Validates it against our policies.
Shows the developer a diff of what will change in the cluster.
Saves lives and prevents hair loss.

YAML is now a tool again, not a trap.

The afterlife is peaceful

YAML didn’t ruin our lives. We did, by refusing to treat it with the respect it demands. Templating gives you incredible power, but with great power comes great responsibility… and redundancy, and confusion, and pull requests with 10,000 lines of whitespace changes. Now, we treat our YAML like we treat our application code. We lint it. We test it. We render it. And most importantly, we’ve built a system that makes it difficult to do the wrong thing. It’s the institutional equivalent of putting childproof locks on the kitchen cabinets. A determined toddler could probably still get to the cleaning supplies, but it would require a conscious, frustrating effort. Our system doesn’t make us smarter; it just makes our inevitable moments of human fallibility less catastrophic. It’s the guardrail on the scenic mountain road of configuration. You can still drive off the cliff, but you have to really mean it. Our infrastructure is no longer a hieroglyphic. It’s just… configuration. And the resulting boredom is a beautiful thing.

August 3, 2025 by Fernando SRE Cloud stuff DevOps stuff Kubernetes SRE stuff

What if Kubernetes was the wrong tool for almost everyone?

It was 11 PM on a Tuesday, and for the third time that week, we were on an emergency call. Not because our product had a critical bug, but because a routine deployment had, once again, mysteriously broken the internal DNS. Three of our sharpest engineers weren’t creating value; they were offering sacrifices to the capricious god of Istio, hoping it might bless our pods with connectivity.

As I stared at the sprawling diagram we’d made just to add a single, simple microservice, a heretical thought wormed its way into my brain: When did our job stop being about building software and become about… well, serving Kubernetes?

A rumor we couldn’t ignore

The next morning, amidst our collective YAML-induced hangover, someone dropped a screenshot into our team’s Slack channel. It was from some tiny, no-name startup, claiming they were getting 8x the performance of a standard Kubernetes stack at one-tenth of the cost.

The team’s reaction was a collective, cynical laugh. “Sure,” our lead SRE typed, “and my home server can out-render Pixar.” It was obviously marketing fluff. Propaganda. The post itself was deleted within an hour, but the screenshot had already gone viral. It was absurd, unbelievable, and we all knew it was nonsense.

But the idea, like a catchy, terrible pop song, got stuck in our heads.

Let’s just prove it’s impossible

That Friday, we decided to do it. We’d run a small, contained experiment, mostly for the bragging rights of publicly debunking the ridiculous claim. The plan was simple: take one of our standard, moderately complex services and see what it would take to run it on a simpler stack.

Our service, an image processor, wasn’t a behemoth, but its YAML file had grown… organically. Like a colony of particularly stubborn mold. It had sidecars, persistent volume claims, readiness probes, and enough annotations to qualify as a short novel.

Here’s a sanitized glimpse of the beast we were trying to tame:

# apiVersion: apps/v1
# kind: Deployment
# metadata:
#   name: image-processor-svc
#   labels:
#     app: image-processor
# spec:
#   replicas: 3
#   selector:
#     matchLabels:
#       app: image-processor
#   template:
#     metadata:
#       labels:
#         app: image-processor
#     spec:
#       containers:
#       - name: processor
#         image: our-repo/image-processor:v1.2.4
#         ports:
#         - containerPort: 8080
#         resources:
#           requests:
#             memory: "512Mi"
#             cpu: "250m"
#           limits:
#             memory: "1024Mi"
#             cpu: "500m"
#         readinessProbe:
#           httpGet:
#             path: /healthz
#             port: 8080
#           initialDelaySeconds: 5
#           periodSeconds: 10
#       - name: metrics-sidecar
#         image: prom/statsd-exporter:v0.22.0
#         args:
#         - "--statsd.mapping-config=/etc/statsd/mapper.yml"
#         ports:
#         - name: metrics
#           containerPort: 9102
#         volumeMounts:
#         - name: config-volume
#           mountPath: /etc/statsd
#   # ...and so on, for another 150 lines.

We figured it would take us all afternoon just to untangle it.

The uncomfortable silence of success

We set up a competing stack using Firecracker MicroVMs, essentially tiny, lightning-fast virtual machines. The goal was to run the same container, but without the entire Kubernetes universe orbiting around it.

By 3 PM, we had our first results. And that’s when the room went quiet. It was the kind of uncomfortable silence you get when you realize the joke you’ve been telling for months is actually on you.

The numbers weren’t just holding up; they were embarrassing us. We stared at the Grafana dashboard, waiting for the figures to make sense. They didn’t. Our projected monthly cloud bill for this single service didn’t just shrink; it plummeted.

We had spent years building a complex, expensive, and fragile machine, all to solve a problem that, it turned out, could be handled with a much, much simpler approach.

Our quest for sane infrastructure

That weekend experiment turned into a full-blown obsession. Inspired, we started building our own internal escape hatch. We cobbled together a tool that we jokingly called the “SQL-ifier.” The premise was simple and, to a Kubernetes purist, utterly profane: what if you could manage infrastructure with simple, readable rules instead of YAML incantations?

Instead of a 200-line YAML file for an autoscaler, what if you could just write this?

-- This is our internal, SQL-like syntax for managing MicroVMs.
-- It's not a public standard (yet!).

-- Rule: If CPU usage on the frontend service is over 80% for 3 minutes,
-- add two more instances, but never exceed 20 total.
ON high_cpu(>80%) FOR 3m
IF service.name = 'image-processor-svc'
DO SCALE service.name TO instances + 2
LIMIT 20;

-- Rule: If we see more than 10 critical payment errors in 1 minute,
-- immediately revert to the last stable version.
ON log_error(level='critical', service='payment-gateway') > 10 FOR 1m
DO ROLLBACK service 'payment-gateway' TO previous_stable;

It was declarative, readable, and, most importantly, it could be understood by a human being without needing a certification.

How did we all end up here?

This journey forced us to ask a bigger question. Why did we, and thousands of other smart teams, willingly chain ourselves to this complexity?

The answer is surprisingly human. We bought an 18-wheeler truck to do our weekly grocery shopping.

Sure, a giant truck can carry milk and eggs. But you spend most of your time finding a place to park it, paying for diesel, getting a special license to drive it, and explaining to your neighbors why you just flattened their mailbox. Kubernetes was built for Google-scale problems. Most of us run businesses that need the reliability of a Toyota Camry, not a fleet of space shuttles. We adopted the tool because everyone else did, mistaking its complexity for sophistication.

Is your infrastructure serving you?

This whole experience led us to create a simple “mirror test.” If you’re wondering if you’re in the same boat, ask your team these questions:

Do you dread upgrading your cluster more than you dread a root canal?
Is a significant portion of your engineering time spent on “infra-babysitting” instead of building product features?
Could you confidently explain your service mesh configuration to a new hire without a whiteboard and a two-hour meeting?

If you answered “yes” to two or more, you might not have an infrastructure problem. You might have a Kubernetes problem.

This isn’t a manifesto to uninstall kubectl tomorrow. Some organizations genuinely operate at a scale where Kubernetes is not just useful, but necessary. This is just a friendly nudge. A reminder to look up from your YAML files once in a while and ask: is this tool still serving me, or have I started serving it?

July 30, 2025 by Fernando SRE Cloud stuff DevOps stuff Kubernetes SRE stuff

Choosing your message queue AWS SQS or GCP Pub/Sub

In the world of modern software, applications are rarely monolithic islands. Instead, they are bustling cities of interconnected services, each performing a specific job. For this city to function smoothly, its inhabitants, microservices, functions, and components need a reliable way to communicate without being directly tethered to one another. This is where message brokers come in, acting as the city’s postal service, ensuring that messages are delivered efficiently and reliably.

Two of the most prominent cloud-based postal services are Amazon Web Services’ Simple Queue Service (SQS) and Google Cloud’s Pub/Sub. Both are exceptional at what they do, but they operate on different philosophies. Understanding their unique characteristics is crucial for any cloud architect or DevOps engineer aiming to build robust, scalable, and event-driven systems. This guide will explore their differences to help you choose the right service for your application’s needs.

A quick look at our contenders

Before we examine the details, let’s get a general feel for each service.

AWS SQS is the seasoned veteran of message queuing. Think of it as a highly organized system of mailboxes. A service writes a letter (a message) and places it into a specific mailbox (a queue). The recipient service then comes to that mailbox and picks up its mail when it has the capacity to process it. It’s a straightforward, incredibly reliable system that has been battle-tested for years.

GCP Pub/Sub operates more like a global newspaper subscription. A publisher (your service) doesn’t send a message to a specific recipient. Instead, it publishes a message to a “topic,” like a news flash for the “user-signup” channel. Any service that has subscribed to that topic instantly receives a copy of the message. It’s designed for broad, real-time distribution of information on a global scale.

The delivery dilemma Push versus Pull

The most fundamental difference between SQS and Pub/Sub lies in how messages are delivered. This is often referred to as the “push vs. pull” model.

The pull model, which is SQS’s native approach, is like checking your P.O. box. The consumer application is responsible for periodically asking the queue, “Is there any mail for me?” This gives the consumer complete control over the rate of consumption. If it’s overwhelmed with work, it can slow down its requests or stop asking for new messages altogether. This is ideal for batch processing or any workload where you need to manage the processing pace carefully.

The push model, where Pub/Sub shines, is akin to home mail delivery. When a message is published, Pub/Sub actively “pushes” it to all subscribed endpoints, such as a serverless function or a webhook. The recipient doesn’t have to ask; the message just arrives. This is incredibly efficient for real-time notifications and event-driven workflows where immediate reaction is key. While Pub/Sub also supports a pull model, its architecture is optimized for push-based delivery.

Comparing key features

Let’s break down how these two services stack up in a few critical areas.

Message ordering

Sometimes, the sequence of events is just as important as the events themselves. For these cases, AWS SQS offers a specific FIFO (First-In, First-Out) queue type. This works exactly like a single-file line at a bank; the first person to get in line is the first one to be served. It provides a strict guarantee that messages will be processed in the exact order they were sent, which is critical for tasks like processing financial transactions or application logs.

GCP Pub/Sub, in contrast, does not have a dedicated FIFO queue type. Instead, it achieves partial ordering through the use of ordering keys. You can assign a key to messages (for example, a userId), and Pub/Sub will ensure that all messages with that specific key are delivered in order. However, it doesn’t guarantee order between different keys. To reuse the analogy, it’s less like a single line and more like a deli with separate ticket numbers for the butcher and the bakery. It keeps orders straight within each department, but not across the entire store.

Scale and reach

This is where their architectural differences become clear. SQS is a regional service. It’s incredibly scalable and resilient, but its scope is confined to a single AWS region.

Pub/Sub is inherently global. You publish a message once, and it can be delivered to subscribers in any region around the world with low latency. If your application has a global user base and you need to propagate events worldwide, Pub/Sub has a distinct advantage.

Message size and retention

Think of SQS as being for postcards and letters. It supports messages up to 256 KB. It can hold onto these messages for up to 14 days, giving your consumers plenty of time to process them.

Pub/Sub, on the other hand, can handle larger packages, with a maximum message size of 10 MB. However, its standard retention period is shorter, at 7 days.

Special delivery options

SQS has a native feature called Delay Queues. This allows you to postpone the delivery of a new message for up to 15 minutes. It’s like writing a post-dated check; the message sits in the queue but is invisible to consumers until the timer expires. This is useful for scheduling tasks without a complex scheduling service. Pub/Sub does not offer a similar built-in feature.

When to choose AWS SQS

SQS is your go-to choice when you need a dependable, orderly mailroom for your application. It excels in scenarios where:

Strict ordering is non-negotiable. For task sequencing or financial ledgers, SQS FIFO is the gold standard.
You need to control the pace of consumption. The pull model is perfect for decoupling a fast producer from a slower consumer or for batch processing jobs.
Task scheduling is required. The native delay queue feature is a simple yet powerful tool.
Your application’s architecture is primarily contained within a single AWS region.

When to choose GCP Pub/Sub

Pub/Sub is the right tool when you’re building a global broadcasting system or a highly reactive, event-driven platform. Consider it when:

You need to fan-out messages to many consumers. Pub/Sub’s topic-and-subscription model is designed for this.
Global distribution with low latency is a priority. Its global nature is a massive benefit for distributed systems.
You are sending large messages. The 10 MB limit offers much more flexibility than SQS.
A push-based model fits your architecture. It integrates seamlessly with serverless functions for instant, event-triggered execution.

A final word

So, after all this technical deliberation, which digital courier should you entrust with your precious data packets? The one that meticulously forms a single, orderly queue, or the one that shouts your message through a global megaphone to anyone who will listen?

The truth is, there’s no single “best” service. There’s only the one whose particular brand of crazy best matches your application’s personality. Is your app a stickler for the rules, demanding every event be processed in perfect sequence, lest it have a digital panic attack? Then the quiet, predictable, and slightly obsessive SQS is your soulmate. Or is your app more of a drama queen, needing to announce every minor update to the entire world, immediately? Then the boisterous, globe-trotting Pub/Sub is probably already sliding into your DMs.

Ultimately, the best way to choose is to put them to the test. Think of it as a job interview. Give them both a trial run with your actual workload and see which one handles the pressure with more grace, or at least breaks in a more interesting, less catastrophic way. Go on, do it for science. And for the future sanity of your on-call engineer.

July 6, 2025 by Fernando SRE Cloud stuff

Integrate End-to-End testing for robust cloud native pipelines

We expect daily life to run smoothly. Our cars start instantly, our coffee brews perfectly, and streaming services play without a hitch. Similarly, today’s digital users have zero patience for software hiccups. To meet these expectations, many businesses now build cloud-native applications, highly scalable, flexible, and agile software. However, while our construction materials have changed, the need for sturdy, reliable software has only grown stronger. This is where End-to-End (E2E) testing comes in, verifying entire user workflows to ensure every software component seamlessly works together.

In this article, you’ll see practical ways to embed E2E tests effectively into your Continuous Integration and Continuous Delivery (CI/CD) pipelines, turning complexity into clarity.

Navigating the challenges of cloud-native testing

Traditional software testing was like assembling a static puzzle on a stable surface. Cloud-native testing, however, feels more like putting together a puzzle on a moving vehicle, every piece constantly shifts.

Complex microservice coordination

Cloud-native apps are often built with multiple microservices, each operating independently. Think of these as specialized workers collaborating on a complex project. If one worker stumbles, the whole project suffers. Microservices require precise coordination, making it tricky to identify and fix issues quickly.

Short-lived and shifting environments

Containers and Kubernetes create ephemeral, constantly changing environments. They’re like pop-up stores appearing briefly and disappearing overnight. Managing testing in these environments means handling dynamic URLs and quickly changing configurations, a challenge comparable to guiding customers to a food truck that relocates every day.

The constant quest for good test data

In dynamic environments, consistently managing accurate test data can feel impossible. It’s akin to a chef who finds their pantry randomly restocked every few minutes. Having fresh and relevant ingredients consistently ready becomes a monumental challenge.

Integrating quality directly into your CI/CD pipeline

Incorporating E2E tests into CI/CD is like embedding precision checkpoints directly onto an assembly line, catching problems as soon as they appear rather than after the entire product is built.

Early detection saves the day

Embedding E2E tests acts like multiple smoke detectors installed throughout a building rather than just one centrally located. Issues get pinpointed rapidly, preventing small problems from becoming massive headaches. Tools like Datadog Synthetic or Cypress allow parallel execution, speeding up the testing process dramatically.

Stopping errors before users see them

Failed E2E tests automatically halt deployments, ensuring faulty code doesn’t reach customers. Imagine a vigilant gatekeeper preventing defective products from leaving the factory, this is exactly how integrated E2E tests protect software quality.

Rapid recovery and reduced downtime

Frequent and targeted testing significantly reduces Mean Time To Repair (MTTR). If a recipe tastes off, testing each ingredient individually makes it easy to identify the problematic one swiftly.

Testing advanced deployment methods

E2E tests validate sophisticated deployment strategies like canary or blue-green deployments. They’re comparable to taste-testing new recipes with select diners before serving them to a broader audience.

Strategies for reliable E2E tests in cloud environments

Conducting E2E tests in the cloud is like performing a sensitive experiment outdoors where weather conditions (network latency, traffic spikes) constantly change.

Fighting flakiness in dynamic conditions

Cloud environments often introduce unpredictable elements, network latency, resource contention, and transient service issues. It’s similar to trying to have a detailed conversation in a loud environment; messages can easily be missed.

Robust test locators

Build your tests to find UI elements using multiple identifiers. If the primary path is blocked, alternate paths ensure your tests remain reliable. Think of it like knowing multiple routes home in case one road gets closed.

Intelligent automatic retries

Implement automatic retries for tests that intermittently fail due to transient issues. Just like retrying a phone call after a bad connection, automated retries ensure temporary problems don’t falsely indicate major faults.

Stability matters for operations

Flaky tests create unnecessary alerts, causing teams to lose confidence in their testing suite. SREs need reliable signals, like a fire alarm that only triggers for genuine fires, not burned toast.

Real-Life integration, an example of a QuickCart application

Imagine assembling a complex Lego model, verifying each piece as it’s added.

E-Commerce application scenario

Consider “QuickCart,” a hypothetical cloud-native e-commerce application with services for product catalog, user accounts, shopping cart, and order processing.

Critical user journey

An essential E2E scenario: a user logs in, searches products, adds one to the cart, and proceeds toward checkout. This represents a common user experience path.

CI/CD pipeline workflow

When a developer updates the Shopping Cart service:

The CI/CD pipeline automatically builds the service.
The E2E test suite runs the crucial “Add to Cart” test before deploying to staging.
Test results dictate the next steps:
- Pass: Change promoted to staging.
- Fail: Deployment halted; team immediately notified.

This ensures a broken cart never reaches customers.

Choosing the right tools and automation

Selecting testing tools is like equipping a kitchen: the right tools significantly ease the task.

Popular E2E frameworks

Tools such as Cypress, Selenium, Playwright, and Datadog Synthetics each bring unique strengths to the table, making it easier to choose one that fits your project’s specific needs. Cypress excels with developer experience, allowing quick test creation. Selenium is unbeatable for extensive cross-browser testing. Playwright offers rapid execution ideal for fast-paced environments. Datadog Synthetics integrates seamlessly into monitoring systems, swiftly identifying potential problems.

Smooth integration with CI/CD

These tools work well with CI/CD platforms like Jenkins, GitLab CI, GitHub Actions, or Azure DevOps, orchestrating your automated tests efficiently.

Configurable and adaptable

Adjusting tests between environments (dev, staging, prod) is as simple as tweaking a base recipe, with minimal effort, and maximum adaptability.

Enhanced observability and detailed reporting

Observability and detailed reporting are the navigational instruments of your testing universe. Tools like Prometheus, Grafana, Datadog, or New Relic highlight test failures and offer valuable context through logs, metrics, and traces. Effective observability reduces downtime and stress, transforming complex debugging from tedious guesswork into targeted, effective troubleshooting.

The path to continuous confidence

Embedding E2E tests into your cloud-native CI/CD pipeline is like learning to cook with cast iron pans. Initial skepticism and maintenance worries soon give way to reliably delicious outcomes. Quick feedback, fewer surprises, and less midnight stress transform software cycles into satisfying routines.

Great software doesn’t happen overnight, it’s carefully seasoned and consistently refined. Embrace these strategies, and software quality becomes not just attainable but deliciously predictable.

May 18, 2025 by Fernando SRE DevOps stuff SRE stuff

Podman the secure Daemonless Docker alternative

Podman has emerged as a prominent technology among DevOps professionals, system architects, and infrastructure teams, significantly influencing the way containers are managed and deployed. Podman, standing for “Pod Manager,” introduces a modern, secure, and efficient alternative to traditional container management approaches like Docker. It effectively addresses common challenges related to overhead, security, and scalability, making it a compelling choice for contemporary enterprises.

With the rapid adoption of cloud-native technologies and the widespread embrace of Kubernetes, Podman offers enhanced compatibility and seamless integration within these advanced ecosystems. Its intuitive, user-centric design simplifies workflows, enhances stability, and strengthens overall security, allowing organizations to confidently deploy and manage containers across various environments.

Core differences between Podman and Docker

Daemonless vs Daemon architecture

Docker relies on a centralized daemon, a persistent background service managing containers. The disadvantage here is clear: if this daemon encounters a failure, all containers could simultaneously go down, posing significant operational risks. Podman’s daemonless architecture addresses this problem effectively. Each container is treated as an independent, isolated process, significantly reducing potential points of failure and greatly improving the stability and resilience of containerized applications.

Additionally, Podman simplifies troubleshooting and debugging, as any issues are isolated within individual processes, not impacting an entire network of containers.

Rootless container execution

One notable advantage of Podman is its ability to execute containers without root privileges. Historically, Docker’s default required elevated permissions, increasing the potential risk of security breaches. Podman’s rootless capability enhances security, making it highly suitable for multi-user environments and regulated industries such as finance, healthcare, or government, where compliance with stringent security standards is critical.

This feature significantly simplifies audits, easing administrative efforts and substantially minimizing the potential for security breaches.

Performance and resource efficiency

Podman is designed to optimize resource efficiency. Unlike Docker’s continuously running daemon, Podman utilizes resources only during active container use. This targeted approach makes Podman particularly advantageous for edge computing scenarios, smaller servers, or continuous integration and delivery (CI/CD) pipelines, directly translating into cost savings and improved system performance.

Moreover, Podman supports organizations’ sustainability objectives by reducing unnecessary energy usage, contributing to environmentally conscious IT practices.

Flexible networking with CNI

Podman employs the Container Network Interface (CNI), a standard extensively used in Kubernetes deployments. While CNI might initially require more configuration effort than Docker’s built-in networking, its flexibility significantly eases the transition to Kubernetes-driven environments. This adaptability makes Podman highly valuable for organizations planning to migrate or expand their container orchestration strategies.

Compatibility and seamless transition from Docker

A key advantage of Podman is its robust compatibility with Docker images and command-line tools. Transitioning from Docker to Podman is typically straightforward, requiring minimal adjustments. This compatibility allows DevOps teams to retain familiar workflows and command structures, ensuring minimal disruption during migration.

Moreover, Podman fully supports Dockerfiles, providing a smooth transition path. Here’s a straightforward example demonstrating Dockerfile compatibility with Podman:

FROM alpine:latest

RUN apk update && apk add --no-cache curl

CMD ["curl", "--version"]

Building and running this container in Podman mirrors the Docker experience:

podman build -t myimage .
podman run myimage

This seamless compatibility underscores Podman’s commitment to a user-centric approach, prioritizing ease of transition and ongoing operational productivity.

Enhanced security capabilities

Podman offers additional built-in security enhancements beyond rootless execution. By integrating standard Linux security mechanisms such as SELinux, AppArmor, and seccomp profiles, Podman ensures robust container isolation, safeguarding against common vulnerabilities and exploits. This advanced security model simplifies compliance with rigorous security standards and significantly reduces the complexity of maintaining secure container environments.

These security capabilities also streamline security audits, enabling teams to identify and mitigate potential vulnerabilities proactively and efficiently.

Looking ahead with Podman

As container technology evolves rapidly, staying updated with innovative solutions like Podman is essential for DevOps and system architecture professionals. Podman addresses critical challenges associated with Docker, offering improved security, enhanced performance, and seamless Kubernetes compatibility.

Embracing Podman positions your organization strategically, equipping teams with superior tools for managing container workloads securely and efficiently. In the dynamic landscape of modern DevOps, adopting forward-thinking technologies such as Podman is key to sustained operational success and long-term growth.

Podman is more than an alternative—it’s the next logical step in the evolution of container technology, bringing greater reliability, security, and efficiency to your organization’s operations.

April 20, 2025 by Fernando SRE DevOps stuff Kubernetes Linux Stuff SRE stuff

Why enterprise DevOps initiatives fail and how to fix them

Getting DevOps right in large companies is tricky. It’s been around for nearly two decades, from developers wanting deployment control. It gained traction around 2011-2015, boosted by Gartner, SAFe, and AWS’s rise, pushing CIOs to learn from agile startups.

Despite this history, many DevOps initiatives stumble. Why? Often, the approach misses fundamental truths about making DevOps work in complex enterprises with multi-cloud setups, legacy systems, and pressure for faster results. Let’s explore common pitfalls and how to get back on track.

Thinking DevOps is just another IT project

This is crucial. DevOps isn’t just new tools or org charts; it’s a cultural shift. It’s about Dev, Ops, Sec, and the business working together smoothly, focused on customer value, agility, and stability.

Treating it like a typical project is like fixing a building’s crumbling foundation by painting the walls, you ignore the deep, structural changes needed. CIOs might focus narrowly on IT implementation, missing the vital cultural shift. Overlooking connections to customer value, security, scaling, and governance is easy but detrimental. Siloing DevOps leads to slower cycles and business disconnects.

How to Fix It: Ensure shared understanding of DevOps/Agile principles. Run workshops for Dev and Ops to map the value stream and find bottlenecks. Forge a shared vision balancing innovation speed and operational stability, the core DevOps tension.

Rushing continuous delivery without solid operations

The allure of CI/CD is strong, but pushing continuous deployment everywhere without robust operations is like building a race car without good brakes or steering, you might crash.

Not every app needs constant updates, nor do users always want them. Does the business grasp the cost of rigorous automated testing required for safe, frequent deployments? Do teams have the operational muscle: solid security, deep observability, mature AIOps, reliable rollbacks? Too often, we see teams compromise quality for speed.

The massive CrowdStrike outage is a stark reminder: pushing changes fast without sufficient safeguards is risky. To keep evolving… without breaking things, we need to test everything. Remember benchmarks: only 18% achieve elite performance (on-demand deploys, <5% failure, <1hr recovery); high performers deploy daily/weekly (<10% failure, <1 day recovery).

How to Fix It: Use a risk-based approach per application. For frequent deployments, demand rigorous testing, deep observability (using SRE principles like SLOs), canary releases, and clear Error Budgets.

Neglecting user and developer experiences

Focusing solely on automation pipelines forgets the humans involved: end-users and developers.

Feature flags, for instance, are often just used as on/off switches. They’re versatile tools for safer rollouts, A/B testing, and resilience, missing this potential is a loss.

Another pitfall: overloading developers by shifting too much infrastructure, testing, and security work “left” without proper support. This creates cognitive overload and kills productivity, imposing a “developer tax”, it’s unrealistic to expect developers to master everything.

How to Fix It: Discuss how DevOps practices impact people. Is the user experience good? Is the developer experience smooth, or are engineers drowning? Define clear roles. Consider a Platform Engineering team to provide self-service tools that reduce developer burden.

Letting tool choices run wild without standards

Empowering teams to choose tools is good, but complete freedom leads to chaos, like builders using incompatible materials. It creates technical debt and fragility.

Platform Engineering helps by providing reusable, self-service components (CI/CD, observability, etc.), creating “paved roads” with embedded standards. Most orgs now have platform teams, boosting productivity and quality. Focusing only on tools without solid architecture causes issues. “Automation can show quick wins… but poor architecture leads to operational headaches”.

How to Fix It: Balance team autonomy with clear standards via Platform Engineering or strong architectural guidance. Define tool adoption processes. Foster collaboration between DevOps, platform, architecture, and delivery teams on shared capabilities.

Expecting teams to magically handle risk

Shifting security “left” doesn’t automatically mean risks are managed effectively. Do teams have the time, expertise, and tools for proactive mitigation? Many orgs lack sufficient security support for all teams.

Thinking security is just managing vulnerability lists is reactive. True DevSecOps builds security in. Data security is also often overlooked, with severe consequences. AI code generation adds another layer requiring rigorous testing.

How to Fix It: Don’t just assume teams handle risk. Require risk mitigation and tech debt on roadmaps. Implement automated security testing, regular security reviews, and threat modeling. Define release management with risk checkpoints. Leverage SRE practices like production readiness reviews (PRRs).

The CIO staying Hands-Off until there’s a crisis

A fundamental mistake CIOs make is fully delegating DevOps and only getting involved during crises. Because DevOps often feels “in the weeds,” it tends to be pushed down the hierarchy. But DevOps is strategic, it’s about delivering value faster and more reliably.

Given DevOps’ evolution, expect varied interpretations. As a CIO, be proactively involved. Shape the culture, engage regularly (not just during crises), champion investments (platforms, training, SRE), and ensure alignment with business needs and risk tolerance.

How to Fix It: Engage early and consistently. Champion the culture shift. Ask about value delivery, risk management, and developer productivity. Sponsor platform/SRE teams. Ensure business alignment. Your active leadership is crucial.

Avoiding these pitfalls isn’t magic, DevOps is a continuous journey. But understanding these traps and focusing on culture, solid operations, user/developer experience, sensible standards, proactive risk management, and engaged leadership significantly boosts your chances of building a DevOps capability that delivers real business value.

April 19, 2025 by Fernando SRE DevOps stuff SRE stuff