Serverless

The sandbox finally gets its own apartment

There is a particular kind of optimism that appears in architecture diagrams. It is usually represented by a neat rectangle labelled “sandbox”, placed between “user code” and “production systems”, as if malicious input could be discouraged by tidy geometry.

For years, we have treated this rectangle with a level of trust usually reserved for elevators, bridges, and the one person in the meeting who says, “This should be simple.” Inside the box went user scripts, notebook cells, plugin code, CI jobs, vulnerability scanners, browser IDE sessions and, more recently, AI agents with the confidence of a junior developer and the impulse control of a caffeinated raccoon.

The problem was never running code. Computers are embarrassingly eager to do that. The problem was trusting the code not to bite the furniture.

AWS Lambda MicroVMs are interesting because they arrive at the exact moment when everyone is trying to run more untrusted code, more often, with fewer humans staring nervously at terminal output. They are not just another Lambda feature wearing a party hat. They are AWS trying to give each user, job, or agentic session its own small, isolated room, complete with VM-level walls, preserved state, and a serverless landlord who handles the plumbing.

That is worth paying attention to.

The old sandbox had suspicious drywall

Containers made modern platform engineering possible. They package applications beautifully, start quickly and behave well enough that we have built entire civilizations on top of them, some of which even survive Monday mornings.

But containers share the host kernel. That is not a moral failure. It is simply the bargain. You get speed and density, but the isolation boundary is not the same as a virtual machine boundary. For normal workloads, this is often a perfectly reasonable trade. For hostile or unknown code, it becomes a slightly more awkward dinner party.

Running untrusted code in containers usually requires extra hardening. Seccomp profiles, AppArmor or SELinux policies, rootless runtimes, careful filesystem restrictions, capability dropping, network controls, resource limits, image hygiene, and a collection of internal documents with titles like “please do not run customer code as root again”.

This does not mean containers are bad. It means they have spent years being asked to babysit workloads they would probably rather not meet in a dark data center.

Virtual machines have always had a stronger answer. Each workload gets a separate kernel and a more substantial isolation boundary. Security teams like that sort of thing. They sleep better when the boundary is not made entirely of namespace optimism.

The problem is that traditional VMs are heavier. They take longer to boot, require more lifecycle management and make interactive platforms feel like they are starting a small moon every time someone clicks “run”. If you are building a browser IDE, a notebook platform, or an AI coding assistant, waiting minutes for an execution environment is not a product experience. It is a hostage situation with a spinner.

So teams built their own sandboxes. Some used hardened containers. Some orchestrated pools of VMs. Some built elaborate custom schedulers. Many discovered that safely running other people’s code is less a feature and more a lifestyle choice involving pager alerts.

Firecracker was already doing the quiet work

The interesting part is that AWS already had a lightweight virtualization story hiding in plain sight. Firecracker has powered Lambda and Fargate-style isolation patterns for years. It was designed to provide lightweight MicroVMs with fast startup and strong boundaries, without dragging the full emotional luggage of traditional VM management into every request.

Lambda MicroVMs take that idea and expose it as a managed primitive for a very specific class of applications. These are not ordinary Lambda functions that got promoted. They are separate resources with a distinct lifecycle and a different purpose.

Regular Lambda functions are still excellent for event-driven work. A request arrives, a function runs, a result returns, and the function disappears back into the mist like a responsible woodland creature. That model works beautifully for stateless tasks.

Lambda MicroVMs are for the messier cases. A user opens an interactive coding environment. An AI agent needs to inspect files, run code, call tools, and maintain context. A data analytics session installs packages, loads working files, and waits while the human operator goes to lunch, a meeting, or an unexpected argument with Kubernetes.

These sessions need state. They need isolation. They need to start quickly. They need to pause without losing their belongings.

That combination used to be annoying. Now it has a first-class shape.

Lambda functions answer the doorbell, MicroVMs let the guest stay

The lifecycle is the real story.

You create a MicroVM image from a Dockerfile and a code artifact. AWS initializes the environment and captures a Firecracker snapshot of disk and memory. Later, when you launch a MicroVM from that image, it starts from the pre-initialized snapshot instead of beginning from scratch.

This is the difference between cooking dinner from raw ingredients and reheating yesterday’s lasagna. One is noble. The other is faster and, under the correct emotional conditions, superior.

Snapshots matter because many interactive environments spend a surprising amount of time getting ready to do the thing the user actually asked for. Installing packages, loading runtimes, preparing tools, arranging directories, initializing services, warming models, importing dependencies, and opening files. All of this can become part of the prepared environment.

For AI agents, this is especially useful. The environment can already contain the tools, SDKs, interpreters, and supporting processes the agent needs. When a session starts or resumes, it does not have to assemble itself from a cold and resentful universe.

There is a catch, because distributed systems are legally required to have at least one. Snapshotting state means engineers must think carefully about what is safe to preserve and clone. Unique identifiers, network connections, ephemeral credentials, time-sensitive tokens, and initialization-time randomness may all need special handling.

A snapshot is not just a performance trick. It is a memory of a machine. And memories, as anyone who has debugged a stale cache knows, can lie with great confidence.

Eight hours change the serverless conversation

The 8-hour runtime limit is one of the details that make this more than a faster function wrapper.

Traditional serverless thinking has usually pushed us toward short, stateless execution. Do a job, return a result, vanish. This is clean, scalable, and architecturally pleasing, which means real users immediately invent workflows that do not fit it.

Interactive sessions are not short. Vulnerability scans can run for a while. Data analysis can wander. AI agents can start with “inspect this repository” and somehow end up creating four temporary files, two package installs, and a philosophical dispute with a linter.

Lambda MicroVMs support longer-running sessions while preserving memory, disk, and process state. During idle periods, the environment can be suspended and later resumed. That matters because many workloads are not constantly busy. Humans pause. Agents wait. Browsers sleep. Developers get pulled into meetings where someone says “alignment” until the will to compute leaves the room.

Idle suspension changes the economics and the experience. The user can return to a session that still has its working state, but the platform does not have to pay as if the machine were actively sweating the entire time.

This is still serverless, but not the old cartoon version where everything is tiny, stateless, and over before the coffee cools. It is serverless with a memory.

That sentence would have terrified architects ten years ago. Now it may be exactly what agent platforms need.

AI agents need walls, not vibes

Agentic AI has made sandboxing fashionable in the same way kitchen fires make extinguishers fashionable.

An AI agent can generate code, run commands, inspect files, call APIs, and modify state. Sometimes it does this brilliantly. Sometimes it does this like a raccoon found a shell prompt and decided to modernize procurement.

The security model for agents must assume mistakes. It must assume prompt injection. It must assume unexpected tool use. It must assume that the agent is not evil, just helpful in the way a toddler with admin privileges might be helpful.

This is where per-user or per-session MicroVM isolation becomes important. If each agent run gets its own dedicated execution environment, the blast radius becomes easier to reason about. One confused agent should not be able to interfere with another user’s session. One malicious script should not get a friendly wave from a shared kernel and an invitation to explore the neighborhood.

But this is not magic safety dust. VM-level isolation is a stronger primitive, not a complete architecture.

You still need tight IAM roles. You still need network egress controls. You still need to decide whether the MicroVM can reach the internet, internal APIs, metadata services, package repositories or that one ancient endpoint nobody admits owns the business process. You still need secrets management, audit trails, logging, quotas, timeouts and cleanup.

The agent may now have its own padded serverless studio. You still need to check what tools you left in the drawers.

The security story is better, but not finished

The most dangerous sentence in cloud security is “the service handles that”.

Sometimes the service does handle that. Sometimes it handles one very important layer, and everything around it remains your problem, wearing a fake moustache.

Lambda MicroVMs improve the isolation layer. A dedicated MicroVM with no shared kernel between users is a meaningful architectural improvement for untrusted execution. It gives platform teams a stronger foundation than a basic container sandbox. It also reduces the amount of custom virtualization machinery teams need to build themselves.

But the surrounding design still decides whether the result is secure or just expensive theater.

Before production, architects should ask some unfashionable questions.

What can the MicroVM reach on the network?

What credentials exist inside the environment?

Can the workload call internal APIs?

Can it download arbitrary packages?

How is user code authenticated before execution?

How are sessions terminated?

What happens to files after suspension?

What gets written to logs?

How are snapshots built, versioned, and retired?

How are malicious or runaway workloads detected?

What prevents one customer from turning your lovely sandbox platform into a cryptocurrency-themed space heater?

These questions are not reasons to avoid the service. These are the reasons to use it properly. The stronger the primitive, the more tempting it is to stop thinking. Resist this. Architecture diagrams are very good at looking innocent.

The bill is also part of the architecture

Serverless has trained us to associate managed services with financial elegance. Sometimes this is true. Sometimes the invoice arrives carrying a clipboard and asking to speak with the adult who designed the retry policy.

Lambda MicroVMs have a cost model that deserves actual modeling. You pay for baseline resources while a MicroVM is running, and for additional active usage when the workload exceeds that baseline. That is reasonable, but it means usage patterns matter.

A short-lived scan that starts, does work, and terminates has one shape. An interactive coding session that runs, idles, suspends, resumes, and idles again has another. A swarm of AI agents that each receive their own environment has a third shape, one that may require finance to breathe into a paper bag if nobody adds quotas.

The point is not that this will be expensive. The point is that “serverless” does not automatically mean “cheap”. It means the operational model changes. Billing still obeys physics, usage, and the quiet vengeance of concurrency.

If you are building a platform on top of Lambda MicroVMs, model the boring details. Average session duration. Idle time. Resume frequency. Memory size. CPU requirements. Disk usage. Regional availability. Expected concurrency. Abuse scenarios. Free-tier fantasies, if your product team has recently been exposed to those.

Cloud bills are where optimistic architecture diagrams go to receive adult supervision.

Platform engineering gets a new building block

For platform teams, the most promising part of Lambda MicroVMs is not that they make one workload easier. It is that they could simplify a recurring pattern.

Many organizations need a safe way to run untrusted or semi-trusted code. Internal developer platforms need execution sandboxes. AI products need agent workspaces. Security platforms need isolated scanning environments. Data platforms need notebook-like sessions. Education platforms need per-student runtimes. CI systems need disposable workers.

Historically, building this well meant assembling a small private cloud inside your cloud. VM pools, schedulers, cleanup jobs, image builders, network isolation, capacity management, logging pipelines, and emergency scripts named things like final_cleanup_v7_really_final.sh.

Lambda MicroVMs do not remove all platform work. They move the boundary.

You still need golden images. You still need lifecycle policies. You still need guardrails. You still need quotas and tenant mapping. You still need observability. You still need deployment pipelines. You still need to decide how MicroVMs integrate with Lambda functions, S3, CloudWatch, IAM, and whatever control plane runs your product.

But you may not need to own as much of the virtualization substrate yourself. That is not glamorous. It is better. Glamour is what vendors put in keynote slides before someone asks about day two operations.

A sensible pattern might use regular Lambda functions for the event-driven control plane, APIs, and orchestration logic. Lambda MicroVMs handle the isolated execution path, where user code or AI-generated code actually runs. S3 stores build artifacts. CloudWatch captures logs. IAM defines what each environment can do. Network controls define what each environment can reach.

That division feels clean. Not simple, because nothing involving untrusted code is simple. But clean enough to be useful.

The uncomfortable production checklist

Before adopting Lambda MicroVMs, it is worth being deliberately boring. Boring is underrated in architecture. Boring systems sleep through the night.

Ask which workloads actually need VM-level isolation. Some do. Some are perfectly fine as regular Lambda functions, ECS tasks, or containers in Kubernetes. Using a stronger primitive everywhere can become expensive and operationally awkward.

Ask how long sessions really run. Not how long the demo runs. Not how long the product manager hopes they run. How long do users actually keep them alive while opening seven browser tabs and going to lunch?

Ask what state must survive suspension. Then ask what state must never survive suspension. Secrets, temporary credentials, one-time tokens, and sensitive intermediate files need careful treatment.

Ask how images are built and patched. A MicroVM image is still software. It needs supply chain controls, vulnerability scanning, versioning, and retirement. Yesterday’s golden image can become tomorrow’s antique security incident.

Ask what happens when a workload wants more. More CPU. More memory. More disk. GPU acceleration. x86 compatibility. A different region. A longer session. Some requests will fit the service. Some will not. That is not a failure. That is architecture doing its job before production does it for you with alarms.

Ask how abuse is stopped. Per-user quotas, concurrency limits, session expiration, network restrictions, and cost controls should be designed early. Waiting until after the first runaway agent is a traditional learning method, but not a relaxing one.

A better sandbox is not a finished safety plan

Lambda MicroVMs arrive at a useful moment. The industry is asking applications to execute more code that the application developer did not write. Some of that code comes from users. Some comes from AI systems. Some comes from a collaboration between both, which is how you get an incident report with a surprisingly emotional timeline.

The old choices were uncomfortable. Containers were fast but needed serious hardening for hostile workloads. Traditional VMs were safer but heavy. Regular serverless functions were elegant but not designed for long, stateful, interactive sessions.

Lambda MicroVMs offer a new middle ground. VM-level isolation. Fast launch and resume from snapshots. Stateful sessions. Idle suspension. Serverless lifecycle management. A dedicated environment per user, job, or agentic session.

That combination will not remove the need for security architecture. It will not absolve anyone from IAM, networking, secrets, observability, patching, or cost controls. It will not make AI agents safe by virtue of being placed inside a branded rectangle.

But it gives the sandbox real walls.

And that matters. Because the sandbox did not become safer because someone wrote “AI” on the diagram. It became safer because the walls got real.

June 28, 2026 by Cloud stuff DevOps stuff SRE stuff

AWS architecture choices I would not repeat

I was holding a lukewarm Americano in my left hand and a lukewarm sense of dread in my right when the Slack notifications started arriving. It was one of those golden hour afternoons where the light hits your monitor at exactly the wrong angle, turning your screen into a mirror that reflects your own panic back at you. CloudWatch was screaming. Not the dignified beep of a minor alert, but the full banshee wail of latency charts gone vertical.

My coffee had developed that particular skin on top that lukewarm coffee gets when you have forgotten it exists. I stared at the graph. Our system, which I had personally architected with the confidence of a man who had read half a documentation page, was melting in real time. The app was not even big. We had fewer concurrent users than a mid-sized bowling league, yet there we were. Throttling errors stacked up like dirty dishes in a shared apartment kitchen. Cold starts multiplied like rabbits on a vitamin regimen. Costs were rising faster than my blood pressure, which at that moment could have powered a small turbine.

That afternoon changed how I design systems. After four years of writing Python and just enough AWS experience to be dangerous, I learned the cardinal rule. Most architectures that look elegant at small scale are just disasters wearing tuxedos. Here is how I built a Rube Goldberg machine of regret, and how I eventually stopped lighting my own infrastructure on fire.

The Godzilla Lambda and the art of overeating

At first, it felt elegant. One Lambda function to handle everything. Image resizing, email sending, report generation, user authentication, and probably the kitchen sink if I had thought to attach plumbing. One deployment. One mental model. One massive mistake.

I called it my Swiss Army knife approach. Except this particular knife weighed eighty pounds and required three weeks’ notice to open. The function had more conditional branches than a family tree in a soap opera. If the event type was ‘resize_image’, it did one thing. If it was ‘send_email’, it did another. It was essentially a diner where the chef was also the waiter, the dishwasher, and the person who had to physically restrain customers who complained about the meatloaf.

The cold starts were spectacular. My function would wake up slower than a teenager on a Monday morning after an all-night gaming session. It dragged itself into consciousness, looked around, and slowly remembered it had responsibilities. Deployments became existential gambles. Change a comma in the email formatting logic, and you risk taking down the image processing pipeline that paying customers actually cared about. Logs turned into a crime scene where every suspect had the same fingerprint.

The automation scripts I had written to manage this beast were just duct tape on top of more duct tape. They had to account for the fact that the entry point was a fragile monolith masquerading as serverless elegance.

Now I build small, single-purpose functions. Each one does exactly one thing, like a very boring but highly reliable employee. My resize handler resizes. My email handler emails. They do not mingle. They do not gossip. They do not share IAM policies at the same coffee station.

Here is the only snippet of code you need to see today, mostly because it is so short it could fit in a tweet from someone with a short attention span.

def handler(event, context):
    return process_invoice(event.get("invoice_id"))

That is it. No if statements doing interpretive dance. No switch cases having an identity crisis. If a Lambda needs more than one IAM policy, it is already too big. It is like needing two different keys to open your refrigerator. If that is the case, you have designed a refrigerator incorrectly.

Using HTTP to check the mailbox

API Gateway is powerful. It is also expensive, verbose, and absolutely overkill for workflows where no human is holding a browser. I learned this the day I decided to route every single background job through API Gateway because I valued consistency over solvency. My AWS bill arrived looking like a phone number. A long one.

I was using HTTP requests for internal automation. Let that sink in. I was essentially hiring a limousine to drive across the street to check my mailbox. Every time a background job needed to trigger another background job, it went through API Gateway. That meant authentication layers, request validation, and pricing tiers designed for enterprise traffic handling, my little cron job that cleaned up temporary files.

Debugging was a nightmare wrapped in an OAuth flow. I spent three hours one Tuesday trying to figure out why an internal service could not authenticate, only to realize I had designed a system where my left hand needed to show my right hand three forms of government ID just to borrow a stapler.

The fix was to remember that computers can talk to each other without pretending to be web browsers. I switched to event-driven architecture using SNS and SQS. Now my producers throw messages into a queue like dropping letters into a mailbox, and they do not care who picks them up. The consumers grab what they need when they are ready.

sns_client = boto3.client("sns")
sns_client.publish(
    TopicArn=REPORT_GENERATION_TOPIC,
    Message=json.dumps({"customer_id": "CUST-8842", "report_type": "quarterly"})
)

The producers have no idea who consumes the message. They do not need to know. It is like leaving a note on the fridge instead of calling your roommate on their cell phone every time you need to tell them the milk is sour. If humans are not calling the endpoint, it probably should not be HTTP. Save your API Gateway budget for something that actually faces the internet, like that side project you will never finish.

The Server with amnesia

This one still stings. I used to run cron jobs on EC2 instances. Backups, cleanup scripts, data pipelines, all scheduled on a server that I treated like a reliable employee instead of the forgetful intern it actually was.

It worked perfectly until the instance restarted. Which instances do. They reboot for maintenance, for updates, for mysterious AWS reasons that arrive in emails written in that particular corporate tone that suggests everything is fine while your world burns. Every time the server came back up, it had the memory of a goldfish with a head injury. Scheduled jobs vanished into the ether. Backups did not happen. Cleanup scripts sat idle while storage costs climbed.

I spent three mornings a week SSHing into instances like a nervous parent checking if a sleeping teenager is still breathing. I would type crontab -l with the same trepidation one might use when opening a credit card statement after a vacation. Is everything there? Did it forget? Is the database backup running, or am I going to explain to the CEO why our disaster recovery plan is actually just a disaster?

If your automation depends on a server staying alive, it is not automation. It is hope dressed up in a shell script.

I replaced it with EventBridge and Lambda. EventBridge does not forget. EventBridge does not take vacations. EventBridge does not require you to log in at 3 AM in your pajamas to check if it is still breathing. It triggers the function, the function does the work, and if something breaks, it either retries or sends a message to a dead letter queue where you can ignore it at your leisure during business hours.

Trusting the Database to save itself

I trusted RDS autoscaling because the documentation made it sound intelligent. Like having a butler who watches your dinner party and quietly brings more chairs when guests arrive. The reality was more like having a butler who stands in the corner watching the house catch fire, then asks if you would like a chair.

The database would hit a traffic spike. Connections would pile up like shoppers at a Black Friday doorbuster sale. The application layer would be perfectly healthy, humming along, wondering why the database was on fire. By the time RDS autoscaling decided to add capacity, the damage was done. The connection pool had already exhausted itself. Automation scripts designed to recover the situation could not even connect to run their recovery logic. It was like calling the fire department only to find out they start driving when they smell smoke, not when the alarm rings.

Now I automate predictive scaling. It is not fancy. It is just intentional. I have scripts that check expected connection loads against current capacity. If we are going to hit five hundred connections, the script starts warming up a larger instance class before we need it. It is like preheating an oven instead of shoving a turkey into a cold metal box and hoping for the best.

AWS gives you primitives. Architecture is deciding when not to trust the defaults, because the defaults are designed to keep AWS running, not to keep you sane.

Reading tea leaves in a hurricane

I once thought centralized logging meant dumping everything into CloudWatch and calling it observability. This is the equivalent of shoveling all your mail into a closet and claiming you have a filing system. Technically true, practically useless.

My automation depended on parsing these logs. I wrote regex patterns that looked like ancient Sumerian curses. They would match error messages sometimes, ignore them other times, and occasionally trigger alerts on completely irrelevant noise because someone had logged the word error in a debugging statement about their lunch order.

During incidents, I would stare at these logs trying to find patterns. It was like trying to identify a specific scream in a horror movie marathon. Everything was urgent. Nothing was actionable. My scripts could not tell the difference between a critical database failure and a debug message about cache expiration. They were essentially reading entrails.

Structured logs saved my sanity. Now everything gets dumped as JSON with actual fields. Event types, durations, identifiers, all labeled and searchable. My automation can trigger follow-up jobs when specific events complete. It can detect anomalies by looking at actual numeric fields instead of trying to parse human-readable text like some kind of desperate fortune teller.

logger.info(
    "task_completed",
    extra={
        "job_type": "inventory_sync",
        "warehouse_id": "WH-15",
        "duration_ms": 1420,
        "items_processed": 847
    }
)

Logs are not for humans anymore. They are for systems. Humans should read dashboards. Systems should read logs. Confuse the two, and you end up with alerts that cry wolf at 3 AM because someone spelled success wrong.

The quiet killer wearing a price tag

This is the one that really hurts. Everything worked. Latency was acceptable. Automation was smooth. The system scaled. Then the bill arrived, and I nearly spilled my coffee onto the keyboard. If cost is not part of your architecture, scale will punish you like a gym teacher who has decided you need motivation.

I had built something that scaled technically but not financially. It was like designing an airplane that flies beautifully but requires fuel that costs more than the GDP of a small nation. Every request through API Gateway, every idle EC2 waiting for a cron job that might not come, every poorly optimized Lambda running for fifteen seconds because I had not bothered to trim the dependencies, it all added up.

Now I automate cost checks. Before expensive jobs run, they estimate their impact. If the daily budget threshold approaches, the system starts making choices. It defers non-critical tasks. It sends warnings. It acts like a responsible adult at a bar when the tab starts getting too high.

def should_process_batch(estimated_cost, daily_spend):
    remaining_budget = DAILY_LIMIT - daily_spend
    return estimated_cost < (remaining_budget * 0.8)

Simple guardrails save real money. There is a saying I keep taped to my monitor now. If it scales technically but not financially, it does not scale. It is just a very efficient way to go bankrupt.

The art of rehearsed failure

Every bad decision I made had the same DNA. I optimized for speed of development. I ignored the longevity of automation. I trusted defaults because reading the full documentation seemed like work for people who had more time than I did. I treated AWS like a magic wand instead of a very powerful, very expensive tool that requires respect.

Good architecture is not about services. It is about failure modes you have already rehearsed in your head. It is about assuming you will forget what you built in six months, because you will. It is about assuming growth will happen, failure will happen, and at some point, you will be trying to debug this thing while your phone buzzes with angry messages from people who just want the system to work.

Build like you are designing a kitchen for a very forgetful, very busy chef who might be slightly drunk. Label everything. Make the dangerous stuff hard to do by accident. Keep the receipts. And for the love of all that is holy, do not put cron jobs on EC2.

January 30, 2026 by Cloud stuff

Let IAM handle the secrets you can avoid

There are two kinds of secrets in cloud security.

The first kind is the legitimate kind: a third-party API token, a password for something you do not control, a certificate you cannot simply wish into existence.

The second kind is the kind we invent because we are in a hurry: long-lived access keys, copied into a config file, then copied into a Docker image, then copied into a ticket, then copied into the attacker’s weekend plans.

This article is about refusing to participate in that second category.

Not because secrets are evil. Because static credentials are the “spare house key under the flowerpot” of AWS. Convenient, popular, and a little too generous with access for something that can be photographed.

The goal is not “no secrets exist.” The goal is no secrets live in code, in images, or in long-lived credentials.

If you do that, your security posture stops depending on perfect human behavior, which is great because humans are famously inconsistent. (We cannot all be trusted with a jar of cookies, and we definitely cannot all be trusted with production AWS keys.)

Why this works in real life

AWS already has a mechanism designed to prevent your applications from holding permanent credentials: IAM roles and temporary credentials (STS).

When your Lambda runs with an execution role, AWS hands it short-lived credentials automatically. They rotate on their own. There is nothing to copy, nothing to stash, nothing to rotate in a spreadsheet named FINAL-final-rotation-plan.xlsx.

What remains are the unavoidable secrets, usually tied to systems outside AWS. For those, you store them in AWS Secrets Manager and retrieve them at runtime. Not at build time. Not at deploy time. Not by pasting them into an environment variable and calling it “secure” because you used uppercase letters.

This gives you a practical split:

  • Avoidable secrets are replaced by IAM roles and temporary credentials
  • Unavoidable secrets go into Secrets Manager, encrypted and tightly scoped

The architecture in one picture

A simple flow to keep in mind:

  1. A Lambda function runs with an IAM execution role
  2. The function fetches one third-party API key from Secrets Manager at runtime
  3. The function calls the third-party API and writes results to DynamoDB
  4. Network access to Secrets Manager stays private through a VPC interface endpoint (when the Lambda runs in a VPC)

The best part is what you do not see.

No access keys. No “temporary” keys that have been temporary since 2021. No secrets baked into ZIPs or container layers.

What this protects you from

This pattern is not a magic spell. It is a seatbelt.

It helps reduce the chance of:

  • Credentials leaking through Git history, build logs, tickets, screenshots, or well-meaning copy-paste
  • Forgotten key rotation schedules that quietly become “never.”
  • Overpowered policies that turn a small bug into a full account cleanup
  • Unnecessary public internet paths for sensitive AWS API calls

Now let’s build it, step by step, with code snippets that are intentionally sanitized.

Step 1 build an IAM execution role with tight policies

The execution role is the front door key your Lambda carries.

If you give it access to everything, it will eventually use that access, if only because your future self will forget why it was there and leave it in place “just in case.”

Keep it boring. Keep it small.

Here is an example IAM policy for a Lambda that only needs to:

  • write to one DynamoDB table
  • read one secret from Secrets Manager
  • decrypt using one KMS key (optional, depending on how you configure encryption)
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WriteToOneTable",
      "Effect": "Allow",
      "Action": [
        "dynamodb:PutItem",
        "dynamodb:UpdateItem"
      ],
      "Resource": "arn:aws:dynamodb:eu-west-1:111122223333:table/app-results-prod"
    },
    {
      "Sid": "ReadOneSecret",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:thirdparty/weather-api-key-*"
    },
    {
      "Sid": "DecryptOnlyThatKey",
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt"
      ],
      "Resource": "arn:aws:kms:eu-west-1:111122223333:key/12345678-90ab-cdef-1234-567890abcdef",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "secretsmanager.eu-west-1.amazonaws.com"
        }
      }
    }
  ]
}

A few notes that save you from future regret:

  • The secret ARN ends with -* because Secrets Manager appends a random suffix.
  • The KMS condition helps ensure the key is used only through Secrets Manager, not as a general-purpose decryption service.
  • You can skip the explicit kms:Decrypt statement if you use the AWS-managed key and accept the default behavior, but customer-managed keys are common in regulated environments.

Step 2 store the unavoidable secret properly

Secrets Manager is not a place to dump everything. It is a place to store what you truly cannot avoid.

A third-party API key is a perfect example because IAM cannot replace it. AWS cannot assume a role in someone else’s SaaS.

Use a JSON secret so you can extend it later without creating a new secret every time you add a field.

{
  "api_key": "REDACTED-EXAMPLE-TOKEN"
}

If you like the CLI (and I do, because buttons are too easy to misclick), create the secret like this:

aws secretsmanager create-secret \
  --name "thirdparty/weather-api-key" \
  --description "Token for the Weatherly API used by the ingestion Lambda" \
  --secret-string '{"api_key":"REDACTED-EXAMPLE-TOKEN"}' \
  --region eu-west-1

Then configure:

  • encryption with a customer-managed KMS key if required
  • rotation if the provider supports it (rotation is amazing when it is real, and decorative when the vendor does not allow it)

If the vendor does not support rotation, you still benefit from central storage, access control, audit logging, and removing the secret from code.

Step 3 lock down secret access with a resource policy

Identity-based policies on the Lambda role are necessary, but resource policies are a nice extra lock.

Think of it like this: your role policy is the key. The resource policy is the bouncer who checks the wristband.

Here is a resource policy that allows only one role to read the secret.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowOnlyIngestionRole",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/lambda-ingestion-prod"
      },
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*"
    },
    {
      "Sid": "DenyEverythingElse",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:PrincipalArn": "arn:aws:iam::111122223333:role/lambda-ingestion-prod"
        }
      }
    }
  ]
}

This is intentionally strict. Strict is good. Strict is how you avoid writing apology emails.

Step 4 keep Secrets Manager traffic private with a VPC endpoint

If your Lambda runs inside a VPC, it will not automatically have internet access. That is often the point.

In that case, you do not want the function reaching Secrets Manager through a NAT gateway if you can avoid it. NAT works, but it is like walking your valuables through a crowded shopping mall because the back door is locked.

Use an interface VPC endpoint for Secrets Manager.

Here is a Terraform example (sanitized) that creates the endpoint and limits access using a dedicated security group.

resource "aws_security_group" "secrets_endpoint_sg" {
  name        = "secrets-endpoint-sg"
  description = "Allow HTTPS from Lambda to Secrets Manager endpoint"
  vpc_id      = aws_vpc.main.id

  ingress {
    from_port       = 443
    to_port         = 443
    protocol        = "tcp"
    security_groups = [aws_security_group.lambda_sg.id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_vpc_endpoint" "secretsmanager" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.eu-west-1.secretsmanager"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = [aws_subnet.private_a.id, aws_subnet.private_b.id]
  private_dns_enabled = true
  security_group_ids  = [aws_security_group.secrets_endpoint_sg.id]
}

If your Lambda is not in a VPC, you do not need this step. The function will reach Secrets Manager over AWS’s managed network path by default.

If you want to go further, consider adding a DynamoDB gateway endpoint too, so your function can write to DynamoDB without touching the public internet.

Step 5 retrieve the secret at runtime without turning logs into a confession

This is where many teams accidentally reinvent the problem.

They remove the secret from the code, then log it. Or they put it in an environment variable because “it is not in the repository,” which is a bit like saying “the spare key is not under the flowerpot, it is under the welcome mat.”

The clean approach is:

  • store only the secret name (not the secret value) as configuration
  • retrieve the value at runtime
  • cache it briefly to reduce calls and latency
  • never print it, even when debugging, especially when debugging

Here is a Python example for AWS Lambda with a tiny TTL cache.

import json
import os
import time
import boto3

_secrets_client = boto3.client("secretsmanager")
_cached_value = None
_cached_until = 0

SECRET_ID = os.getenv("THIRDPARTY_SECRET_ID", "thirdparty/weather-api-key")
CACHE_TTL_SECONDS = int(os.getenv("SECRET_CACHE_TTL_SECONDS", "300"))


def _get_api_key() -> str:
    global _cached_value, _cached_until

    now = int(time.time())
    if _cached_value and now < _cached_until:
        return _cached_value

    resp = _secrets_client.get_secret_value(SecretId=SECRET_ID)
    payload = json.loads(resp["SecretString"])

    api_key = payload["api_key"]
    _cached_value = api_key
    _cached_until = now + CACHE_TTL_SECONDS
    return api_key


def lambda_handler(event, context):
    api_key = _get_api_key()

    # Use the key without ever logging it
    results = call_weatherly_api(api_key=api_key, city=event.get("city", "Seville"))

    write_to_dynamodb(results)

    return {
        "status": "ok",
        "items": len(results) if hasattr(results, "__len__") else 1
    }

This snippet is intentionally short. The important part is the pattern:

  • minimal secret access
  • controlled cache
  • zero secret output

If you prefer a library, AWS provides a Secrets Manager caching client for some runtimes, and AWS Lambda Powertools can help with structured logging. Use them if they fit your stack.

Step 6 make security noisy with logs and alarms

Security without visibility is just hope with a nicer font.

At a minimum:

  • enable CloudTrail in the account
  • ensure Secrets Manager events are captured
  • alert on unusual secret access patterns

A simple and practical approach is a CloudWatch metric filter for GetSecretValue events coming from unexpected principals. Another is to build a dashboard showing:

  • Lambda errors
  • Secrets Manager throttles
  • sudden spikes in secret reads

Here is a tiny Terraform example that keeps your Lambda logs from living forever (because storage is forever, but your attention span is not).

resource "aws_cloudwatch_log_group" "lambda_logs" {
  name              = "/aws/lambda/lambda-ingestion-prod"
  retention_in_days = 14
}

Also consider:

  • IAM Access Analyzer to spot risky resource policies
  • AWS Config rules or guardrails if your organization uses them
  • an alarm on unexpected NAT data processing if you intended to keep traffic private

Common mistakes I have made, so you do not have to

I am listing these because I have either done them personally or watched them happen in slow motion.

  1. Using a wildcard secret policy
    secretsmanager:GetSecretValue on * feels convenient until it is a breach multiplier.
  2. Putting secret values into environment variables
    Environment variables are not evil, but they are easy to leak through debugging, dumps, tooling, or careless logging. Store secret names there, not secret contents.
  3. Retrieving secrets at build time
    Build logs live forever in the places you forget to clean. Runtime retrieval keeps secrets out of build systems.
  4. Logging too much while debugging
    The fastest way to leak a secret is to print it “just once.” It will not be just once.
  5. Skipping the endpoint and relying on NAT by accident
    The NAT gateway is not evil either. It is just an expensive and unnecessary hallway if a private door exists.

A two minute checklist you can steal

  • Your Lambda uses an IAM execution role, not access keys
  • The role policy scopes Secrets Manager access to one secret ARN pattern
  • The secret has a resource policy that only allows the expected role
  • Secrets are encrypted with KMS when required
  • The secret value is never stored in code, images, build logs, or environment variables
  • If Lambda runs in a VPC, you use an interface VPC endpoint for Secrets Manager
  • You have CloudTrail enabled and you can answer “who accessed this secret” without guessing

Extra thoughts

If you remove long-lived credentials from your applications, you remove an entire class of problems.

You stop rotating keys that should never have existed in the first place.

You stop pretending that “we will remember to clean it up later” is a security strategy.

And you get a calmer life, which is underrated in engineering.

Let IAM handle the secrets you can avoid.

Then let Secrets Manager handle the secrets you cannot.

And let your code do what it was meant to do: process data, not babysit keys like they are a toddler holding a permanent marker.

December 14, 2025 by Cloud stuff DevOps stuff

AWS Lambda SQS provisioned mode is cheaper than therapy

There is a specific flavor of nausea reserved for serverless engineering teams. It usually strikes at 2 a.m., shortly after a major product launch, when someone posts a triumphant screenshot of user traffic in Slack. While the marketing team is virtually high-fiving, CloudWatch quietly begins to draw a perfect, vertical line that looks less like a growth chart and more like a cliff edge.

Your SQS queues swell. Lambda invocations crawl. Suddenly, the phrase “fully managed service” sounds less comforting and more like a cruel punchline delivered by a distant cloud provider.

For years, the relationship between Amazon SQS and AWS Lambda has been the backbone of event-driven architecture. You wire up an event source mapping, let Lambda poll the queue, and trust the system to scale as messages arrive. Most days, this works beautifully. On the wrong day, under the wrong kind of spike, it works “eventually.”

But in the world of high-frequency trading or flash sales, “eventually” is just a polite synonym for “too late.”

With the release of AWS Lambda SQS Provisioned Mode on November 14, Amazon is finally admitting that sometimes magic is too slow. It grants you explicit control over the invisible workers that poll SQS for your function. It ensures they are already awake, caffeinated, and standing in line before the mob shows up. It allows you to trade a bit of extra planning (and money) for the guarantee that your system won’t hit the snooze button while your backlog turns into a towering monument to failure.

The uncomfortable truth about standard SQS polling

To understand why we need Provisioned Mode, we have to look at the somewhat lazy nature of the standard behavior.

Out of the box, Lambda uses an event source mapping to poll SQS on your behalf. You give it a queue and some basic configuration, and Lambda spins up pollers to check for work. You never see these pollers. They are the ghosts in the machine.

The problem with ghosts is that they are not particularly urgent. When a massive spike hits your queue, Lambda realizes it needs more pollers and more concurrent function invocations. However, it does not do this instantly. It ramps up. It adds capacity in increments, like a cautious driver merging onto a freeway.

For a steady workload, you will never notice this ramp-up. But during a viral marketing campaign or a market crash, those minutes of warming up feel like an eternity. You are essentially watching a barista who refuses to start grinding coffee beans until the line of customers has already curled around the block.

Standard SQS polling gives you tools like batch size, but it denies you direct influence over the urgency of the consumption. You cannot tell the system, “I need ten workers ready right now.” You can only stand in line and hope the algorithm notices you are drowning.

This is acceptable for background jobs like resizing images or sending emails. It is decidedly less acceptable for payment processing or fraud detection. In those cases, watching twenty thousand messages pile up while your system “automatically scales” is not an architectural feature. It is a resume-generating event.

Paying for a standing army instead of volunteers

Provisioned Mode flips the script on this reactive behavior. Instead of letting Lambda decide how many pollers to use based purely on demand, you tell it the minimum and maximum number of event pollers you want reserved for that queue.

An event poller is a dedicated worker that reads from SQS and hands batches of messages to your function. In standard mode, these pollers are summoned from a shared pool when needed. In Provisioned Mode, you are paying to keep them on retainer.

Think of it as the difference between calling a ride-share service and hiring a private driver to sit in your driveway with the engine running. One is efficient for the general public; the other is necessary if you need to leave the house in exactly three seconds.

The benefits are stark when translated into human terms.

First, you get speed. AWS advertises significantly faster scaling for SQS event source mappings in Provisioned Mode. We are talking about adding up to one thousand new concurrent invocations per minute.

Second, you get capacity. Provisioned Mode can support massive concurrency per SQS mapping, far higher than the default capabilities.

Third, and perhaps most importantly, you get predictability. A single poller is not just a warm body. It is a unit of throughput (handling up to 1 MB per second or 10 concurrent invokes). By setting a minimum number of pollers, you are mathematically guaranteeing a baseline of throughput. You are no longer hoping the waiters show up; you have paid their salaries in advance.

Configuring this without losing your mind

The good news is that Provisioned Mode is not a new service with its own terrifying learning curve. It is just a configuration toggle on the event source mapping you are already using. You can set it up in the AWS Console, the CLI, or your Infrastructure as Code tool of choice.

The interface asks for two numbers, and this is where the engineering art form comes in.

First, it asks for Minimum Pollers. This is the number of workers you always want ready.

Second, it asks for Maximum Pollers. This is the ceiling, the limit you set to ensure you do not accidentally DDoS your own database.

Choosing these numbers feels a bit like gambling, but there is a logic to it. For the minimum, pick a number that comfortably handles your typical traffic plus a standard spike. Start small. Setting this to 100 when you usually need 2 is the serverless equivalent of buying a school bus to commute to work alone.

For the maximum, look at your downstream systems. There is no point in setting a maximum that allows 5,000 concurrent Lambda functions if your relational database curls into a fetal position at 500 connections.

Once you enable it, you need to watch your metrics. Keep an eye on “Queue Depth” and “Age of Oldest Message.” If the backlog clears too slowly, buy more pollers. If your database administrator starts sending you angry emails in all caps, reduce the maximum. The goal is not perfection on day one; it is to replace guesswork with a feedback loop.

The financial hangover

Nothing in life is free, and this applies doubly to AWS features that solve headaches.

When you enable Provisioned Mode, AWS begins charging you for “Event Poller Units.” You pay for the minimum pollers you configure, regardless of whether there are messages in the queue. You are paying for readiness.

This is a mental shift for serverless purists. The whole promise of serverless was “pay for what you use.” Provisioned Mode is “pay for what you might need.”

You are essentially renting a standing army. Most of the time, they will just stand there, playing cards and eating your budget. But when the enemy (traffic) attacks, they are already in position. Standard SQS polling is cheaper because it relies on volunteers. Volunteers are free, but they take a while to put on their boots.

From a FinOps perspective, or simply from the perspective of explaining the bill to your boss, the question is not “Is this expensive?” The question is “What is the cost of latency?”

For a background report generator, a five-minute delay costs nothing. For a high-frequency trading platform, a five-second delay costs everything. You should not enable Provisioned Mode on every queue in your account. That would be financial malpractice. You reserve it for the critical paths, the workflows where the price of slowness is measured in lost customers rather than just infrastructure dollars.

Why you should care about the fourth dial

Architecturally, Provisioned Mode gives us a new layer of control. Previously, we had three main dials in event-driven systems: how fast we write to the queue, how fast the consumers process messages, and how much concurrency Lambda is allowed.

Provisioned Mode adds a fourth dial: the aggression of the retrieval.

It allows you to reason about your system deterministically. If you know that one poller provides X amount of throughput, you can stack them to meet a specific Service Level Agreement. It turns a “best effort” system into a “calculated guarantee” system.

Serverless was sold to us as freedom from capacity planning. We were told we could just write code and let the cloud handle the undignified details of scaling. For many workloads, that promise holds true.

But as your workloads become more critical, you discover the uncomfortable corners where “just let it scale” is not enough. Latency budgets shrink. Compliance rules tighten. Customers grow less patient.

AWS Lambda SQS Provisioned Mode is a small, targeted answer to that discomfort. It allows you to say, “I want at least this much readiness,” and have the platform respect that wish, even when your traffic behaves like a toddler on a sugar high.

So, pick your most critical queue. The one that keeps you awake at night. Enable Provisioned Mode, set a modest minimum, and watch the metrics. Your future self, staring at a flat latency graph during the next Black Friday, will be grateful you decided to stop trusting in magic and started paying for physics.

November 20, 2025 by Cloud stuff DevOps stuff SRE stuff

Escaping the AWS NAT Gateway toll booth

My coffee went cold. I was staring at my AWS bill, and one line item was staring back at me with a judgmental smirk: NAT Gateway: 33,01 €.

This wasn’t for compute. This wasn’t for storing terabytes of crucial data. This was for the simple, mundane privilege of letting my Lambda functions send emails and tell Stripe to charge a credit card.

Let’s talk about NAT Gateway pricing. It’s a special kind of pain.

  • $0.045 per hour (That’s roughly $33 a month, just for existing).
  • $0.045 per GB processed (You get charged for your own data).
  • …and that’s per Availability Zone. For High Availability, you multiply by two or three.

I was suddenly paying more for a digital toll booth operator than I was for the actual application logic running my startup. That’s when I started asking questions. Did I really need this? What was I actually paying for? And more importantly, was there another way?

This is the story of how I hunted down that 33€ line item. By the end, you’ll know exactly if you need a NAT Gateway, or if you’re just burning money to keep the AWS machine fed.

The great NAT lie

Every AWS tutorial, every Stack Overflow answer, every “serverless best practice” blog post chants the same mantra: “If your Lambda needs to access the internet, and it’s in a VPC, you need a NAT Gateway.”

It’s presented as a law of physics. Like gravity, or the fact that DNS will always be the problem. And I, like a good, obedient engineer, followed the instructions. I clicked the button. I added the NAT. And then the bill came.

It turns out that obedience is expensive.

The gilded cage we call a VPC

Before we storm the castle, we have to understand why we built the castle in the first place. Why are our Lambdas in this mess? The answer is the Virtual Private Cloud (VPC).

By default, a Lambda function is a free spirit. It’s born with a magical, AWS-managed connection to the outside world. It can call any API it wants. It’s a social butterfly.

But then, security happens.

We have a managed database, like MongoDB Atlas. We absolutely, positively do not want this database exposed to the public internet. That’s like shouting your bank details across a crowded shopping mall. So, we rightly configure it to only accept private connections.

To let our Lambda talk to this database, we have to build a “gated community” for it. That’s our VPC. We move the Lambda inside this community and set up a “VPC Peering” connection, which is like a private, guarded footpath between our VPC and the MongoDB VPC.

Our Lambda can now securely whisper secrets to the database. The traffic never touches the public internet. We are secure. We are compliant. We are… trapped.

House arrest

We solved one problem but created a massive new one. In building this fortress to protect our database, we built it with no doors to the outside world.

Our Lambda is now on house arrest.

Sure, it can talk to the database in the adjoining room. But it can no longer call the Stripe API to process a payment. It can’t call an email service. It can’t even phone its own cousins in the AWS family, like AWS Secrets Manager or S3 (not without extra work, anyway). Any attempt to reach the internet just… times out. It’s the sound of silence.

This is the dilemma. To be secure, our Lambda must be in a VPC. But once in a VPC, it’s useless for half its job.

Enter the expensive chaperone

This is where the AWS Gospel presents its solution: the NAT Gateway.

The NAT (Network Address Translation) Gateway is, in our analogy, an extremely expensive, bonded chaperone.

You place this chaperone in a “public” part of your gated community (a public subnet). When your Lambda on house arrest needs to send a letter to the outside world (like an API call to Stripe), it gives the letter to the chaperone.

The chaperone (the NAT) takes the letter, walks it to the main gate, puts its own public return address on it, and sends it. When the reply comes back, the chaperone receives it, verifies it’s for the Lambda, and delivers it.

This works. It’s secure. The Lambda’s private address is never exposed.

But this chaperone charges you. It charges you by the hour just to be on call. It charges you for every letter it carries (data processed). And as we established, you need three of them if you want to be properly redundant.

This is a racket.

The “Split Personality” solution

I refused to pay the toll. There had to be another way. The solution came from realizing I was trying to make one Lambda do two completely opposite jobs.

What if, instead of one “do-it-all” Lambda, I created two specialists?

  1. The hermit: This Lambda lives inside the VPC. Its one and only job is to talk to the database. It is antisocial, secure, and has no idea the internet exists.
  2. The messenger: This Lambda lives outside the VPC. It’s a “free-range” Lambda. Because it’s not attached to any VPC, AWS magically gives it that default internet access. It cannot talk to the database (which is good!), but it can talk to Stripe all day long.

The plan is simple: when The hermit (VPC Lambda) needs something from the internet, it invokes The messenger (Proxy Lambda). It hands it a note: “Please tell Stripe to charge $25.00.” The messenger runs the errand, gets the receipt, and passes it back to The hermit, who then safely logs the result in the database.

It’s a “split personality” architecture.

But is it safe?

I can hear you asking: “Wait. A Lambda with internet access? Isn’t that like leaving your front door wide open for attackers?”

No. And this is the most beautiful part.

A Lambda function, whether in a VPC or not, never gets a public IP address. It can make outbound calls, but nothing from the public internet can initiate a call to it.

It’s like having a phone that can only make calls, not receive them. It’s unreachable. The “Messenger” Lambda is perfectly safe to live outside the VPC, ready to do our bidding.

The secret tunnel system

So, I built it. The hermit. The messenger. I was a genius. I hit “test.”

…timeout.

Of course. I forgot. The hermit is still on house arrest. “Invoking” another Lambda is, itself, an AWS API call. It’s a request that has to leave the VPC to reach the AWS Lambda service. My Lambda couldn’t even call its own lawyer.

This is where the real solution lies. Not in a gateway, but in a series of tunnels.

They’re called VPC Endpoints.

A VPC Endpoint is not a big, expensive, public chaperone. It’s a private, secret tunnel that you build directly from your VPC to a specific AWS service, all within the AWS network.

So, I built two tunnels:

  1. A tunnel to AWS Secrets Manager: Now my hermit Lambda can get its API keys directly, without ever leaving the house.
  2. A tunnel to AWS Lambda: Now my hermit Lambda can use its private phone to “invoke” The messenger.

These endpoints have a small hourly cost, but it’s a fraction of a NAT Gateway, and the data processing fee is either tiny or free, depending on the endpoint type. We’ve replaced a $100/mo toll road with a $5/mo private footpath.

(A grumpy side note: annoyingly, some AWS services like Cognito don’t support VPC Endpoints. For those, you still have to use the Messenger proxy pattern. But for most, the tunnels work.)

Our glorious new contraption

Let’s look at our payment handler again. This little function needed to:

  1. Get API keys from AWS Secrets Manager.
  2. Call Stripe’s API.
  3. Write the transaction to MongoDB.

Here is how our new, glorious, Rube Goldberg machine works:

  • Step 1: The Payment Lambda (The hermit) gets a request.
  • Step 2: It needs keys. It pops over to AWS Secrets Manager through its private tunnel (the VPC Endpoint). No internet needed.
  • Step 3: It needs to charge a card. It calls the invoke command, which goes through its other private tunnel to the AWS Lambda service, triggering The messenger.
  • Step 4: The messenger (Proxy Lambda), living in the free-range world, makes the outbound call to Stripe. Stripe, delighted, processes the payment and sends a reply.
  • Step 5: The messenger passes the success (or failure) response back to The hermit.
  • Step 6: The hermit, now holding the result, calmly turns and writes the transaction record to MongoDB via its private VPC Peering connection.

Everything works. Nothing is exposed. And the NAT Gateway bill is 0€.

For those who speak in code

Here is a simplified look at what our two specialist Lambdas are doing.

Payment Lambda (The hermit – INSIDE VPC)

// This Lambda is attached to your VPC
// It needs VPC Endpoints for 'lambda' and 'secretsmanager'

import { InvokeCommand, LambdaClient } from "@aws-sdk/client-lambda";
// ... (imports for Secrets Manager and Mongo)

const lambda = new LambdaClient({});

export const handler = async (event) => {
  try {
    const amountToCharge = 2500; // 25.00

    // 1. Get secrets via VPC Endpoint
    // const apiKeys = await getSecretsFromManager();
    
    // 2. Prepare to invoke the proxy
    const command = new InvokeCommand({
      FunctionName: process.env.PAYMENT_PROXY_FUNCTION_NAME,
      InvocationType: "RequestResponse",
      Payload: JSON.stringify({
        chargeDetails: { amount: amountToCharge, currency: "usd" },
      }),
    });

    // 3. Invoke the proxy Lambda via VPC Endpoint
    const response = await lambda.send(command);
    const proxyResponse = JSON.parse(
      Buffer.from(response.Payload).toString()
    );

    if (proxyResponse.status === "success") {
      // 4. Write to MongoDB via VPC Peering
      // await writePaymentRecordToMongo(proxyResponse.transactionId);
      
      return {
        statusCode: 200,
        body: `Payment succeeded! TxID: ${proxyResponse.transactionId}`,
      };
    } else {
      // Handle payment failure
      return { statusCode: 400, body: "Payment failed." };
    }
  } catch (error) {
    console.error(error);
    return { statusCode: 500, body: "Server error" };
  }
};

Proxy Lambda (The messenger – OUTSIDE VPC)

// This Lambda is NOT attached to a VPC
// It has default internet access

// ... (import for your Stripe client)
// const stripe = new Stripe(process.env.STRIPE_SECRET_KEY);

export const handler = async (event) => {
  // 1. Extract the data from the invoking Hermit
  const { chargeDetails } = event.payload;

  try {
    // 2. Call the external Stripe API
    // const stripeResponse = await stripe.charges.create({
    //   amount: chargeDetails.amount,
    //   currency: chargeDetails.currency,
    //   source: "tok_visa", // Example token
    // });
   
    // Mocking the Stripe call for this example
    const stripeResponse = {
        id: `txn_${Math.random().toString(36).substring(2, 15)}`,
        status: 'succeeded'
    };


    if (stripeResponse.status === 'succeeded') {
      // 3. Return the successful result
      return {
        status: "success",
        transactionId: stripeResponse.id,
      };
    } else {
      return { status: "failed", error: "Stripe decline" };
    }
  } catch (err) {
    // 4. Return any errors
    return {
      status: "failed",
      error: `Error contacting Stripe: ${err.message}`,
    };
  }
};

Was it worth it?

And there it is. A production-grade, secure, and resilient system. Our hermit Lambda is safe in its VPC, talking to the database, our Messenger Lambda is happily running errands on the internet, and our secret tunnels are connecting everything privately.

That said, figuring all this out and integrating it into a production system takes a significant amount of time. This… this contraption of proxies and endpoints is, frankly, a headache.

If you don’t want the headache, sometimes it’s easier to just pay that damn 30€ for a NAT Gateway and move on with your life.

The purpose of this article wasn’t just to save a few bucks. It was to pull back the curtain. To show that the “one true way” isn’t the only way, and to prove that with a little bit of architectural curiosity, you can, in fact, escape the AWS NAT Gateway toll booth.

November 15, 2025 by Cloud stuff SRE stuff

Burst traffic realities for AWS API Gateway Architects

Let’s be honest. Cloud architecture promises infinite scalability, but sometimes it feels like we’re herding cats wearing rocket boots. I learned this the hard way when my shiny serverless app, built with all the modern best practices, started hiccuping like a soda-drunk kangaroo during a Black Friday sale. The culprit? AWS API Gateway throttling under bursty traffic. And no, it wasn’t my coffee intake causing the chaos.

The token bucket, a simple idea with a sneaky side

AWS API Gateway uses a token bucket algorithm to manage traffic. Picture a literal bucket. Tokens drip into it at a steady rate, your rate limit. Each incoming request steals a token to pass through. If the bucket is empty? Requests get throttled. Simple, right? Like a bouncer checking IDs at a club.

But here’s the twist: This bouncer has a strict hourly wage. If 100 requests arrive in one second, they’ll drain the bucket faster than a toddler empties a juice box. Then, even if traffic calms down, the bucket refills slowly. Your API is stuck in timeout purgatory while tokens trickle back. AWS documents this, but it’s easy to miss until your users start tweeting about your “haunted API.”

Bursty traffic is life’s unpredictable roommate

Bursty traffic isn’t a bug; it’s a feature of modern apps. Think flash sales, mobile app push notifications, or that viral TikTok dance challenge your marketing team insisted would go viral (bless their optimism). Traffic doesn’t flow like a zen garden stream. It arrives in tsunami waves.

I once watched a client’s analytics dashboard spike at 3 AM. Turns out, their smart fridge app pinged every device simultaneously after a firmware update. The bucket emptied. Alarms screamed. My weekend imploded. Bursty traffic doesn’t care about your sleep schedule.

When bursts meet buckets, the throttling tango

Here’s where things get spicy. API Gateway’s token bucket has a burst capacity. For stage-level throttling, it’s tied to your rate limit. Set a rate of 100 requests/second? Your bucket holds 100 tokens. Send 150 requests in one burst? The first 100 sail through. The next 50 get throttled, even if the average traffic is below 100/second.

It’s like a theater with 100 seats. If 150 people rush the door at once, 50 get turned away, even if half the theater is empty later. AWS isn’t being petty. It’s protecting downstream services (like your database) from sudden stampedes. But when your app is the one getting trampled? Less poetic. More infuriating.

Does this haunt all throttling types?

Good news: This quirk primarily targets stage-level and account-level throttling. Usage Plans? They play by different rules. Their buckets refill steadily, making them more burst-friendly. But stage-level throttling? It’s the diva of the trio. Configure it carelessly, and it will sabotage your bursts like a jealous ex.

If you’ve layered all three throttling types (account, stage, usage plan), stage-level settings often dominate the drama. Check your stage settings first. Always.

Taming the beast, practical fixes that work

After several caffeine-fueled debugging sessions, I’ve learned a few tricks to keep buckets full and bursts happy. None requires sacrificing a rubber chicken to the cloud gods.

1. Resize your bucket
Stage-level throttling lets you set a burst limit alongside your rate limit. Double it. Triple it. AWS allows bursts up to 5,000 requests for some tiers. Calculate your peak bursts (use CloudWatch metrics!), then set burst capacity 20% higher. Safety margins are boring until they save your launch day.

2. Queue the chaos
Offload bursts to SQS or Kinesis. Front your API with a lightweight service that accepts requests instantly, dumps them into a queue, and processes them at a civilized pace. Users get a “we got this” response. Your bucket stays calm. Everyone wins. Except the throttling gremlins.

3. Smarter clients are your friends
Teach client apps to retry intelligently. Exponential backoff with jitter isn’t just jargon, it’s the art of politely asking “Can I try again later?” instead of spamming “HELLO?!” every millisecond. AWS SDKs bake this in. Use it.

4. Distribute the pain
Got multiple stages or APIs? Spread bursts across them. A load balancer or Route 53 weighted routing can turn one screaming bucket into several murmuring ones. It’s like splitting a rowdy party into smaller rooms.

5. Monitor like a paranoid squirrel
CloudWatch alarms for 429 Too Many Requests are non-negotiable. Track ThrottledRequests and Count metrics per stage. Set alerts at 70% of your burst limit. Because knowing your bucket is half-empty is far better than discovering it via customer complaints.

The quiet triumph of preparedness

Cloud architecture is less about avoiding fires and more about not using gasoline as hand sanitizer. Bursty traffic will happen. Token buckets will empty. But with thoughtful configuration, you can transform throttling from a silent assassin into a predictable gatekeeper.

AWS gives you the tools. It’s up to us to wield them without setting the data center curtains ablaze. Start small. Test bursts in staging. And maybe keep that emergency coffee stash stocked. Just in case.

Your APIs deserve grace under pressure. Now go forth and throttle wisely. Or better yet, throttle less.

November 4, 2025 by Cloud stuff DevOps stuff SRE stuff

The slow unceremonious death of EC2 Autoscaling

Let’s pour one out for an old friend.

AWS recently announced a small, seemingly boring new feature for EC2 Auto Scaling: the ability to cancel a pending instance refresh. If you squinted, you might have missed it. It sounds like a minor quality-of-life update, something to make a sysadmin’s Tuesday slightly less terrible.

But this isn’t a feature. It’s a gold watch. It’s the pat on the back and the “thanks for your service” speech at the awkward retirement party.

The EC2 Auto Scaling Group (ASG), the bedrock of cloud elasticity, the one tool we all reflexively reached for, is being quietly put out to pasture.

No, AWS hasn’t officially killed it. You can still spin one up, just like you can still technically send a fax. AWS will happily support it. But its days as the default, go-to solution for modern workloads are decisively over. The battle for the future of scaling has ended, and the ASG wasn’t the winner. The new default is serverless containers, hyper-optimized Spot fleets, and platforms so abstract they’re practically invisible.

If you’re still building your infrastructure around the ASG, you’re building a brand-new house with plumbing from 1985. It’s time to talk about why our old friend is retiring and meet the eager new hires who are already measuring the drapes in its office.

So why is the ASG getting the boot?

We loved the ASG. It was a revolutionary idea. But like that one brilliant relative everyone dreads sitting next to at dinner, it was also exhausting. Its retirement was long overdue, and the reasons are the same frustrations we’ve all been quietly grumbling about into our coffee for years.

It promised automation but gave us chores

The ASG’s sales pitch was simple: “I’ll handle the scaling!” But that promise came with a three-page, fine-print addendum of chores.

It was the operational overhead that killed us. We were promised a self-driving car and ended up with a stick-shift that required constant, neurotic supervision. We became part-time Launch Template librarians, meticulously versioning every tiny change. We became health-check philosophers, endlessly debating the finer points of ELB vs. EC2 health checks.

And then… the Lifecycle Hooks.

A “Lifecycle Hook” is a polite, clinical term for a Rube Goldberg machine of desperation. It’s a panic button that triggers a Lambda, which calls a Systems Manager script, which sends a carrier pigeon to… maybe… drain a connection pool before the instance is ruthlessly terminated. Trying to debug one at 3 AM was a rite of passage, a surefire way to lose precious engineering time and a little bit of your soul.

It moves at a glacial pace

The second nail in the coffin was its speed. Or rather, the complete lack of it.

The ASG scales at the speed of a full VM boot. In our world of spiky, unpredictable traffic, that’s an eternity. It’s like pre-heating a giant, industrial pizza oven for 45 minutes just to toast a single slice of bread. By the time your new instance is booted, configured, service-discovered, and finally “InService,” the spike in traffic has already come and gone, leaving you with a bigger bill and a cohort of very annoyed users.

It’s an expensive insurance policy

The ASG model is fundamentally wasteful. You run a “warm” fleet, paying for idle capacity just in case you need it. It’s like paying rent on a 5-bedroom house for your family of three, just in case 30 cousins decide to visit unannounced.

This “scale-up” model was slow, and the “scale-down” was even worse, riddled with fears of terminating the wrong instance and triggering a cascading failure. We ended up over-provisioning to avoid the pain of scaling, which completely defeats the purpose of “auto-scaling.”

The eager interns taking over the desk

So, the ASG has cleared out its desk. Who’s moving in? It turns out there’s a whole line of replacements, each one leaner, faster, and blissfully unconcerned with managing a “fleet.”

1. The appliance Fargate and Cloud Run

First up is the “serverless container”. This is the hyper-efficient new hire who just says, “Give me the Dockerfile. I’ll handle the rest.”

With AWS Fargate or Google’s Cloud Run, you don’t have a fleet. You don’t manage VMs. You don’t patch operating systems. You don’t even think about an instance. You just define a task, give it some CPU and memory, and tell it how many copies you want. It scales from zero to a thousand in seconds.

This is the appliance model. When you buy a toaster, you don’t worry about wiring the heating elements or managing its power supply. You just put in bread and get toast. Fargate is the toaster. The ASG was the “build-your-own-toaster” kit that came with a 200-page manual on electrical engineering.

Just look at the cognitive load. This is what it takes to get a basic ASG running via the CLI:

# The "Old Way": Just one of the many steps...
aws autoscaling create-auto-scaling-group \
    --auto-scaling-group-name my-legacy-asg \
    --launch-template "LaunchTemplateName=my-launch-template,Version='1'" \
    --min-size 1 \
    --max-size 5 \
    --desired-capacity 2 \
    --vpc-zone-identifier "subnet-0571c54b67EXAMPLE,subnet-0c1f4e4776EXAMPLE" \
    --health-check-type ELB \
    --health-check-grace-period 300 \
    --tag "Key=Name,Value=My-ASG-Instance,PropagateAtLaunch=true"

You still need to define the launch template, the subnets, the load balancer, the health checks…

Now, here’s the core of a Fargate task definition. It’s just a simple JSON file:

// The "New Way": A snippet from a Fargate Task Definition
{
  "family": "my-modern-app",
  "containerDefinitions": [
    {
      "name": "my-container",
      "image": "nginx:latest",
      "cpu": 256,
      "memory": 512,
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 80
        }
      ]
    }
  ],
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512"
}

You define what you need, and the platform handles everything else.

2. The extreme couponer Spot fleets

For workloads that are less “instant spike” and more “giant batch job,” we have the “optimized fleet”. This is the high-stakes, high-reward world of Spot Instances.

Spot used to be terrifying. AWS could pull the plug with two minutes’ notice, and your entire workload would evaporate. But now, with Spot Fleets and diversification, it’s the smartest tool in the box. You can tell AWS, “I need 1,000 vCPUs, and I don’t care what instance types you give me, just find the cheapest ones.”

The platform then builds a diversified fleet for you across multiple instance types and Availability Zones, making it incredibly resilient to any single Spot pool termination. It’s perfect for data processing, CI/CD runners, and any batch job that can be interrupted and resumed. The ASG was always too rigid for this kind of dynamic, cost-driven scaling.

3. The paranoid security guard MicroVMs

Then there’s the truly weird stuff: Firecracker. This is the technology that powers AWS Lambda and Fargate. It’s a “MicroVM” that gives you the iron-clad security isolation of a full virtual machine but with the lightning-fast startup speed of a container.

We’re talking boot times of under 125 milliseconds. This is for when you need to run thousands of tiny, separate, untrusted workloads simultaneously without them ever being able to see each other. It’s the ultimate “multi-tenant” dream, giving every user their own tiny, disposable, fire-walled VM in the blink of an eye.

4. The invisible platform Edge runtimes

Finally, we have the platforms that are so abstract they’re “scaled to invisibility”. This is the world of Edge. Think Lambda@Edge or CloudFront Functions.

With these, you’re not even scaling in a region anymore. Your logic, your code, is automatically replicated and executed at hundreds of Points of Presence around the globe, as close to the end-user as possible. The entire concept of a “fleet” or “instance” just… disappears. The logic scales with the request.

Life after the funeral. How to adapt

Okay, the eulogy is over. The ASG is in its rocking chair on the porch. What does this mean for us, the builders? It’s time to sort through the old belongings and modernize the house.

Go full Marie Kondo on your architecture

First, you need to re-evaluate. Open up your AWS console and take a hard look at every single ASG you’re running. Be honest. Ask the tough questions:

  • Does this workload really need to be stateful?
  • Do I really need VM-level control, or am I just clinging to it for comfort?
  • Is this a stateless web app that I’ve just been too lazy to containerize?

If it doesn’t spark joy (or isn’t a snowflake legacy app that’s impossible to change), thank it for its service and plan its migration.

Stop shopping for engines, start shopping for cars

The most important shift is this: Pick the runtime, not the infrastructure.

For too long, our first question was, “What EC2 instance type do I need?” That’s the wrong question. That’s like trying to build a new car by starting at the hardware store to buy pistons.

The right question is, “What’s the best runtime for my workload?”

  • Is it a simple, event-driven piece of logic? That’s a Function (Lambda).
  • Is it a stateless web app in a container? That’s a Serverless Container (Fargate).
  • Is it a massive, interruptible batch job? That’s an Optimized Fleet (Spot).
  • Is it a cranky, stateful monolith that needs a pet VM? Only then do you fall back to an Instance (EC2, maybe even with an ASG).

Automate logic, not instance counts

Your job is no longer to be a VM mechanic. Your team’s skills need to shift. Stop manually tuning desired_capacity and start designing event-driven systems.

Focus on scaling logic, not servers. Your scaling trigger shouldn’t be “CPU is at 80%.” It should be “The SQS queue depth is greater than 100” or “API latency just breached 200ms”. Let the platform, be it Lambda, Fargate, or a KEDA-powered Kubernetes cluster, figure out how to add more processing power.

Was it really better in the old days?

Of course, this move to abstraction isn’t without trade-offs. We’re gaining a lot, but we’re also losing something.

The gain is obvious: We get our nights and weekends back. We get drastically reduced operational overhead, faster scaling, and for most stateless workloads, a much lower bill.

The loss is control. You can’t SSH into a Fargate container. You can’t run a custom kernel module on Lambda. For those few, truly special, high-customization legacy workloads, this is a dealbreaker. They will be the ASG’s loyal companions in the retirement home.

But for everything else? The ASG is a relic. It was a brilliant, necessary solution for the problems of 2010. But the problems of 2025 and beyond are different. The cloud has evolved to scale logic, functions, and containers, not just nodes.

The king isn’t just dead. The very concept of a throne has been replaced by a highly efficient, distributed, and slightly impersonal serverless committee. And frankly, it’s about time.

October 24, 2025 by Cloud stuff DevOps stuff SRE stuff

When invisible limits beat warm Lambdas

My team had a problem. Or rather, we had a cause. A noble crusade that consumed our sprints, dominated our Slack channels, and haunted our architectural diagrams. We were on a relentless witch hunt for the dreaded Lambda cold start.

We treated those extra milliseconds of spin-up time like a personal insult from Jeff Bezos himself. We became amateur meteorologists, tracking “cold start storms” across regions. We had dashboards so finely tuned they could detect the faint, quantum flutter of an EC2 instance thinking about starting up. We proudly spent over $3,000 a month on provisioned concurrency¹, a financial sacrifice to the gods of AWS to keep our functions perpetually toasty.

We had done it. Cold starts were a solved problem. We celebrated with pizza and self-congratulatory Slack messages. The system was invincible.

Or so we thought.

The 2:37 am wake-up call

It was a Tuesday, of course. The kind of quiet, unassuming Tuesday that precedes all major IT disasters. At 2:37 AM, my phone began its unholy PagerDuty screech. The alert was as simple as it was terrifying: “API timeouts.”

I stumbled to my laptop, heart pounding, expecting to see a battlefield. Instead, I saw a paradox.

The dashboards were an ocean of serene green.

  • Cold starts? 0%. Our $3,000 was working perfectly. Our Lambdas were warm, cozy, and ready for action.
  • Lambda health? 100%. Every function was executing flawlessly, not an error in sight.
  • Database queries? 100% failure rate.

It was like arriving at a restaurant to find the chefs in the kitchen, knives sharpened and stoves hot, but not a single plate of food making it to the dining room. Our Lambdas were warm, our dashboards were green, and our system was dying. It turns out that for $3,000 a month, you can keep your functions perfectly warm while they helplessly watch your database burn to the ground.

We had been playing Jenga with AWS’s invisible limits, and someone had just pulled the wrong block.

Villain one, The great network card famine

Every Lambda function that needs to talk to services within your VPC, like a database, requires a virtual network card, an Elastic Network Interface (ENI). It’s the function’s physical connection to the world. And here’s the fun part that AWS tucks away in its documentation: your account has a default, region-wide limit on these. Usually around 250.

We discovered this footnote from 2018 when the Marketing team, in a brilliant feat of uncoordinated enthusiasm, launched a flash promo.

Our traffic surged. Lambda, doing its job beautifully, began to scale. 100 concurrent executions. 200. Then 300.

The 251st request didn’t fail. Oh no, that would have been too easy. Instead, it just… waited. For fourteen seconds. It was waiting in a silent, invisible line for AWS to slowly hand-carve a new network card from the finest, artisanal silicon.

Our “optimized” system had become a lottery.

  • The winners: Got an existing ENI and a zippy 200ms response.
  • The losers: Waited 14,000ms for a network card to materialize out of thin air, causing their request to time out.

The worst part? This doesn’t show up as a Lambda error. It just looks like your code is suddenly, inexplicably slow. We were hunting for a bug in our application, but the culprit was a bureaucrat in the AWS networking department.

Do this right now. Seriously. Open a terminal and check your limit. Don’t worry, we’ll wait.

# This command reveals the 'Maximum network interfaces per Region' quota.
# You might be surprised at what you find.
aws service-quotas get-service-quota \
  --service-code vpc \
  --quota-code L-F678F1CE

Villain two, The RDS proxy’s velvet rope policy

Having identified the ENI famine, we thought we were geniuses. But fixing that only revealed the next layer of our self-inflicted disaster. Our Lambdas could now get network cards, but they were all arriving at the database party at once, only to be stopped at the door.

We were using RDS Proxy, the service AWS sells as the bouncer for your database, managing connections so your Aurora instance doesn’t get overwhelmed. What we failed to appreciate is that this bouncer has its own… peculiar rules. The proxy itself has CPU limits. When hundreds of Lambdas tried to get a connection simultaneously, the proxy’s CPU spiked to 100%.

It didn’t crash. It just became incredibly, maddeningly slow. It was like a nightclub bouncer enforcing a strict one-in, one-out policy, not because the club was full, but because he could only move his arms so fast. The queue of connections grew longer and longer, each one timing out, while the database inside sat mostly idle, wondering where everybody went.

The humbling road to recovery

The fixes weren’t complex, but they were humbling. They forced us to admit that our beautiful, perfectly-tuned relational database architecture was, for some tasks, the wrong tool for the job.

  1. The great VPC escape
    For any Lambda that only needed to talk to public AWS services like S3 or SQS, we ripped it out of the VPC. This is Lambda 101, but we had put everything in the VPC for “security.” Moving them out meant they no longer needed an ENI to function. We implemented VPC Endpoints², allowing these functions to access AWS services over a private link without the ENI overhead.
  2. RDS proxy triage
    For the databases we couldn’t escape, we treated the proxy like the delicate, overworked bouncer it was. We massively over-provisioned the proxy instances, giving them far more CPU than they should ever need. We also implemented client-side jitter, a small, random delay before retrying a connection, to stop our Lambdas from acting like a synchronized mob storming the gates.
  3. The nuclear option DynamoDB
    For one critical, high-throughput service, we did the unthinkable. We migrated it from Aurora to DynamoDB. The hardest part wasn’t the code; it was the ego. It was admitting that the problem didn’t require a Swiss Army knife when all we needed was a hammer. The team’s reaction after the migration was telling: “Wait… you mean we don’t need to worry about connection pooling at all?” Every developer, after their first taste of NoSQL freedom.

The real lesson we learned

Obsessing over cold starts is like meticulously polishing the chrome on your car’s engine while the highway you’re on is crumbling into a sinkhole. It’s a visible, satisfying metric to chase, but it often distracts from the invisible, systemic limits that will actually kill you.

Yes, optimize your cold starts. Shave off those milliseconds. But only after you’ve pressure-tested your system for the real bottlenecks. The unsexy ones. The ones buried in AWS service quota pages and 5-year-old forum posts.

Stop micro-optimizing the 50ms you can see and start planning for the 14-second delays you can’t. We learned that the hard way, at 2:37 AM on a Tuesday.

¹ The official term for ‘setting a pile of money on fire to keep your functions toasty’.

² A fancy AWS term for ‘a private, secret tunnel to an AWS service so your Lambda doesn’t have to go out into the scary public internet’. It’s like an employee-only hallway in a giant mall.

October 10, 2025 by Cloud stuff DevOps stuff SRE stuff

Avoiding serverless chaos with 3 essential Lambda patterns

Your first Lambda function was a thing of beauty. Simple, elegant, it did one job and did it well. Then came the second. And the tenth. Before you knew it, you weren’t running an application; you were presiding over a digital ant colony, with functions scurrying in every direction without a shred of supervision.

AWS Lambda, the magical service that lets us run code without thinking about servers, can quickly devolve into a chaotic mess of serverless spaghetti. Each function lives happily in its own isolated bubble, and when demand spikes, AWS kindly hands out more and more bubbles. The result? An anarchic party of concurrent executions.

But don’t despair. Before you consider a career change to alpaca farming, let’s introduce three seasoned wranglers who will bring order to your serverless circus. These are the architectural patterns that separate the rookies from the maestros in the art of building resilient, scalable systems.

Meet the micromanager boss

First up is a Lambda with a clipboard and very little patience. This is the Command Pattern function. Its job isn’t to do the heavy lifting—that’s what the interns are for. Its sole purpose is to act as the gatekeeper, the central brain that receives an order, scrutinizes it (request validation), consults its dusty rulebook (business logic), and then barks commands at its underlings to do the actual work.

It’s the perfect choice for workflows where bringing in AWS Step Functions would be like using a sledgehammer to crack a nut. It centralizes decision-making and maintains a crystal-clear separation between those who think and those who do.

When to hire this boss

  • For small to medium workflows that need a clear, single point of control.
  • When you need a bouncer at the door to enforce rules before letting anyone in.
  • If you appreciate a clean hierarchy: one boss, many workers.

A real-world scenario

An OrderProcessor Lambda receives a new order via API Gateway. It doesn’t trust anyone. It first validates the payload, saves a record to DynamoDB so it can’t get lost, and only then does it invoke other Lambdas: one to handle the payment, another to send a confirmation email, and a third to notify the shipping department. The boss orchestrates; the workers execute. Clean and effective.

Visually, it looks like a central hub directing traffic:

Here’s how that boss might delegate a task to the notifications worker:

// The Command Lambda (e.g., process-order-command)
import { LambdaClient, InvokeCommand } from "@aws-sdk/client-lambda";

const lambdaClient = new LambdaClient({ region: "us-east-1" });

export const handler = async (event) => {
    const orderDetails = JSON.parse(event.body);

    // 1. Validate and save the order (your business logic here)
    console.log(`Processing order ${orderDetails.orderId}...`);
    // ... logic to save to DynamoDB ...

    // 2. Delegate to the notification worker
    const invokeParams = {
        FunctionName: 'arn:aws:lambda:us-east-1:123456789012:function:send-confirmation-email',
        InvocationType: 'Event', // Fire-and-forget
        Payload: JSON.stringify({
            orderId: orderDetails.orderId,
            customerEmail: orderDetails.customerEmail,
        }),
    };

    await lambdaClient.send(new InvokeCommand(invokeParams));

    return {
        statusCode: 202, // Accepted
        body: JSON.stringify({ message: "Order received and is being processed." }),
    };
};

The dark side of micromanagement

Be warned. This boss can become a bottleneck. If all decisions flow through one function, it can get overwhelmed. It also risks becoming a “God Object,” a monstrous function that knows too much and does too much, making it a nightmare to maintain and a single, terrifying point of failure.

Enter the patient courier

So, what happens when the micromanager gets ten thousand requests in one second? It chokes, your system grinds to a halt, and you get a frantic call from your boss. The Command Pattern’s weakness is its synchronous nature. We need a buffer. We need an intermediary.

This is where the Messaging Pattern comes in, embodying the art of asynchronous patience. Here, instead of talking directly, services drop messages into a queue or stream (like SQS, SNS, or Kinesis). A consumer Lambda then picks them up whenever it’s ready. This builds healthy boundaries between your services, absorbs sudden traffic bursts like a sponge, and ensures that if something goes wrong, the message can be retried.

When to Call the Courier

  • For bursty or unpredictable workloads that would otherwise overwhelm your system.
  • To isolate slow or unreliable third-party services from your main request path.
  • When you need to offload heavy tasks to be processed in the background.
  • If you need a guarantee that a task will be executed at least once, with a safety net (a Dead-Letter Queue) for messages that repeatedly fail.

A Real-World Scenario

A user clicks “Checkout.” Instead of processing everything right away, the API Lambda simply drops an OrderPlaced event into an SQS queue and immediately returns a success message to the user. On the other side, a ProcessOrderQueue Lambda consumes events from the queue at its own pace. It reserves inventory, charges the credit card, and sends notifications. If the payment service is down, SQS holds the message, and the Lambda tries again later. No lost orders, no frustrated users.

The flow decouples the producer from the consumer:

The producer just needs to drop the message and walk away:

// The Producer Lambda (e.g., checkout-api)
import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";

const sqsClient = new SQSClient({ region: "us-east-1" });

export const handler = async (event) => {
    const orderDetails = JSON.parse(event.body);

    const command = new SendMessageCommand({
        QueueUrl: "[https://sqs.us-east-1.amazonaws.com/123456789012/OrderProcessingQueue](https://sqs.us-east-1.amazonaws.com/123456789012/OrderProcessingQueue)",
        MessageBody: JSON.stringify(orderDetails),
        MessageGroupId: orderDetails.orderId // For FIFO queues
    });

    await sqsClient.send(command);

    return {
        statusCode: 200,
        body: JSON.stringify({ message: "Your order is confirmed!" }),
    };
};

The price of patience

This resilience isn’t free. The biggest trade-off is added latency; you’re introducing an extra step. It also makes end-to-end tracing more complex. Debugging a journey that spans across a queue can feel like trying to track a package with no tracking number.

Unleash the Ttown crier

Sometimes, one piece of news needs to be told to everyone, all at once, without waiting for them to ask. You don’t want a single boss delegating one by one, nor a courier delivering individual letters. You need a proclamation.

The Fan-Out Pattern is your digital town crier. A single event is published to a central hub (typically an SNS topic or EventBridge), which then broadcasts it to any services that have subscribed. Each subscriber is a Lambda function that kicks into action in parallel, completely unaware of the others.

When to shout from the rooftops

  • When a single event needs to trigger multiple, independent downstream processes.
  • For building real-time, event-driven architectures where services react to changes.
  • In high-scale systems where parallel processing is a must.

A real-world scenario

An OrderPlaced event is published to an SNS topic. Instantly, this triggers multiple Lambdas in parallel: one to update inventory, another to send a confirmation email, and a third for the analytics pipeline. The beauty is that the publisher doesn’t know or care who is listening. You can add a fifth or sixth subscriber later without ever touching the original publishing code.

One event triggers many parallel actions:

The publisher’s job is delightfully simple:

// The Publisher Lambda (e.g., reservation-service)
import { SNSClient, PublishCommand } from "@aws-sdk/client-sns";

const snsClient = new SNSClient({ region: "us-east-1" });

export const handler = async (event) => {
    // ... logic to create a reservation ...
    const reservationDetails = {
        reservationId: "res-xyz-123",
        customerEmail: "jane.doe@example.com",
    };

    const command = new PublishCommand({
        TopicArn: "arn:aws:sns:us-east-1:123456789012:NewReservationsTopic",
        Message: JSON.stringify(reservationDetails),
        MessageAttributes: {
            'eventType': {
                DataType: 'String',
                StringValue: 'RESERVATION_CONFIRMED'
            }
        }
    });

    await snsClient.send(command);

    return { status: "SUCCESS", reservationId: reservationDetails.reservationId };
};

The dangers of a loud voice

With great power comes a great potential for a massive, distributed failure. A single poison-pill event could trigger dozens of Lambdas, each failing and retrying, leading to an invocation storm and a bill that will make your eyes water. Careful monitoring and robust error handling in each subscriber are non-negotiable.

Choosing your champions

There you have it: the Micromanager, the Courier, and the Town Crier. Three patterns that form the bedrock of almost any serverless architecture worth its salt.

  • Use the Command Pattern when you need a firm hand on the tiller.
  • Adopt the Messaging Pattern to give your services breathing room and resilience.
  • Leverage the Fan-Out Pattern when one event needs to efficiently kickstart a flurry of activity.

The real magic begins when you combine them. But for now, start seeing your Lambdas not as a chaotic mob of individual functions, but as a team of specialists. With a little architectural guidance, they can build systems that are complex, resilient, and, best of all, cause you far fewer operational headaches.

September 7, 2025 by Cloud stuff SRE stuff

Serverless without the wait

I once bought a five-minute rice cooker that spent four of those minutes warming up with a pathetic hum. It delivered the goods, eventually, but the promise felt… deceptive. For years, AWS Lambda felt like that gadget. It was the perfect kitchen tool for the odd jobs: a bit of glue code here, a light API there. It was the brilliant, quick-fire microwave of our architecture.

Then our little kitchen grew into a full-blown restaurant. Our “hot path”, the user checkout process, became the star dish on our menu. And our diners, quite rightly, expected it to be served hot and fast every time, not after a polite pause while the oven preheated. That polite pause was our cold start, and it was starting to leave a bad taste.

This isn’t a story about how we fell out of love with Lambda. We still adore it. This is the story of how we moved our main course to an industrial-grade, always-on stove. It’s about what we learned by obsessively timing every step of the process and why we still keep that trusty microwave around for the side dishes it cooks so perfectly. Because when your p95 latency needs to be boringly predictable, keeping the kitchen warm isn’t a preference; it’s a law of physics.

What forced us to remodel the kitchen

No single event pushed us over the edge. It was more of a slow-boiling frog situation, a gradual realization that our ambitions were outgrowing our tools. Three culprits conspired against our sub-300ms dream.

First, our traffic got moody. What used to be a predictable tide of requests evolved into sudden, sharp tsunamis during business hours. We needed a sea wall, not a bucket.

Second, our user expectations tightened. We set a rather tyrannical goal of a sub-300ms p95 for our checkout and search paths. Suddenly, the hundreds of milliseconds Lambda spent stretching and yawning before its first cup of coffee became a debt we couldn’t afford.

Finally, our engineers were getting tired. We found ourselves spending more time performing sacred rituals to appease the cold start gods, fiddling with layers, juggling provisioned concurrency, than we did shipping features our users actually cared about. When your mechanics spend more time warming up the engine than driving the car, you know something’s wrong.

The punchline isn’t that Lambda is “bad.” It’s that our requirements changed. When your performance target drops below the cost of a cold start plus dependency initialization, physics sends you a sternly worded letter.

Numbers don’t lie, but anecdotes do

We don’t ask you to trust our feelings. We ask you to trust the stopwatch. Replicate this experiment, adjust it for your own tech stack, and let the data do the talking. The setup below is what we used to get our own facts straight. All results are our measurements as of September 2025.

The test shape

  • Endpoint: Returns a simple 1 KB JSON payload.
  • Comparable Compute: Lambda set to 512 MB vs. an ECS Fargate container task with 0.5 vCPU and 1 GB of memory.
  • Load Profile: A steady, closed-loop 100 requests per second (RPS) for 10 minutes.
  • Metrics Reported: p50, p90, p95, p99 latency, and the dreaded error rate.

Our trusty tools

  • Load Generator: The ever-reliable k6.
  • Metrics: A cocktail of CloudWatch and Prometheus.
  • Dashboards: Grafana, to make the pretty charts that managers love.

Your numbers will be different. That’s the entire point. Run the tests, get your own data, and then make a decision based on evidence, not a blog post (not even this one).

Where our favorite gadget struggled

Under the harsh lights of our benchmark, Lambda’s quirks on our hot path became impossible to ignore.

  • Cold start spikes: Provisioned Concurrency can tame these, but it’s like hiring a full-time chauffeur to avoid a random 10-minute wait for a taxi. It costs you a constant fee, and during a real rush hour, you might still get stuck in traffic.
  • The startup toll: Initializing SDKs and warming up connections added tens to hundreds of milliseconds. This “entry fee” was simply too high to hide under our 300ms p95 goal.
  • The debugging labyrinth: Iterating was slow. Local emulators helped, but parity was a myth that occasionally bit us. Debugging felt like detective work with half the clues missing.

Lambda continues to be a genius for event glue, sporadic jobs, and edge logic. It just stopped being the right tool to serve our restaurant’s most popular dish at rush hour.

Calling in the heavy artillery

We moved our high-traffic endpoints to container-native services. For us, that meant ECS on Fargate fronted by an Application Load Balancer (ALB). The core idea is simple: keep a few processes warm and ready at all times.

Here’s why it immediately helped:

  • Warm processes: No more cold start roulette. Our application was always awake, connection pools were alive, and everything was ready to go instantly.
  • Standardized packaging: We traded ZIP files for standard Docker images. What we built and tested on our laptops was, byte for byte, what we shipped to production.
  • Civilized debugging: We could run the exact same image locally and attach a real debugger. It was like going from candlelight to a floodlight.
  • Smarter scaling: We could maintain a small cadre of warm tasks as a baseline and then scale out aggressively during peaks.

A quick tale of the tape

Here’s a simplified look at how the two approaches stacked up for our specific needs.

Our surprisingly fast migration plan

We did this in days, not weeks. The key was to be pragmatic, not perfect.

1. Pick your battles: We chose our top three most impactful endpoints with the worst p95 latency.

2. Put it in a box: We converted the function handler into a tiny web service. It’s less dramatic than it sounds.

# Dockerfile (Node.js example)
FROM node:22-slim
WORKDIR /usr/src/app

COPY package*.json ./
RUN npm ci --only=production

COPY . .

ENV NODE_ENV=production PORT=3000
EXPOSE 3000
CMD [ "node", "server.js" ]
// server.js
const http = require('http');
const port = process.env.PORT || 3000;

const server = http.createServer((req, res) => {
  if (req.url === '/health') {
    res.writeHead(200, { 'Content-Type': 'text/plain' });
    return res.end('ok');
  }

  // Your actual business logic would live here
  const body = JSON.stringify({ success: true, timestamp: Date.now() });
  res.writeHead(200, { 'Content-Type': 'application/json' });
  res.end(body);
});

server.listen(port, () => {
  console.log(`Server listening on port ${port}`);
});

3. Set up the traffic cop: We created a new target group for our service and pointed a rule on our Application Load Balancer to it.

{
  "family": "payment-api",
  "networkMode": "awsvpc",
  "cpu": "512",
  "memory": "1024",
  "requiresCompatibilities": ["FARGATE"],
  "executionRoleArn": "arn:aws:iam::987654321098:role/ecsTaskExecutionRole",
  "taskRoleArn": "arn:aws:iam::987654321098:role/paymentTaskRole",
  "containerDefinitions": [
    {
      "name": "app-container",
      "image": "[987654321098.dkr.ecr.us-east-1.amazonaws.com/payment-api:2.1.0](https://987654321098.dkr.ecr.us-east-1.amazonaws.com/payment-api:2.1.0)",
      "portMappings": [{ "containerPort": 3000, "protocol": "tcp" }],
      "environment": [{ "name": "NODE_ENV", "value": "production" }]
    }
  ]
}

4. The canary in the coal mine: We used weighted routing to dip our toes in the water. We started by sending just 5% of traffic to the new container service.

# Terraform Route 53 weighted canary
resource "aws_route53_record" "api_primary_lambda" {
  zone_id = var.zone_id
  name    = "api.yourapp.com"
  type    = "A"

  alias {
    name                   = aws_api_gateway_domain_name.main.cloudfront_domain_name
    zone_id                = aws_api_gateway_domain_name.main.cloudfront_zone_id
    evaluate_target_health = true
  }

  set_identifier = "primary-lambda-path"
  weight         = 95
}

resource "aws_route53_record" "api_canary_container" {
  zone_id = var.zone_id
  name    = "api.yourapp.com"
  type    = "A"

  alias {
    name                   = aws_lb.main_alb.dns_name
    zone_id                = aws_lb.main_alb.zone_id
    evaluate_target_health = true
  }

  set_identifier = "canary-container-path"
  weight         = 5
}

5. Stare at the graphs: For one hour, we watched four numbers like hawks: p95 latency, error rates, CPU/memory headroom on the new service, and our estimated cost per million requests.

6. Go all in (or run away): The graphs stayed beautifully, boringly flat. So we shifted to 50%, then 100%. The whole affair was done in an afternoon.

The benchmark kit you can steal

Don’t just read about it. Run a quick test yourself.

// k6 script (save as test.js)
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  vus: 100,
  duration: '5m',
  thresholds: {
    'http_req_duration': ['p(95)<250'], // Aim for a 250ms p95
    'checks': ['rate>0.999'],
  },
};

export default function () {
  const url = __ENV.TARGET_URL || '[https://api.yourapp.com/checkout/v2/quote](https://api.yourapp.com/checkout/v2/quote)';
  const res = http.get(url);
  check(res, { 'status is 200': r => r.status === 200 });
  sleep(0.2); // Small pause between requests
}

Run it from your terminal like this:

k6 run -e TARGET_URL=https://your-canary-endpoint.com test.js

Our results for context

These aren’t universal truths; they are snapshots of our world. Your mileage will vary.

The numbers in bold are what kept us up at night and what finally let us sleep. For our steady traffic, the always-on container was not only faster and more reliable, but it was also shaping up to be cheaper.

Lambda is still in our toolbox

We didn’t throw the microwave out. We just stopped using it to cook the Thanksgiving turkey. Here’s where we still reach for Lambda without a second thought:

  • Sporadic or bursty workloads: Those once-a-day reports or rare event handlers are perfect for scale-to-zero.
  • Event glue: It’s the undisputed champion of transforming S3 puts, reacting to DynamoDB streams, and wiring up EventBridge.
  • Edge logic: For tiny header manipulations or rewrites, Lambda@Edge and CloudFront Functions are magnificent.

Lambda didn’t fail us. We outgrew its default behavior for a very specific, high-stakes workload. We cheated physics by keeping our processes warm, and in return, our p95 stopped stretching like hot taffy.

If your latency targets and traffic shape look anything like ours, please steal our tiny benchmark kit. Run a one-day canary. See what the numbers tell you. The goal isn’t to declare one tool a winner, but to spend less time arguing with physics and more time building things that people love.

September 2, 2025 by Cloud stuff DevOps stuff SRE stuff
NivelEpsilon
Exploring the Cloud: Insights into DevOps, Linux & Architectural Evolution

Recent Comments

No comments to show.