Lambda

When invisible limits beat warm Lambdas

My team had a problem. Or rather, we had a cause. A noble crusade that consumed our sprints, dominated our Slack channels, and haunted our architectural diagrams. We were on a relentless witch hunt for the dreaded Lambda cold start.

We treated those extra milliseconds of spin-up time like a personal insult from Jeff Bezos himself. We became amateur meteorologists, tracking “cold start storms” across regions. We had dashboards so finely tuned they could detect the faint, quantum flutter of an EC2 instance thinking about starting up. We proudly spent over $3,000 a month on provisioned concurrency¹, a financial sacrifice to the gods of AWS to keep our functions perpetually toasty.

We had done it. Cold starts were a solved problem. We celebrated with pizza and self-congratulatory Slack messages. The system was invincible.

Or so we thought.

The 2:37 am wake-up call

It was a Tuesday, of course. The kind of quiet, unassuming Tuesday that precedes all major IT disasters. At 2:37 AM, my phone began its unholy PagerDuty screech. The alert was as simple as it was terrifying: “API timeouts.”

I stumbled to my laptop, heart pounding, expecting to see a battlefield. Instead, I saw a paradox.

The dashboards were an ocean of serene green.

Cold starts? 0%. Our $3,000 was working perfectly. Our Lambdas were warm, cozy, and ready for action.
Lambda health? 100%. Every function was executing flawlessly, not an error in sight.
Database queries? 100% failure rate.

It was like arriving at a restaurant to find the chefs in the kitchen, knives sharpened and stoves hot, but not a single plate of food making it to the dining room. Our Lambdas were warm, our dashboards were green, and our system was dying. It turns out that for $3,000 a month, you can keep your functions perfectly warm while they helplessly watch your database burn to the ground.

We had been playing Jenga with AWS’s invisible limits, and someone had just pulled the wrong block.

Villain one, The great network card famine

Every Lambda function that needs to talk to services within your VPC, like a database, requires a virtual network card, an Elastic Network Interface (ENI). It’s the function’s physical connection to the world. And here’s the fun part that AWS tucks away in its documentation: your account has a default, region-wide limit on these. Usually around 250.

We discovered this footnote from 2018 when the Marketing team, in a brilliant feat of uncoordinated enthusiasm, launched a flash promo.

Our traffic surged. Lambda, doing its job beautifully, began to scale. 100 concurrent executions. 200. Then 300.

The 251st request didn’t fail. Oh no, that would have been too easy. Instead, it just… waited. For fourteen seconds. It was waiting in a silent, invisible line for AWS to slowly hand-carve a new network card from the finest, artisanal silicon.

Our “optimized” system had become a lottery.

The winners: Got an existing ENI and a zippy 200ms response.
The losers: Waited 14,000ms for a network card to materialize out of thin air, causing their request to time out.

The worst part? This doesn’t show up as a Lambda error. It just looks like your code is suddenly, inexplicably slow. We were hunting for a bug in our application, but the culprit was a bureaucrat in the AWS networking department.

Do this right now. Seriously. Open a terminal and check your limit. Don’t worry, we’ll wait.

# This command reveals the 'Maximum network interfaces per Region' quota.
# You might be surprised at what you find.
aws service-quotas get-service-quota \
  --service-code vpc \
  --quota-code L-F678F1CE

Villain two, The RDS proxy’s velvet rope policy

Having identified the ENI famine, we thought we were geniuses. But fixing that only revealed the next layer of our self-inflicted disaster. Our Lambdas could now get network cards, but they were all arriving at the database party at once, only to be stopped at the door.

We were using RDS Proxy, the service AWS sells as the bouncer for your database, managing connections so your Aurora instance doesn’t get overwhelmed. What we failed to appreciate is that this bouncer has its own… peculiar rules. The proxy itself has CPU limits. When hundreds of Lambdas tried to get a connection simultaneously, the proxy’s CPU spiked to 100%.

It didn’t crash. It just became incredibly, maddeningly slow. It was like a nightclub bouncer enforcing a strict one-in, one-out policy, not because the club was full, but because he could only move his arms so fast. The queue of connections grew longer and longer, each one timing out, while the database inside sat mostly idle, wondering where everybody went.

The humbling road to recovery

The fixes weren’t complex, but they were humbling. They forced us to admit that our beautiful, perfectly-tuned relational database architecture was, for some tasks, the wrong tool for the job.

The great VPC escape
For any Lambda that only needed to talk to public AWS services like S3 or SQS, we ripped it out of the VPC. This is Lambda 101, but we had put everything in the VPC for “security.” Moving them out meant they no longer needed an ENI to function. We implemented VPC Endpoints², allowing these functions to access AWS services over a private link without the ENI overhead.
RDS proxy triage
For the databases we couldn’t escape, we treated the proxy like the delicate, overworked bouncer it was. We massively over-provisioned the proxy instances, giving them far more CPU than they should ever need. We also implemented client-side jitter, a small, random delay before retrying a connection, to stop our Lambdas from acting like a synchronized mob storming the gates.
The nuclear option DynamoDB
For one critical, high-throughput service, we did the unthinkable. We migrated it from Aurora to DynamoDB. The hardest part wasn’t the code; it was the ego. It was admitting that the problem didn’t require a Swiss Army knife when all we needed was a hammer. The team’s reaction after the migration was telling: “Wait… you mean we don’t need to worry about connection pooling at all?” Every developer, after their first taste of NoSQL freedom.

The real lesson we learned

Obsessing over cold starts is like meticulously polishing the chrome on your car’s engine while the highway you’re on is crumbling into a sinkhole. It’s a visible, satisfying metric to chase, but it often distracts from the invisible, systemic limits that will actually kill you.

Yes, optimize your cold starts. Shave off those milliseconds. But only after you’ve pressure-tested your system for the real bottlenecks. The unsexy ones. The ones buried in AWS service quota pages and 5-year-old forum posts.

Stop micro-optimizing the 50ms you can see and start planning for the 14-second delays you can’t. We learned that the hard way, at 2:37 AM on a Tuesday.

¹ The official term for ‘setting a pile of money on fire to keep your functions toasty’.

² A fancy AWS term for ‘a private, secret tunnel to an AWS service so your Lambda doesn’t have to go out into the scary public internet’. It’s like an employee-only hallway in a giant mall.

Serverless without the wait

I once bought a five-minute rice cooker that spent four of those minutes warming up with a pathetic hum. It delivered the goods, eventually, but the promise felt… deceptive. For years, AWS Lambda felt like that gadget. It was the perfect kitchen tool for the odd jobs: a bit of glue code here, a light API there. It was the brilliant, quick-fire microwave of our architecture.

Then our little kitchen grew into a full-blown restaurant. Our “hot path”, the user checkout process, became the star dish on our menu. And our diners, quite rightly, expected it to be served hot and fast every time, not after a polite pause while the oven preheated. That polite pause was our cold start, and it was starting to leave a bad taste.

This isn’t a story about how we fell out of love with Lambda. We still adore it. This is the story of how we moved our main course to an industrial-grade, always-on stove. It’s about what we learned by obsessively timing every step of the process and why we still keep that trusty microwave around for the side dishes it cooks so perfectly. Because when your p95 latency needs to be boringly predictable, keeping the kitchen warm isn’t a preference; it’s a law of physics.

What forced us to remodel the kitchen

No single event pushed us over the edge. It was more of a slow-boiling frog situation, a gradual realization that our ambitions were outgrowing our tools. Three culprits conspired against our sub-300ms dream.

First, our traffic got moody. What used to be a predictable tide of requests evolved into sudden, sharp tsunamis during business hours. We needed a sea wall, not a bucket.

Second, our user expectations tightened. We set a rather tyrannical goal of a sub-300ms p95 for our checkout and search paths. Suddenly, the hundreds of milliseconds Lambda spent stretching and yawning before its first cup of coffee became a debt we couldn’t afford.

Finally, our engineers were getting tired. We found ourselves spending more time performing sacred rituals to appease the cold start gods, fiddling with layers, juggling provisioned concurrency, than we did shipping features our users actually cared about. When your mechanics spend more time warming up the engine than driving the car, you know something’s wrong.

The punchline isn’t that Lambda is “bad.” It’s that our requirements changed. When your performance target drops below the cost of a cold start plus dependency initialization, physics sends you a sternly worded letter.

Numbers don’t lie, but anecdotes do

We don’t ask you to trust our feelings. We ask you to trust the stopwatch. Replicate this experiment, adjust it for your own tech stack, and let the data do the talking. The setup below is what we used to get our own facts straight. All results are our measurements as of September 2025.

The test shape

Endpoint: Returns a simple 1 KB JSON payload.
Comparable Compute: Lambda set to 512 MB vs. an ECS Fargate container task with 0.5 vCPU and 1 GB of memory.
Load Profile: A steady, closed-loop 100 requests per second (RPS) for 10 minutes.
Metrics Reported: p50, p90, p95, p99 latency, and the dreaded error rate.

Our trusty tools

Load Generator: The ever-reliable k6.
Metrics: A cocktail of CloudWatch and Prometheus.
Dashboards: Grafana, to make the pretty charts that managers love.

Your numbers will be different. That’s the entire point. Run the tests, get your own data, and then make a decision based on evidence, not a blog post (not even this one).

Where our favorite gadget struggled

Under the harsh lights of our benchmark, Lambda’s quirks on our hot path became impossible to ignore.

Cold start spikes: Provisioned Concurrency can tame these, but it’s like hiring a full-time chauffeur to avoid a random 10-minute wait for a taxi. It costs you a constant fee, and during a real rush hour, you might still get stuck in traffic.
The startup toll: Initializing SDKs and warming up connections added tens to hundreds of milliseconds. This “entry fee” was simply too high to hide under our 300ms p95 goal.
The debugging labyrinth: Iterating was slow. Local emulators helped, but parity was a myth that occasionally bit us. Debugging felt like detective work with half the clues missing.

Lambda continues to be a genius for event glue, sporadic jobs, and edge logic. It just stopped being the right tool to serve our restaurant’s most popular dish at rush hour.

Calling in the heavy artillery

We moved our high-traffic endpoints to container-native services. For us, that meant ECS on Fargate fronted by an Application Load Balancer (ALB). The core idea is simple: keep a few processes warm and ready at all times.

Here’s why it immediately helped:

Warm processes: No more cold start roulette. Our application was always awake, connection pools were alive, and everything was ready to go instantly.
Standardized packaging: We traded ZIP files for standard Docker images. What we built and tested on our laptops was, byte for byte, what we shipped to production.
Civilized debugging: We could run the exact same image locally and attach a real debugger. It was like going from candlelight to a floodlight.
Smarter scaling: We could maintain a small cadre of warm tasks as a baseline and then scale out aggressively during peaks.

A quick tale of the tape

Here’s a simplified look at how the two approaches stacked up for our specific needs.

Our surprisingly fast migration plan

We did this in days, not weeks. The key was to be pragmatic, not perfect.

1. Pick your battles: We chose our top three most impactful endpoints with the worst p95 latency.

2. Put it in a box: We converted the function handler into a tiny web service. It’s less dramatic than it sounds.

# Dockerfile (Node.js example)
FROM node:22-slim
WORKDIR /usr/src/app

COPY package*.json ./
RUN npm ci --only=production

COPY . .

ENV NODE_ENV=production PORT=3000
EXPOSE 3000
CMD [ "node", "server.js" ]

// server.js
const http = require('http');
const port = process.env.PORT || 3000;

const server = http.createServer((req, res) => {
  if (req.url === '/health') {
    res.writeHead(200, { 'Content-Type': 'text/plain' });
    return res.end('ok');
  }

  // Your actual business logic would live here
  const body = JSON.stringify({ success: true, timestamp: Date.now() });
  res.writeHead(200, { 'Content-Type': 'application/json' });
  res.end(body);
});

server.listen(port, () => {
  console.log(`Server listening on port ${port}`);
});

3. Set up the traffic cop: We created a new target group for our service and pointed a rule on our Application Load Balancer to it.

{
  "family": "payment-api",
  "networkMode": "awsvpc",
  "cpu": "512",
  "memory": "1024",
  "requiresCompatibilities": ["FARGATE"],
  "executionRoleArn": "arn:aws:iam::987654321098:role/ecsTaskExecutionRole",
  "taskRoleArn": "arn:aws:iam::987654321098:role/paymentTaskRole",
  "containerDefinitions": [
    {
      "name": "app-container",
      "image": "[987654321098.dkr.ecr.us-east-1.amazonaws.com/payment-api:2.1.0](https://987654321098.dkr.ecr.us-east-1.amazonaws.com/payment-api:2.1.0)",
      "portMappings": [{ "containerPort": 3000, "protocol": "tcp" }],
      "environment": [{ "name": "NODE_ENV", "value": "production" }]
    }
  ]
}

4. The canary in the coal mine: We used weighted routing to dip our toes in the water. We started by sending just 5% of traffic to the new container service.

# Terraform Route 53 weighted canary
resource "aws_route53_record" "api_primary_lambda" {
  zone_id = var.zone_id
  name    = "api.yourapp.com"
  type    = "A"

  alias {
    name                   = aws_api_gateway_domain_name.main.cloudfront_domain_name
    zone_id                = aws_api_gateway_domain_name.main.cloudfront_zone_id
    evaluate_target_health = true
  }

  set_identifier = "primary-lambda-path"
  weight         = 95
}

resource "aws_route53_record" "api_canary_container" {
  zone_id = var.zone_id
  name    = "api.yourapp.com"
  type    = "A"

  alias {
    name                   = aws_lb.main_alb.dns_name
    zone_id                = aws_lb.main_alb.zone_id
    evaluate_target_health = true
  }

  set_identifier = "canary-container-path"
  weight         = 5
}

5. Stare at the graphs: For one hour, we watched four numbers like hawks: p95 latency, error rates, CPU/memory headroom on the new service, and our estimated cost per million requests.

6. Go all in (or run away): The graphs stayed beautifully, boringly flat. So we shifted to 50%, then 100%. The whole affair was done in an afternoon.

The benchmark kit you can steal

Don’t just read about it. Run a quick test yourself.

// k6 script (save as test.js)
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  vus: 100,
  duration: '5m',
  thresholds: {
    'http_req_duration': ['p(95)<250'], // Aim for a 250ms p95
    'checks': ['rate>0.999'],
  },
};

export default function () {
  const url = __ENV.TARGET_URL || '[https://api.yourapp.com/checkout/v2/quote](https://api.yourapp.com/checkout/v2/quote)';
  const res = http.get(url);
  check(res, { 'status is 200': r => r.status === 200 });
  sleep(0.2); // Small pause between requests
}

Run it from your terminal like this:

k6 run -e TARGET_URL=https://your-canary-endpoint.com test.js

Our results for context

These aren’t universal truths; they are snapshots of our world. Your mileage will vary.

The numbers in bold are what kept us up at night and what finally let us sleep. For our steady traffic, the always-on container was not only faster and more reliable, but it was also shaping up to be cheaper.

Lambda is still in our toolbox

We didn’t throw the microwave out. We just stopped using it to cook the Thanksgiving turkey. Here’s where we still reach for Lambda without a second thought:

Sporadic or bursty workloads: Those once-a-day reports or rare event handlers are perfect for scale-to-zero.
Event glue: It’s the undisputed champion of transforming S3 puts, reacting to DynamoDB streams, and wiring up EventBridge.
Edge logic: For tiny header manipulations or rewrites, Lambda@Edge and CloudFront Functions are magnificent.

Lambda didn’t fail us. We outgrew its default behavior for a very specific, high-stakes workload. We cheated physics by keeping our processes warm, and in return, our p95 stopped stretching like hot taffy.

If your latency targets and traffic shape look anything like ours, please steal our tiny benchmark kit. Run a one-day canary. See what the numbers tell you. The goal isn’t to declare one tool a winner, but to spend less time arguing with physics and more time building things that people love.

September 2, 2025 by Fernando SRE Cloud stuff DevOps stuff SRE stuff

AWS Step Functions for absolute beginners

While everyone else is busy wrapping presents and baking cookies, we’re going to unwrap something even more exciting: the world of AWS Step Functions. Now, I know what you might be thinking: “Step Functions? That sounds about as fun as getting socks for Christmas.” But trust me, this is way cooler than it sounds.

Imagine you’re Santa Claus for a second. You’ve got this massive list of kids, a whole bunch of elves, and a sleigh full of presents. How do you make sure everything gets done on time? You need a plan, a workflow. You wouldn’t just tell the elves, “Go do stuff!” and hope for the best, right? No, you’d say, “First, check the list. Then, build the toys. Next, wrap the presents. Finally, load up the sleigh.”

That’s essentially what AWS Step Functions does for your code in the cloud. It’s like a super-organized Santa Claus for your computer programs, ensuring everything happens in the right order, at the right time.

Why use AWS Step Functions? Because even Santa needs a plan

What are Step Functions anyway?

Think of AWS Step Functions as a flowchart on steroids. It’s a service that lets you create visual workflows for your applications. These workflows, called “state machines,” are made up of different steps, or “states,” that tell your application what to do and when to do it. These steps can be anything from simple tasks to complex operations, and they often involve our little helpers called AWS Lambda functions.

A quick chat about AWS Lambda

Before we go further, let’s talk about Lambdas. Imagine you have a tiny robot that’s really good at one specific task, like tying bows on presents. That’s a Lambda function. It’s a small piece of code that does one thing and does it well. You can have lots of these little robots, each doing their own thing, and Step Functions helps you organize them into a productive team. They are like the Christmas elves of the cloud!

Why orchestrate multiple Lambdas?

Now, you might ask, “Why not just have one big, all-knowing Lambda function that does everything?” Well, you could, but it would be like having one giant elf try to build every toy, wrap every present, and load the sleigh all by themselves. It would be chaotic, and hard to manage, and if that elf gets tired (or your code breaks), everything grinds to a halt.

Having specialized elves (or Lambdas) for each task is much better. One is for checking the list, one is for building toys, one is for wrapping, and so on. This way, if one elf needs a break (or a code update), the others can keep working. That’s the beauty of breaking down complex tasks into smaller, manageable steps.

Our scenario Santa’s data dilemma

Let’s imagine Santa has a modern problem. He’s got a big list of kids and their gift requests, but it’s all in a digital file (a JSON file, to be precise) stored in a magical cloud storage called S3 (Simple Storage Service). His goal is to read this list, make sure it’s not corrupted, add some extra Christmas magic to each request (like a “Ho Ho Ho” stamp), and then store the updated list back in S3. Finally, he wants a little notification to make sure everything went smoothly.

Breaking down the task with multiple lambdas

Here’s how we can break down Santa’s task into smaller, Lambda-sized jobs:

Validation Lambda: This little helper checks the list to make sure it’s in the right format and that no naughty kids are trying to sneak extra presents onto the list.
Transformation Lambda: This is where the magic happens. This Lambda adds that special “Ho Ho Ho” to each gift request, making sure every kid gets a personalized touch.
Notification Lambda: This is our town crier. Once everything is done, this Lambda shouts “Success!” (or sends a more sophisticated message) to let Santa know the job is complete.

Step Functions Santa’s master plan

This is where Step Functions comes in. It’s the conductor of our Lambda orchestra. It makes sure each Lambda function runs in the right order, passing the list from one Lambda to the next like a relay race.

Our High-Level architecture

Let’s draw a simple picture of what’s happening (even Santa loves a good diagram):

The data’s journey

The list (JSON file) lands in an S3 bucket.
This triggers our Step Functions workflow.
The Validation Lambda grabs the list, checks it, and passes the validated list to the Transformation Lambda.
The Transformation Lambda works its magic, adds the “Ho Ho Ho,” and saves the new list to another S3 bucket.
Finally, the Notification Lambda sends out a message confirming success.

The secret sauce passing data between steps

Step Functions automatically passes the output from each step as input to the next. It’s like each elf handing the partially completed present to the next elf in line. This is a crucial part of what makes Step Functions so powerful.

A look at each Lambda function

Let’s peek inside each of our Lambda functions. Don’t worry; we’ll keep it simple.

The list checker validation Lambda

This Lambda, written in Python (a very friendly programming language), does the following:

Downloads the list from S3.
Checks if the list is in the correct format (like making sure it’s actually a list and not a drawing of a reindeer).
If something’s wrong, it raises an error (handled gracefully by Step Functions).
If everything’s good, it returns the validated list.

Adding Christmas magic with the transformation Lambda

This Lambda receives the validated list and:

Adds that special “Ho Ho Ho” to each gift request.
Saves the new, transformed list to a new file in S3.
Returns the location of the newly created file.

Spreading the news with the notification Lambda

This Lambda gets the path to the transformed file and:

Could send a message to Santa’s phone, write “Success!” in the snow, or simply print a message in the cloud logs.
Marks the end of our workflow.

Configuring the state machine

Now, how do we tell Step Functions what to do? We use something called the Amazon States Language (ASL), which is just a fancy way of describing our workflow in a JSON format. Here’s a simplified snippet:

{
  "StartAt": "ValidateData",
  "States": {
    "ValidateData": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:123456789012:function:ValidateData",
      "Next": "TransformData"
    },
    "TransformData": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:123456789012:function:TransformData",
      "Next": "Notify"
    },
    "Notify": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:123456789012:function:Notify",
      "End": true
    }
  }
}

Don’t be scared by the code! It’s just a structured way of saying:

Start with “ValidateData.”
Then go to “TransformData.”
Finally, go to “Notify” and we’re done.

Each “Resource” is the address of our Lambda function in the AWS world.

Error handling for dropped tasks

What happens if an elf drops a present? Step Functions can handle that! We can tell it to retry the step or go to a special “Fix It” state if something goes wrong.

Passing output between steps

Remember how we talked about passing data between steps? Here’s a simplified example of how we tell Step Functions to do that:

"TransformData": {
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:123456789012:function:TransformData",
  "InputPath": "$.validatedData", 
  "OutputPath": "$.transformedData",
  "Next": "Notify"
}

This tells the “TransformData” step to take the “validatedData” from the previous step’s output and put its output in “transformedData.”

Making sure everything works before the big day

Before we unleash our workflow on the world (or Santa’s list), we need to make absolutely sure it works as expected. Testing is like a dress rehearsal for Christmas Eve, ensuring every elf knows their part and Santa’s sleigh is ready to fly.

Two levels of testing

We’ll approach testing in two ways:

Testing each Lambda individually (Local tests):
- Think of this as quality control for each elf. Before they join the assembly line, we need to make sure each Lambda function does its job correctly in isolation.
- We can do this right from the AWS Management Console. Simply find your Lambda function, and look for a “Test” tab or button.
- You’ll be able to create test events, which are like sample inputs for your Lambda. For example, for our Validation Lambda, you could create a test event with a well-formatted JSON and another with a deliberately incorrect JSON to see if the Lambda catches the error.
- Run the test and check the output. Did the Lambda behave as expected? Did it return the correct data or the proper error message?
- Alternatively, if you’re comfortable with the command line, you can use the AWS CLI (Command Line Interface) to invoke your Lambdas with test data. This offers more flexibility for advanced testing.
- It is very important to test each Lambda with different types of inputs to make sure it behaves well under diverse circumstances.
Testing the entire workflow (End-to-End test):
- This is the grand rehearsal, where we test the whole process from start to finish.
- First, prepare a sample JSON file that represents a typical Santa’s list. Make it realistic but simple enough for easy testing.
- Upload this file to your designated S3 bucket. This should automatically trigger your Step Functions workflow.
- Now, head over to the Step Functions section in the AWS Management Console. Find your state machine and look for the execution history. You should see a new execution that corresponds to your test.
- Click on the execution. You’ll see a visual diagram of your workflow, with each step highlighted as it’s executed. This is like tracking Santa’s sleigh in real time!
- Pay close attention to each step. Did it succeed? Did it take roughly the amount of time you expected? If a step fails, the diagram will show you where the problem occurred.
- Once the workflow is complete, check your output S3 bucket. Is the transformed file there? Is it correctly modified according to your Transformation Lambda’s logic?
- Finally, verify that your Notification Lambda did its job. Did it log the success message? Did it send a notification if that’s how you configured it?

Why both types of testing matter

You might wonder, “Why do we need both local and end-to-end tests?” Here’s the deal:

Local tests help you catch problems early on, at the individual component level. It’s much easier to fix a problem with a single Lambda than to debug a complex workflow with multiple failing parts.
End-to-end tests ensure that all the components work together seamlessly. They verify that the data is passed correctly between steps and that the overall workflow produces the desired outcome.

Debugging tips

If a step fails during the end-to-end test, click on the failed step in the Step Functions execution diagram. You’ll often see an error message that can help you pinpoint the issue.
Check the CloudWatch Logs for your Lambda functions. These logs contain valuable information about what happened during the execution, including any error messages or debug output you’ve added to your code.

Iterate and refine

Testing is not a one-time thing. As you develop your workflow, you’ll likely make changes and improvements. Each time you make a significant change, repeat your tests to ensure everything still works as expected. Remember: a well-tested workflow is a reliable workflow. By thoroughly testing our Step Functions workflow, we’re making sure that Santa’s list (and our application) is in good hands. Now, let’s get testing!

Step Functions or single Lambdas?

Maintainability and visibility

Step Functions makes it super easy to see what’s happening in your workflow. It’s like having a map of Santa’s route on Christmas Eve. This makes it much easier to find and fix problems.

Complexity

For simple tasks, a single Lambda might be enough. But as soon as you have multiple steps that need to happen in a specific order, Step Functions is your best friend.

Beyond Christmas Eve

Key takeaways

Step Functions is a powerful way to chain together Lambda functions in a visual, trackable, and error-tolerant workflow. It’s like having a super-organized Santa Claus for your cloud applications.

Potential improvements

We could add more steps, like extra validation or an automated email to parents. We could use other AWS services like SNS (Simple Notification Service) for more advanced notifications or DynamoDB for storing even more data.

Final words

This was a simple example, but the same ideas apply to much more complex, real-world applications. Step Functions can handle massive workflows with thousands of steps, making it a crucial tool for any aspiring cloud architect.

So, there you have it! You’ve now seen how AWS Step Functions can orchestrate AWS Lambdas to complete a task, just like Santa orchestrates his elves on Christmas Eve. And hopefully, it was a bit more exciting than getting socks for Christmas. 😊

December 24, 2024 by Fernando SRE Cloud stuff

Building a serverless image processor with AWS Step Functions

Let’s build something awesome together, an image-processing application using AWS Step Functions. Don’t worry if that sounds complicated; I’ll break it down step by step, just like explaining how a bicycle works. Ready? Let’s go for it.

1. Introduction

Imagine you’re running a photo gallery website where users upload their precious memories, and you need to process these images automatically, resize them, add filters, and optimize them for the web. That sounds like a lot of work, right? Well, that’s exactly what we’re going to build today.

What We’re building

We’re creating a serverless application that will:

Accept image uploads from users.
Process these images in various ways.
Store the results safely.
Notify users when the process is complete.

Here’s a simplified view of the architecture:

User -> S3 Bucket -> Step Functions -> Lambda Functions -> Processed Images

What You’ll need

An AWS account (don’t worry, most of this fits in the free tier).
Basic understanding of AWS (if you can create an S3 bucket, you’re ready).
A cup of coffee (or tea, I won’t judge!).

2. Designing the architecture

Let’s think about this as a building with LEGO blocks. Each AWS service is a different block type, and we’ll connect them to create something awesome.

Our building blocks:

S3 Buckets: Think of these as fancy folders where we’ll store the images.
Lambda Functions: These are our “workers” that will process the images.
Step Functions: This is the “manager” that coordinates everything.
DynamoDB: This will act as a notebook to keep track of what we’ve done.

Here’s the workflow:

The user uploads an image to S3.
S3 triggers our Step Function.
Step Function coordinates various Lambda functions to:
- Validate the image.
- Resize it.
- Apply filters.
- Optimize it.
Finally, the processed image is stored, and the user is notified.

3. Step-by-Step implementation

3.1 Setting Up the S3 Bucket

First, we’ll set up our image storage. Think of this as creating a filing cabinet for our photos.

aws s3 mb s3://my-image-processor-bucket

Next, configure it to trigger the Step Function whenever a file is uploaded. Here’s the event configuration:

{
    "LambdaFunctionConfigurations": [{
        "LambdaFunctionArn": "arn:aws:lambda:region:account:function:trigger-step-function",
        "Events": ["s3:ObjectCreated:*"]
    }]
}

3.2 Creating the Lambda Functions

Now, let’s create the Lambda functions that will process the images. Each one has a specific job:

Image Validator
This function checks if the uploaded image is valid (e.g., correct format, not corrupted).

import boto3
from PIL import Image
import io

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    
    bucket = event['bucket']
    key = event['key']
    
    try:
        image_data = s3.get_object(Bucket=bucket, Key=key)['Body'].read()
        image = Image.open(io.BytesIO(image_data))
        
        return {
            'statusCode': 200,
            'isValid': True,
            'metadata': {
                'format': image.format,
                'size': image.size
            }
        }
    except Exception as e:
        return {
            'statusCode': 400,
            'isValid': False,
            'error': str(e)
        }

Image Resizer
This function resizes the image to a specific target size.

from PIL import Image
import boto3
import io

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    
    bucket = event['bucket']
    key = event['key']
    target_size = (800, 600)  # Example size
    
    try:
        image_data = s3.get_object(Bucket=bucket, Key=key)['Body'].read()
        image = Image.open(io.BytesIO(image_data))
        resized_image = image.resize(target_size, Image.LANCZOS)
        
        buffer = io.BytesIO()
        resized_image.save(buffer, format=image.format)
        s3.put_object(
            Bucket=bucket,
            Key=f"resized/{key}",
            Body=buffer.getvalue()
        )
        
        return {
            'statusCode': 200,
            'resizedImage': f"resized/{key}"
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'error': str(e)
        }

3.3 Setting Up Step Functions

Now comes the fun part, setting up our workflow coordinator. Step Functions will manage the flow, ensuring each image goes through the right steps.

{
  "Comment": "Image Processing Workflow",
  "StartAt": "ValidateImage",
  "States": {
    "ValidateImage": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account:function:validate-image",
      "Next": "ImageValid",
      "Catch": [{
        "ErrorEquals": ["States.ALL"],
        "Next": "NotifyError"
      }]
    },
    "ImageValid": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.isValid",
          "BooleanEquals": true,
          "Next": "ProcessImage"
        }
      ],
      "Default": "NotifyError"
    },
    "ProcessImage": {
      "Type": "Parallel",
      "Branches": [
        {
          "StartAt": "ResizeImage",
          "States": {
            "ResizeImage": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:region:account:function:resize-image",
              "End": true
            }
          }
        },
        {
          "StartAt": "ApplyFilters",
          "States": {
            "ApplyFilters": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:region:account:function:apply-filters",
              "End": true
            }
          }
        }
      ],
      "Next": "OptimizeImage"
    },
    "OptimizeImage": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account:function:optimize-image",
      "Next": "NotifySuccess"
    },
    "NotifySuccess": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account:function:notify-success",
      "End": true
    },
    "NotifyError": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account:function:notify-error",
      "End": true
    }
  }
}

4. Error Handling and Resilience

Let’s make our application resilient to errors.

Retry Policies

For each Lambda invocation, we can add retry policies to handle transient errors:

{
  "Retry": [{
    "ErrorEquals": ["States.TaskFailed"],
    "IntervalSeconds": 3,
    "MaxAttempts": 2,
    "BackoffRate": 1.5
  }]
}

Error Notifications

If something goes wrong, we’ll want to be notified:

import boto3

def notify_error(event, context):
    sns = boto3.client('sns')
    
    error_message = f"Error processing image: {event['error']}"
    
    sns.publish(
        TopicArn='arn:aws:sns:region:account:image-processing-errors',
        Message=error_message,
        Subject='Image Processing Error'
    )

5. Optimizations and Best Practices

Lambda Configuration

Memory: Set memory based on image size. 1024MB is a good starting point.
Timeout: Set reasonable timeout values, like 30 seconds for image processing.
Environment Variables: Use these to configure Lambda functions dynamically.

Cost Optimization

Use Step Functions Express Workflows for high-volume processing.
Implement caching for frequently accessed images.
Clean up temporary files in /tmp to avoid running out of space.

Security

Use IAM policies to ensure only necessary access is granted to S3:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::my-image-processor-bucket/*"
        }
    ]
}

6. Deployment

Finally, let’s deploy everything using AWS SAM, which simplifies the deployment process.

Project Structure

image-processor/
├── template.yaml
├── functions/
│   ├── validate/
│   │   └── app.py
│   ├── resize/
│   │   └── app.py
└── statemachine/
    └── definition.asl.json

SAM Template

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
  ImageProcessorStateMachine:
    Type: AWS::Serverless::StateMachine
    Properties:
      DefinitionUri: statemachine/definition.asl.json
      Policies:
        - LambdaInvokePolicy:
            FunctionName: !Ref ValidateFunction
        - LambdaInvokePolicy:
            FunctionName: !Ref ResizeFunction

  ValidateFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: functions/validate/
      Handler: app.lambda_handler
      Runtime: python3.9
      MemorySize: 1024
      Timeout: 30

  ResizeFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: functions/resize/
      Handler: app.lambda_handler
      Runtime: python3.9
      MemorySize: 1024
      Timeout: 30

Deployment Commands

# Build the application
sam build

# Deploy (first time)
sam deploy --guided

# Subsequent deployments
sam deploy

After deployment, test your application by uploading an image to your S3 bucket:

aws s3 cp test-image.jpg s3://my-image-processor-bucket/raw/

Yeah, You have built a robust, serverless image-processing application. The beauty of this setup is its scalability, from a handful of images to thousands, it can handle them all seamlessly.

And like any good recipe, feel free to tweak the process to fit your needs. Maybe you want to add extra processing steps or fine-tune the Lambda configurations, there’s always room for experimentation.

October 24, 2024 by Fernando SRE Cloud stuff

Scaling for Success. Cost-Effective Cloud Architectures on AWS

One of the most exciting aspects of cloud computing is the promise of scalability, the ability to expand or contract resources to meet demand. But how do you design an architecture that can handle unexpected traffic spikes without breaking the bank during quieter periods? This question often comes up in AWS Solution Architect interviews, and for good reason. It’s a core challenge that many businesses face when moving to the cloud. Let’s explore some AWS services and strategies that can help you achieve both scalability and cost efficiency.

Building a Dynamic and Cost-Aware AWS Architecture

Imagine your application is like a bustling restaurant. During peak hours, you need a full staff and all tables ready. But during off-peak times, you don’t want to be paying for idle resources. Here’s how we can translate this concept into a scalable AWS architecture:

Auto Scaling Groups (ASGs): Think of ASGs as your restaurant’s staffing manager. They automatically adjust the number of EC2 instances (your servers) based on predefined rules. If your website traffic suddenly spikes, ASGs will spin up additional instances to handle the load. When traffic dies down, they’ll scale back, saving you money. You can even combine ASGs with Spot Instances for even greater cost savings.
Amazon EC2 Spot Instances: These are like the temporary staff you might hire during a particularly busy event. Spot Instances let you take advantage of unused EC2 capacity at a much lower cost. If your demand is unpredictable, Spot Instances can be a great way to save money while ensuring you have enough resources to handle peak loads.
Amazon Lambda: Lambda is your kitchen staff that only gets paid when they’re cooking, and they’re really good at their job, they can whip up a dish in under 15 minutes! It’s a serverless compute service that runs your code in response to events (like a new file being uploaded or a database change). You only pay for the compute time you actually use, making it ideal for sporadic or unpredictable workloads.
AWS Fargate: Fargate is like having a catering service handle your entire kitchen operation. It’s a serverless compute engine for containers, meaning you don’t have to worry about managing the underlying servers. Fargate automatically scales your containerized applications based on demand, and you only pay for the resources your containers consume.

How the Pieces Fit Together

Now, let’s see how these services can work together in harmony:

Core Application on EC2 with Auto Scaling: Your main application might run on EC2 instances within an Auto Scaling Group. You can configure this group to monitor the CPU utilization of your servers and automatically launch new instances if the average CPU usage reaches a threshold, such as 75% (this is known as a Target Tracking Scaling Policy). This ensures you always have enough servers running to handle the current load, even during unexpected traffic spikes.
Spot Instances for Cost Optimization: To save costs, you could configure your Auto Scaling Group to use Spot Instances whenever possible. This allows you to take advantage of lower prices while still scaling up when needed. Importantly, you’ll also want to set up a recovery policy within your Auto Scaling Group. This policy ensures that if Spot Instances are not available (due to high demand or price fluctuations), your Auto Scaling Group will automatically launch On-Demand Instances instead. This way, you can reliably meet your application’s resource needs even when Spot Instances are unavailable.
Lambda for Event-Driven Tasks: Lambda functions excel at handling event-driven tasks that don’t require a constantly running server. For example, when a new image is uploaded to your S3 bucket, you can trigger a Lambda function to automatically resize it or convert it to a different format. Similarly, Lambda can be used to send notifications to users when certain events occur in your application, such as a new order being placed or a payment being processed. Since Lambda functions are only active when triggered, they can significantly reduce your costs compared to running dedicated EC2 instances for these tasks.
Fargate for Containerized Microservices: If your application is built using microservices, you can run them in containers on Fargate. This eliminates the need to manage servers and allows you to scale each microservice independently. By decoupling your microservices and using Amazon Simple Queue Service (SQS) queues for communication, you can ensure that even under heavy load, all requests will be handled and none will be lost. For applications where the order of operations is critical, such as financial transactions or order processing, you can use FIFO (First-In-First-Out) SQS queues to maintain the exact order of messages.

Monitoring and Optimization: Imagine having a restaurant manager who constantly monitors how busy the restaurant is, how much food is being wasted, and how satisfied the customers are. This is what Amazon CloudWatch does for your AWS environment. It provides detailed metrics and alarms, allowing you to fine-tune your scaling policies and optimize your resource usage. With CloudWatch, you can visualize the health and performance of your entire AWS infrastructure at a glance through intuitive dashboards and graphs. These visualizations make it easy to identify trends, spot potential issues, and make informed decisions about resource allocation and optimization.

The Outcome, A Satisfied Customer and a Healthy Bottom Line

By combining these AWS services and strategies, you can build a cloud architecture that is both scalable and cost-effective. This means your application can gracefully handle unexpected traffic spikes, ensuring a smooth user experience even during peak demand. At the same time, you won’t be paying for idle resources during quieter periods, keeping your cloud costs under control.

Final Analysis

Designing for scalability and cost efficiency is a fundamental aspect of cloud architecture. By leveraging AWS services like Auto Scaling, EC2 Spot Instances, Lambda, and Fargate, you can create a dynamic and responsive environment that adapts to your application’s needs. Remember, the key is to understand your workload patterns and choose the right tools for the job. With careful planning and the right AWS services, you can build a cloud architecture that is both powerful and cost-effective, setting your business up for success in the cloud and in the restaurant. 😉

July 24, 2024 by Fernando SRE Cloud stuff