Building a serverless image processor with AWS Step Functions

Let’s build something awesome together, an image-processing application using AWS Step Functions. Don’t worry if that sounds complicated; I’ll break it down step by step, just like explaining how a bicycle works. Ready? Let’s go for it.

1. Introduction

Imagine you’re running a photo gallery website where users upload their precious memories, and you need to process these images automatically, resize them, add filters, and optimize them for the web. That sounds like a lot of work, right? Well, that’s exactly what we’re going to build today.

What We’re building

We’re creating a serverless application that will:

• Accept image uploads from users.
• Process these images in various ways.
• Store the results safely.
• Notify users when the process is complete.

Here’s a simplified view of the architecture:

User -> S3 Bucket -> Step Functions -> Lambda Functions -> Processed Images

What You’ll need

• An AWS account (don’t worry, most of this fits in the free tier).
• Basic understanding of AWS (if you can create an S3 bucket, you’re ready).
• A cup of coffee (or tea, I won’t judge!).

2. Designing the architecture

Let’s think about this as a building with LEGO blocks. Each AWS service is a different block type, and we’ll connect them to create something awesome.

Our building blocks:

• S3 Buckets: Think of these as fancy folders where we’ll store the images.
• Lambda Functions: These are our “workers” that will process the images.
• Step Functions: This is the “manager” that coordinates everything.
• DynamoDB: This will act as a notebook to keep track of what we’ve done.

Here’s the workflow:

1. The user uploads an image to S3.
2. S3 triggers our Step Function.
3. Step Function coordinates various Lambda functions to:
   • Validate the image.
   • Resize it.
   • Apply filters.
   • Optimize it.
4. Finally, the processed image is stored, and the user is notified.

3. Step-by-Step implementation

3.1 Setting Up the S3 Bucket

First, we’ll set up our image storage. Think of this as creating a filing cabinet for our photos.

aws s3 mb s3://my-image-processor-bucket

Next, configure it to trigger the Step Function whenever a file is uploaded. Here’s the event configuration:

{
    "LambdaFunctionConfigurations": [{
        "LambdaFunctionArn": "arn:aws:lambda:region:account:function:trigger-step-function",
        "Events": ["s3:ObjectCreated:*"]
    }]
}

3.2 Creating the Lambda Functions

Now, let’s create the Lambda functions that will process the images. Each one has a specific job:

Image Validator
This function checks if the uploaded image is valid (e.g., correct format, not corrupted).

import boto3
from PIL import Image
import io

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    
    bucket = event['bucket']
    key = event['key']
    
    try:
        image_data = s3.get_object(Bucket=bucket, Key=key)['Body'].read()
        image = Image.open(io.BytesIO(image_data))
        
        return {
            'statusCode': 200,
            'isValid': True,
            'metadata': {
                'format': image.format,
                'size': image.size
            }
        }
    except Exception as e:
        return {
            'statusCode': 400,
            'isValid': False,
            'error': str(e)
        }

Image Resizer
This function resizes the image to a specific target size.

from PIL import Image
import boto3
import io

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    
    bucket = event['bucket']
    key = event['key']
    target_size = (800, 600)  # Example size
    
    try:
        image_data = s3.get_object(Bucket=bucket, Key=key)['Body'].read()
        image = Image.open(io.BytesIO(image_data))
        resized_image = image.resize(target_size, Image.LANCZOS)
        
        buffer = io.BytesIO()
        resized_image.save(buffer, format=image.format)
        s3.put_object(
            Bucket=bucket,
            Key=f"resized/{key}",
            Body=buffer.getvalue()
        )
        
        return {
            'statusCode': 200,
            'resizedImage': f"resized/{key}"
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'error': str(e)
        }

3.3 Setting Up Step Functions

Now comes the fun part, setting up our workflow coordinator. Step Functions will manage the flow, ensuring each image goes through the right steps.

{
  "Comment": "Image Processing Workflow",
  "StartAt": "ValidateImage",
  "States": {
    "ValidateImage": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account:function:validate-image",
      "Next": "ImageValid",
      "Catch": [{
        "ErrorEquals": ["States.ALL"],
        "Next": "NotifyError"
      }]
    },
    "ImageValid": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.isValid",
          "BooleanEquals": true,
          "Next": "ProcessImage"
        }
      ],
      "Default": "NotifyError"
    },
    "ProcessImage": {
      "Type": "Parallel",
      "Branches": [
        {
          "StartAt": "ResizeImage",
          "States": {
            "ResizeImage": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:region:account:function:resize-image",
              "End": true
            }
          }
        },
        {
          "StartAt": "ApplyFilters",
          "States": {
            "ApplyFilters": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:region:account:function:apply-filters",
              "End": true
            }
          }
        }
      ],
      "Next": "OptimizeImage"
    },
    "OptimizeImage": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account:function:optimize-image",
      "Next": "NotifySuccess"
    },
    "NotifySuccess": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account:function:notify-success",
      "End": true
    },
    "NotifyError": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account:function:notify-error",
      "End": true
    }
  }
}

4. Error Handling and Resilience

Let’s make our application resilient to errors.

Retry Policies

For each Lambda invocation, we can add retry policies to handle transient errors:

{
  "Retry": [{
    "ErrorEquals": ["States.TaskFailed"],
    "IntervalSeconds": 3,
    "MaxAttempts": 2,
    "BackoffRate": 1.5
  }]
}

Error Notifications

If something goes wrong, we’ll want to be notified:

import boto3

def notify_error(event, context):
    sns = boto3.client('sns')
    
    error_message = f"Error processing image: {event['error']}"
    
    sns.publish(
        TopicArn='arn:aws:sns:region:account:image-processing-errors',
        Message=error_message,
        Subject='Image Processing Error'
    )

5. Optimizations and Best Practices

Lambda Configuration

• Memory: Set memory based on image size. 1024MB is a good starting point.
• Timeout: Set reasonable timeout values, like 30 seconds for image processing.
• Environment Variables: Use these to configure Lambda functions dynamically.

Cost Optimization

• Use Step Functions Express Workflows for high-volume processing.
• Implement caching for frequently accessed images.
• Clean up temporary files in /tmp to avoid running out of space.

Security

Use IAM policies to ensure only necessary access is granted to S3:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::my-image-processor-bucket/*"
        }
    ]
}

6. Deployment

Finally, let’s deploy everything using AWS SAM, which simplifies the deployment process.

Project Structure

image-processor/
├── template.yaml
├── functions/
│   ├── validate/
│   │   └── app.py
│   ├── resize/
│   │   └── app.py
└── statemachine/
    └── definition.asl.json

SAM Template

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
  ImageProcessorStateMachine:
    Type: AWS::Serverless::StateMachine
    Properties:
      DefinitionUri: statemachine/definition.asl.json
      Policies:
        - LambdaInvokePolicy:
            FunctionName: !Ref ValidateFunction
        - LambdaInvokePolicy:
            FunctionName: !Ref ResizeFunction

  ValidateFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: functions/validate/
      Handler: app.lambda_handler
      Runtime: python3.9
      MemorySize: 1024
      Timeout: 30

  ResizeFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: functions/resize/
      Handler: app.lambda_handler
      Runtime: python3.9
      MemorySize: 1024
      Timeout: 30

Deployment Commands

# Build the application
sam build

# Deploy (first time)
sam deploy --guided

# Subsequent deployments
sam deploy

After deployment, test your application by uploading an image to your S3 bucket:

aws s3 cp test-image.jpg s3://my-image-processor-bucket/raw/

Yeah, You have built a robust, serverless image-processing application. The beauty of this setup is its scalability, from a handful of images to thousands, it can handle them all seamlessly.

And like any good recipe, feel free to tweak the process to fit your needs. Maybe you want to add extra processing steps or fine-tune the Lambda configurations, there’s always room for experimentation.

Tags: AutomationAWSCloudArchitectureCloudComputingDevOpsImageProcessingLambdaServerlessStepFunctions