At 4:12 PM on a Thursday, the deployment pipeline turned a soothing shade of emerald green. Forty-seven integration tests had just sprinted through a DynamoDB suite without a single complaint. Someone took a celebratory sip of lukewarm coffee, merged the pull request, and watched the container roll out to production. Six seconds later, the application tried to write its very first record to DynamoDB and was promptly slapped down with an “AccessDeniedException”.
Engineers love green checkmarks the way hypochondriacs love a clean blood panel; we rarely pause to ask which organs the doctor actually examined. If every database test passed in CI, what, precisely, did that green pipeline prove?
To find out, I ran a small experiment pitting AWS Emulator, AWS’s local, Rust-built API emulator that runs on your laptop without an AWS account, against a real AWS sandbox to see where local testing shines and where cloud reality bites back.
Giving the Emulator and AWS the Same Job
For this test, I used a deliberately modest Python service that writes an order record to a DynamoDB table and immediately reads it back. Nothing exotic, just standard boto3 calls, a partition key, and a handful of attributes.
The beauty of AWS Emulator is that your application code remains blissfully unaware that it is talking to a local process instead of a sprawling data center in Oregon. When the emulator starts up, it generates local credentials and serves standard AWS APIs over HTTPS. Switching between the local emulator and a real AWS sandbox requires zero code edits; you simply swap two environment variables, “AWS_ENDPOINT_URL” and your profile credentials, before running “pytest”.
Here is the core test both environments had to survive:
import boto3
import pytest
from botocore.exceptions import ClientError
def test_write_read_and_duplicate_rejection(dynamodb_table):
item = {"order_id": "ord-90210", "status": "CONFIRMED", "total": 42}
# 1. Write and read back
dynamodb_table.put_item(Item=item)
stored = dynamodb_table.get_item(Key={"order_id": "ord-90210"}).get("Item")
assert stored == item
# 2. Reject duplicate write conditionally
with pytest.raises(ClientError) as exc:
dynamodb_table.put_item(
Item={"order_id": "ord-90210", "status": "OVERWRITTEN", "total": 99},
ConditionExpression="attribute_not_exists(order_id)",
)
assert exc.value.response["Error"]["Code"] == "ConditionalCheckFailedException"
assert dynamodb_table.get_item(Key={"order_id": "ord-90210"})["Item"]["status"] == "CONFIRMED"When I compared the two runs, I split the stopwatch into two phases, environment setup and actual test execution, because lumping them together is how benchmarks lie to people.
- Setup time was a blowout. Starting AWS Emulator and creating the table locally took roughly 0.4 seconds. Provisioning the equivalent table in a real AWS sandbox took just over 7 seconds of polite waiting while cloud control planes shuffled paperwork.
- Test execution time showed a similar gap. Against the emulator, the write and read round-trip finished in 14 milliseconds. Against real AWS, network physics intervened, clocking in at 185 milliseconds.
- Returned values were identical down to the last dictionary key.
Making a Write Fail on Purpose
Happy-path tests are charming, like testing an umbrella indoors. Real confidence comes from watching software fail properly.
In the second half of the snippet above, the application attempts a conditional write using “attribute_not_exists(order_id)” to prevent duplicate orders from clobbering existing ones. Both environments had to reject the duplicate write, return a “ConditionalCheckFailedException”, and leave the original “CONFIRMED” record untouched.
Both agreed completely. Traditional SDK mocks are polite yes-men; they return whatever cheerful hallucination you programmed them to return, which means they happily pass tests for bugs you invented yourself. AWS Emulator, by contrast, is generated from official AWS API models and maintains real resource state across calls. Watching it enforce condition expressions and reject malformed requests with authentic AWS error codes explains why running these tests locally actually means something.
Of course, no emulator is omniscient. AWS publishes its own fidelity tests and release-by-release scorecards in the “aws-emulator-evaluation-suite” repository on GitHub. Before trusting any local emulator with a tricky service feature, say complex DynamoDB transactions or niche stream filters, you can check those public fidelity reports to verify that the exact API behaviors your app relies on have already been tested against live AWS.
Running One Test in the Actual Uniform
So if the emulator faithfully reproduces DynamoDB’s API behavior, why did our hypothetical Thursday deployment catch fire?
Because an emulator checks whether your code knows how to knock on DynamoDB’s door. IAM checks whether the bouncer lets you inside. By design, AWS Emulator does not evaluate IAM policies or resource-based permissions the way live AWS does. Inside the emulator, your code effectively wears an all-access VIP lanyard.
To see this gap in action, I moved into the AWS sandbox and gave the deployed application’s IAM role permission to write to a table named “orders-allowed”, while leaving a second table, orders-restricted, outside that grant. Then I ran a tiny smoke test from inside the deployed workload itself.
The write to “orders-allowed” sailed through. The write to “orders-restricted” hit a brick wall and threw “AccessDeniedException”, the exact failure mode from our Thursday afternoon post-mortem.
Look back at what happened earlier. Our integration suite proved that the application speaks fluent DynamoDB. This single, ten-line smoke test proved something entirely different, that the IAM role attached to the deployed container actually possesses the keys to the rooms it plans to enter.
What Changes in Your Pipeline
You do not have to choose between glacial cloud test suites and blissful local ignorance. You just have to stop asking one tool to answer both questions. Here is a cleaner division of labor for your CI/CD pipeline:
- On every pull request, run your full integration suite, including writes, reads, edge cases, and conditional failures, against AWS Emulator. It runs in milliseconds, requires zero cloud credentials, and catches broken queries before a human ever reviews the diff.
- Before promoting a deployment, run a short smoke test in a real AWS environment using the deployed workload’s actual IAM role and configuration. Do not test every business rule here; just prove the workload can touch the real tables, queues, and buckets it owns.
- When an SDK or service dependency changes, rerun your integration tests against real AWS if your application depends on subtle service behaviors you have not yet verified in the emulator’s fidelity suite.
Local emulators give developers something priceless, speed without the friction of cloud deployments. Just remember that speed and realism are different currencies. The next time your pipeline flashes green and you reach for your coffee mug, pause for two seconds and ask yourself one question.
When CI goes green, which identity actually passed the test?
