SRE stuff

Random comments from a SRE

Secure and simplify EC2 access with AWS Session Manager

Accessing EC2 instances used to be a hassle. Bastion hosts, SSH keys, firewall rules, each piece added another layer of complexity and potential security risks. You had to open ports, distribute keys, and constantly manage access. It felt like setting up an intricate vault just to perform simple administrative tasks.

AWS Session Manager changes the game entirely. No exposed ports, no key distribution nightmares, and a complete audit trail of every session. Think of it as replacing traditional keys and doors with a secure, on-demand teleportation system, one that logs everything.

How AWS Session Manager works

Session Manager is part of AWS Systems Manager, a fully managed service that provides secure, browser-based, and CLI-based access to EC2 instances without needing SSH or RDP. Here’s how it works:

An SSM Agent runs on the instance and communicates outbound to AWS Systems Manager.
When you start a session, AWS verifies your identity and permissions using IAM.
Once authorized, a secure channel is created between your local machine and the instance, without opening any inbound ports.

This approach significantly reduces the attack surface. There is no need to open port 22 (SSH) or 3389 (RDP) for bastion hosts. Moreover, since authentication and authorization are managed by IAM policies, you no longer have to distribute or rotate SSH keys.

Setting up AWS Session Manager

Getting started with Session Manager is straightforward. Here’s a step-by-step guide:

1. Ensure the SSM agent is installed

Most modern Amazon Machine Images (AMIs) come with the SSM Agent pre-installed. If yours doesn’t, install it manually using the following command (for Amazon Linux, Ubuntu, or RHEL):

sudo yum install -y amazon-ssm-agent
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent

2. Create an IAM Role for EC2

Your EC2 instance needs an IAM role to communicate with AWS Systems Manager. Attach a policy that grants at least the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:StartSession"
      ],
      "Resource": [
        "arn:aws:ec2:REGION:ACCOUNT_ID:instance/INSTANCE_ID"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:TerminateSession",
        "ssm:ResumeSession"
      ],
      "Resource": [
        "arn:aws:ssm:REGION:ACCOUNT_ID:session/${aws:username}-*"
      ]
    }
  ]
}

Replace REGION, ACCOUNT_ID, and INSTANCE_ID with your actual values. For best security practices, apply the principle of least privilege by restricting access to specific instances or tags.

3. Connect to your instance

Once the IAM role is attached, you’re ready to connect.

From the AWS Console: Navigate to EC2 > Instances, select your instance, click Connect, and choose Session Manager.

From the AWS CLI: Run:

aws ssm start-session --target i-xxxxxxxxxxxxxxxxx

That’s it, no SSH keys, no VPNs, no open ports.

Built-in security and auditing

Session Manager doesn’t just improve security, it also enhances compliance and auditing. Every session can be logged to Amazon S3 or CloudWatch Logs, capturing a full record of all executed commands. This ensures complete visibility into who accessed which instance and what actions were taken.

To enable logging, navigate to AWS Systems Manager > Session Manager, configure Session Preferences, and enable logging to an S3 bucket or CloudWatch Log Group.

Why Session Manager is better than traditional methods

Let’s compare Session Manager with traditional access methods:

Feature	Bastion Host & SSH	AWS Session Manager
Open inbound ports	Yes (22, 3389)	No
Requires SSH keys	Yes	No
Key rotation required	Yes	No
Logs session activity	Manual setup	Built-in
Works for on-premises	No	Yes

Session Manager removes unnecessary complexity. No more juggling bastion hosts, no more worrying about expired SSH keys, and no more open ports that expose your infrastructure to unnecessary risks.

Real-World applications and operational Benefits

Session Manager is not just a theoretical improvement, it delivers real-world value in multiple scenarios:

Developers can quickly access production or staging instances without security concerns.
System administrators can perform routine maintenance without managing SSH key distribution.
Security teams gain complete visibility into instance access and command history.
Hybrid cloud environments benefit from unified access across AWS and on-premises infrastructure.

With these advantages, Session Manager aligns perfectly with modern cloud-native security principles, helping teams focus on operations rather than infrastructure headaches.

In summary

AWS Session Manager isn’t just another tool, it’s a fundamental shift in how we access EC2 instances securely. If you’re still relying on bastion hosts and SSH keys, it’s time to rethink your approach.Try it out, configure logging, and experience a simpler, more secure way to manage your instances. You might never go back to the old ways.

Boost Performance and Resilience with AWS EC2 Placement Groups

There’s a hidden art to placing your EC2 instances in AWS. It’s not just about spinning up machines and hoping for the best, where they land in AWS’s vast infrastructure can make all the difference in performance, resilience, and cost. This is where Placement Groups come in.

You might have deployed instances before without worrying about placement, and for many workloads, that’s perfectly fine. But when your application needs lightning-fast communication, fault tolerance, or optimized performance, Placement Groups become a critical tool in your AWS arsenal.

Let’s break it down.

What are Placement Groups?

AWS Placement Groups give you control over how your EC2 instances are positioned within AWS’s data centers. Instead of leaving it to chance, you can specify how close, or how far apart, your instances should be placed. This helps optimize either latency, fault tolerance, or a balance of both.

There are three types of Placement Groups: Cluster, Spread, and Partition. Each serves a different purpose, and choosing the right one depends on your application’s needs.

Types of Placement Groups and when to use them

Cluster Placement Groups for speed over everything

Think of Cluster Placement Groups like a Formula 1 pit crew. Every millisecond counts, and your instances need to communicate at breakneck speeds. AWS achieves this by placing them on the same physical hardware, minimizing latency, and maximizing network throughput.

This is perfect for:
✅ High-performance computing (HPC) clusters
✅ Real-time financial trading systems
✅ Large-scale data processing (big data, AI, and ML workloads)

⚠️ The Trade-off: While these instances talk to each other at lightning speed, they’re all packed together on the same hardware. If that hardware fails, everything inside the Cluster Placement Group goes down with it.

Spread Placement Groups for maximum resilience

Now, imagine you’re managing a set of VIP guests at a high-profile event. Instead of seating them all at the same table (risking one bad spill ruining their night), you spread them out across different areas. That’s what Spread Placement Groups do, they distribute instances across separate physical machines to reduce the impact of hardware failure.

Best suited for:
✅ Mission-critical applications that need high availability
✅ Databases requiring redundancy across multiple nodes
✅ Low-latency, fault-tolerant applications

⚠️ The Limitation: AWS allows only seven instances per Availability Zone in a Spread Placement Group. If your application needs more, you may need to rethink your architecture.

Partition Placement Groups, the best of both worlds approach

Partition Placement Groups work like a warehouse with multiple sections, each with its power supply. If one section loses power, the others keep running. AWS follows the same principle, grouping instances into multiple partitions spread across different racks of hardware. This provides both high performance and resilience, a sweet spot between Cluster and Spread Placement Groups.

Best for:
✅ Distributed databases like Cassandra, HDFS, or Hadoop
✅ Large-scale analytics workloads
✅ Applications needing both performance and fault tolerance

⚠️ AWS’s Partitioning Rule: The number of partitions you can use depends on the AWS Region, and you must carefully plan how instances are distributed.

How to Configure Placement Groups

Setting up a Placement Group is straightforward, and you can do it using the AWS Management Console, AWS CLI, or an SDK.

Example using AWS CLI

Let’s create a Cluster Placement Group:

aws ec2 create-placement-group --group-name my-cluster-group --strategy cluster

Now, launch an instance into the group:

aws ec2 run-instances --image-id ami-12345678 --count 1 --instance-type c5.large --placement GroupName=my-cluster-group

For Spread and Partition Placement Groups, simply change the strategy:

aws ec2 create-placement-group --group-name my-spread-group --strategy spread
aws ec2 create-placement-group --group-name my-partition-group --strategy partition

Best practices for using Placement Groups

🚀 Combine with Multi-AZ Deployments: Placement Groups work within a single Availability Zone, so consider spanning multiple AZs for maximum resilience.

📊 Monitor Network Performance: AWS doesn’t guarantee placement if your instance type isn’t supported or there’s insufficient capacity. Always benchmark your performance after deployment.

💰 Balance Cost and Performance: Cluster Placement Groups give the fastest network speeds, but they also increase failure risk. If high availability is critical, Spread or Partition Groups might be a better fit.

Final thoughts

AWS Placement Groups are a powerful but often overlooked feature. They allow you to maximize performance, minimize downtime, and optimize costs, but only if you choose the right type.

The next time you deploy EC2 instances, don’t just launch them randomly, placement matters. Choose wisely, and your infrastructure will thank you for it.

Avoiding security gaps by limiting IAM Role permissions

Think about how often we take security for granted. You move into a new apartment and forget to lock the door because nothing bad has ever happened. Then, one day, someone strolls in, helps themselves to your fridge, sits on your couch, and even uses your WiFi. Feels unsettling, right? That’s exactly what happens in AWS when an IAM role is granted far more permissions than it needs, leaving the door wide open for potential security risks.

This is where the principle of least privilege comes in. It’s a fancy way of saying: “Give just enough permissions for the job to get done, and nothing more.” But how do we figure out exactly what permissions an application needs? Enter AWS CloudTrail and Access Analyzer, two incredibly useful tools that help us tighten security without breaking functionality.

The problem of overly generous permissions

Let’s say you have an application running in AWS, and you assign it a role with AdministratorAccess. It can now do anything in your AWS account, from spinning up EC2 instances to deleting databases. Most of the time, it doesn’t even need 90% of these permissions. But if an attacker gets access to that role, you’re in serious trouble.

What we need is a way to see what permissions the application is actually using and then build a custom policy that includes only those permissions. That’s where CloudTrail and Access Analyzer come to the rescue.

Watching everything with CloudTrail

AWS CloudTrail is like a security camera that records every API call made in your AWS environment. It logs who did what, which service they accessed, and when they did it. If you enable CloudTrail for your AWS account, it will capture all activity, giving you a clear picture of which permissions your application uses.

So, the first step is simple: Turn on CloudTrail and let it run for a while. This will collect valuable data on what the application is doing.

Generating a Custom Policy with Access Analyzer

Now that we have a log of the application’s activity, we can use AWS IAM Access Analyzer to create a tailor-made policy instead of guessing. Access Analyzer looks at the CloudTrail logs and automatically generates a policy containing only the permissions that were used.

It’s like watching a security camera playback of who entered your house and then giving house keys only to the people who actually needed access.

Why this works so well

This approach solves multiple problems at once:

Precise permissions: You stop giving unnecessary access because now you know exactly what is needed.
Automated policy generation: Instead of manually writing a policy full of guesswork, Access Analyzer does the heavy lifting.
Better security: If an attacker compromises the role, they get access only to a limited set of actions, reducing damage.
Following best practices: Least privilege is a fundamental rule in cloud security, and this method makes it easy to follow.

Recap

Instead of blindly granting permissions and hoping for the best, enable CloudTrail, track what your application is doing, and let Access Analyzer craft a custom policy. This way, you ensure that your IAM roles only have the permissions they need, keeping your AWS environment secure without unnecessary exposure.

Security isn’t about making things difficult. It’s about making sure that only the right people, and applications, have access to the right things. Just like locking your door at night.

January 31, 2025 by Fernando SRE Cloud stuff SRE stuff 0

AWS Identity Management – Choosing the right Policy or Role

Let’s be honest, AWS Identity and Access Management (IAM) can feel like a jungle. You’ve got your policies, your roles, your managed this, and your inline that. It’s easy to get lost, and a wrong turn can lead to a security vulnerability or a frustrating roadblock. But fear not! Just like a curious explorer, we’re going to cut through the thicket and understand this thing. Why? Mastering IAM is crucial to keeping your AWS environment secure and efficient. So, which policy type is the right one for the job? Ever scratched your head over when to use a service-linked role? Stick with me, and we’ll figure it out with a healthy dose of curiosity and a dash of common sense.

Understanding Policies and Roles

First things first. Let’s get our definitions straight. Think of policies as rulebooks. They are written in a language called JSON, and they define what actions are allowed or denied on which AWS resources. Simple enough, right?

Now, roles are a bit different. They’re like temporary access badges. An entity, be it a user, an application, or even an AWS service itself, can “wear” a role to gain specific permissions for a limited time. A user or a service is not granted permissions directly, it’s the role that has the permissions.

AWS Policy types

Now, let’s explore the different flavors of policies.

AWS Managed Policies

These are like the standard-issue rulebooks created and maintained by AWS itself. You can’t change them, just like you can’t rewrite the rules of physics! But AWS keeps them updated, which is quite handy.

Use Cases: Perfect for common scenarios. Need to give someone basic access to S3? There’s probably an AWS-managed policy for that.
Pros: Easy to use, always up-to-date, less work for you.
Cons: Inflexible, you’re stuck with what AWS provides.

Customer Managed Policies

These are your rulebooks. You write them, you modify them, you control them.

Use Cases: When you need fine-grained control, like granting access to a very specific resource or creating custom permissions for your application, this is your go-to choice.
Pros: Total control, flexible, adaptable to your unique needs.
Cons: More responsibility, you need to know what you’re doing. You’ll be in charge of updating and maintaining them.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::my-specific-bucket/*"
        }
    ]
}

This simple policy allows getting objects only from my-specific-bucket. You have to adapt it to your necessities.

Inline Policies

These are like sticky notes attached directly to a user, group, or role. They’re tightly bound and can’t be reused.

Use Cases: For precise, one-time permissions. Imagine a developer who needs temporary access to a particular resource for a single task.
Pros: Highly specific, good for exceptions.
Cons: A nightmare to manage at scale, not reusable.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "dynamodb:DeleteItem",
            "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/MyTable"
        }
    ]
}

This policy is directly embedded within users and permits them to delete items from the MyTable DynamoDB table. It does not apply to other users or resources.

Service-Linked Roles. The smooth operators

These are special roles pre-configured by AWS services to interact with other AWS services securely. You don’t create them, the service does.

Use Cases: Think of Auto Scaling needing to launch EC2 instances or Elastic Load Balancing managing resources on your behalf. It’s like giving your trusted assistant a special key to access specific rooms in your house.
Pros: Simplifies setup, and ensures security best practices are followed. AWS takes care of these roles behind the scenes, so you don’t need to worry about them.
Cons: You can’t modify them directly. So, it’s essential to understand what they do.

aws autoscaling create-auto-scaling-group \ --auto-scaling-group-name my-asg \ --launch-template "LaunchTemplateId=lt-0123456789abcdef0,Version=1" \ --min-size 1 \ --max-size 3 \ --vpc-zone-identifier "subnet-0123456789abcdef0" \ --service-linked-role-arn arn:aws:iam::123456789012:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling

This code creates an Auto Scaling group, and the service-linked-role-arn parameter specifies the ARN of the service-linked role for Auto Scaling. It’s usually created automatically by the service when needed.

Best practices

Least Privilege: Always, always, always grant only the necessary permissions. It’s like giving out keys only to the rooms people need to access, not the entire house!
Regular Review: Things change. Regularly review your policies and roles to make sure they’re still appropriate.
Use the Right Tools: AWS provides tools like IAM Access Analyzer to help you manage this stuff. Use them!
Document Everything: Keep track of your policies and roles, their purpose, and why they were created. It will save you headaches later.

In sum

The right policy or role depends on the specific situation. Choose wisely, keep things tidy, and you will have a secure and well-organized AWS environment.

January 29, 2025 by Fernando SRE Cloud stuff SRE stuff 0

Deciphering AWS Network Mysteries with Reachability Analyzer

Let’s talk about the cloud, specifically, the tangled web of networks we build inside AWS. You spin up your Virtual Private Clouds (VPCs), toss in some subnets, sprinkle in a few security groups, configure those route tables, and before you know it, you’ve got a more complex network than a Rube Goldberg machine. Everything works great… until it doesn’t. A connection fails, an application times out, and you’re left scratching your head. Where do you even begin to troubleshoot?

This is the exact headache that AWS Reachability Analyzer is designed to cure. It is not the most known tool in the AWS toolbox, but believe me, it’s a lifesaver when diagnosing network connectivity issues. This article will explore what Reachability Analyzer is, how this handy tool works its magic, and why you should use it to keep your AWS network humming along smoothly.

What exactly is AWS Reachability Analyzer?

So, what’s the deal with Reachability Analyzer? Think of it as your network detective. It’s a configuration analysis tool that lets you test the connectivity between a source and a destination within your AWS environment. The beauty of it is that it doesn’t send any live traffic. Instead, it does something much smarter.

This nifty tool analyzes your network configuration, your security groups, Network Access Control Lists (NACLs), route tables, and all that jazz. It then builds a virtual model of your network and simulates the path that traffic would take. This way it determines whether packets starting their journey at the source could reach their intended destination.

Reachability Analyzer is part of the VPC service but tightly integrates with AWS Network Manager. If you’re dealing with a global network spanning multiple regions, Network Manager lets you run these reachability analyses centrally, giving you a bird’s-eye view of connectivity across your entire infrastructure.

It’s essential to understand what Reachability Analyzer doesn’t do. It won’t test your application-level connectivity or tell you anything about latency. It strictly focuses on the network layer, making sure the path is clear, based on your setup. It also does not take into account firewall rules of the OS, or the capacity of the resources to handle the traffic.

The perks of using Reachability Analyzer

Why bother with Reachability Analyzer? Let me break down the key benefits:

Pinpoint Connectivity Problems Fast: No more endless digging through logs or running manual traceroutes. Reachability Analyzer quickly identifies the root cause of connectivity issues, saving you precious time and frustration.
Validate Your Network Setup: It helps ensure your network is configured exactly as you intended and that your security policies are correctly enforced.
Plan Network Changes with Confidence: Before making any changes to your network, you can use Reachability Analyzer to simulate the impact and avoid accidental outages.
Boost Your Security Posture: By uncovering potential configuration flaws, it helps you strengthen your network’s defenses.
Easy Peasy to Use: The interface is intuitive. You don’t need to be a networking guru to use it effectively.
Identify Components Involved: It shows you hop-by-hop the details of the virtual path between the origin and the destination, giving you visibility of the resources involved in the connection.

Reachability Analyzer in Action

Let’s get our hands dirty with some practical examples to see how Reachability Analyzer shines in real-world scenarios:

Scenario 1 – EC2 Instance Can’t Talk to RDS Database

Your application running on an EC2 instance is throwing a tantrum and can’t connect to your RDS database, even though they’re in the same VPC. Reachability Analyzer to the rescue! You set up an analysis between the EC2 instance’s Elastic Network Interface (ENI) and the RDS instance’s ENI.

Bam! Reachability Analyzer might reveal that the RDS security group is the culprit. It’s not allowing inbound traffic from the EC2 instance’s security group on the database port. The problem is identified, and you can fix the security group rule with surgical precision.
Scenario 2 – Testing Connectivity After Route Table Tweaks

You’ve just modified a route table to direct traffic between two subnets through a firewall. Now you need to be sure that connectivity is still working as expected.

Simply create an analysis between an instance in the source subnet and one in the destination subnet. Reachability Analyzer will show you the complete path, including the hop through the firewall. If there’s a hiccup in the route table or the firewall configuration, you’ll spot it immediately.
Scenario 3 – VPN Connectivity Woes

You’ve set up a VPN connection between your VPC and your on-premise network, but your users are complaining that they can’t access resources on-premise. Time to bring in Reachability Analyzer.

Run an analysis from an instance in your VPC to an IP address of a server in your on-premise network. Reachability Analyzer might show you that your subnet’s route table is missing a route to the on-premise network via the Virtual Private Gateway (VGW). Or maybe there is a problem with the configuration of your VPN tunnel. The results will give you the clues you need to troubleshoot the VPN setup.
Scenario 4 – Transit Gateway Validation

You are using a Transit Gateway to connect multiple VPCs, and you need to verify connectivity between them.

Configure tests between instances in different VPCs attached to the Transit Gateway. Reachability Analyzer will show you if the Transit Gateway route tables are correctly configured and if the VPCs can communicate through the resource. It can also help determine if there are asymmetric routing issues, where traffic flows in one direction but not the other.

How to use Reachability Analyzer

Ready to give it a spin? Here’s a simple step-by-step guide:

Access the Tool: Head over to the AWS Management Console, navigate to the VPC section, and you’ll find Reachability Analyzer there. Or, if you are using Network Manager, you can find it in that section.
Create an Analysis:

.- Select your source and destination. This could be an EC2 instance, an ENI, an Internet Gateway, a VPN Gateway, and more.

.- Specify the protocol (TCP or UDP) and optionally, the destination port.

.- If needed and applicable, enter the source IP address or port.

Run the Analysis: Hit the “Create and run analysis path” button and let Reachability Analyzer do its thing.
Interpret the Results:

.- The tool will tell you if the destination is “Reachable” or “Not reachable.”

.- If there’s a problem, it will provide a detailed breakdown of the path, showing you exactly which component is blocking the connection and an explanation of why.

Run the Analysis from Network Manager: If you have a global network, run the reachability analysis from Network Manager for a broader view.

Wrapping Up

AWS Reachability Analyzer is a powerful tool that simplifies network troubleshooting and gives you greater control over your AWS environment. It’s like having X-ray vision for your network. So, next time you encounter a connectivity mystery in your AWS setup, don’t panic. Fire up Reachability Analyzer, and you will have answers in minutes. Try it out, experiment, and unlock the secrets of your network.

January 23, 2025 by Fernando SRE Cloud stuff DevOps stuff SRE stuff 0

AWS Fault Injection service, the unknown service

Let’s discuss something near and dear to every AWS Architect and DevOps Engineer’s heart: resilience. Or, as I like to call it, “making sure your digital baby doesn’t throw a tantrum when things go sideways.”

We’ve all been there. Like a magnificent sandcastle, you build this beautiful, intricate system in the cloud. It’s got auto-scaling, high availability, and the works. You’re feeling pretty proud of yourself. Then, BAM! Some unforeseen event, a tiny ripple in the force of the internet, and your sandcastle starts to crumble. Panic ensues.

But what if, instead of waiting for disaster to strike, you could be a bit… mischievous? What if you could poke and prod your system before it has a meltdown in front of your users? Enter AWS Fault Injection Simulator (FIS), a service that’s about as well-known as a quiet librarian at a rock concert, but far more useful.

What’s this FIS thing, anyway?

Think of FIS as your friendly neighborhood chaos monkey but with a PhD in engineering and a strict code of conduct. It’s a fully managed service that lets you run controlled chaos experiments on your AWS workloads. Yes, you read that right. You can intentionally break things but in a safe and measured way. It is like playing Jenga but only for advanced players.

Why would you do that, you ask? Well, my friends, it’s all about finding those hidden weaknesses before they become major headaches. It’s like giving your application a stress test, similar to how doctors check your heart’s health. You want to see how it handles the pressure before it’s out there running a marathon in the real world. The idea is simple: you don’t know how strong the dam will be until you put the river on it.

Why is this CHAOS stuff so important?

In the old days (you know, like five years ago), we tested for predictable failures. Server goes down? No problem, we have a backup! But the cloud is a complex beast, and failures can be, well, weird. Latency spikes, partial network outages, API throttling… it’s a jungle out there.

FIS helps you simulate these real-world, often unpredictable scenarios. By deliberately injecting faults, you expose how your system behaves under stress. This way you will discover if your great ideas in whiteboards are translated into a great and resilient system in the cloud.

This isn’t just about avoiding downtime, though that’s a big plus. It’s about:

Improving Reliability: Find and fix weak points, leading to a more robust and dependable system.
Boosting Performance: Identify bottlenecks and optimize your application’s response under duress.
Validating Your Assumptions: Does your fancy auto-scaling work as intended? FIS will tell you.
Building Confidence: Knowing your system can handle the unexpected gives you peace of mind. And maybe, just maybe, you can sleep through the night without getting paged. A DevOps Engineer can dream, right?

Let’s get our hands dirty (Virtually, of course)

So, how does this magical chaos tool work? FIS operates through experiment templates. These are like recipes for disaster (the good kind, of course). In these templates, you define:

Actions: What kind of mischief do you want to unleash? FIS offers a menu of pre-built actions, like:
- aws:ec2:stop-instances: Stop EC2 instances. You pick which ones.
- aws:ec2:terminate-instances: Terminate EC2 instances. Poof, they are gone.
- aws:ssm:send-command: Run a script on an instance that causes, for example, CPU stress, or memory stress.
- aws:fis:inject-api-latency: Add latency to internal or external APIs.
Targets: Where do you want to inject these faults? You can target specific EC2 instances, ECS clusters, EKS clusters, RDS databases… You get the idea. You can select the resources by tags, by name, by percentage… You have plenty of options here.
Stop Conditions: This is your “emergency brake.” You define CloudWatch alarms that, if triggered, will automatically halt the experiment. Safety first, people! Imagine that the experiment is affecting more components than expected, the stop condition will be your friend here.
IAM Role: This role is very important. It will give the FIS service permission to inject the fault into your resources. Remember to assign only the necessary permissions, nothing more.

Once you’ve crafted your experiment template, you can run it and watch the magic (or mayhem) unfold. FIS provides detailed logs and integrates with CloudWatch, so you can monitor the impact in real time.

FIS in the Wild

Let’s say you have a microservices architecture running on ECS. You want to test how your system handles the failure of a critical service. With FIS, you could create an experiment that:

Action: Terminates a percentage of the tasks in your critical service.
Target: Your ECS service, specifically the tasks tagged as “critical-service.”
Stop Condition: A CloudWatch alarm that triggers if your application’s latency exceeds a certain threshold or the error rate increases.

By running this experiment, you can observe how your other services react, whether your load balancing works as expected, and if your system can gracefully recover.

Or, imagine you want to test the resilience of your RDS database. You could simulate a failover by:

Action: aws:rds:reboot-db-instance with the failover option set to true.
Target: Your primary RDS instance.
Stop Condition: A CloudWatch alarm that monitors the database’s availability.

This allows you to validate your read replica setup and ensure a smooth transition in case of a real-world primary instance failure.

I remember one time I was helping a startup that had a critical application running on EC2. They were convinced their auto-scaling was flawless. We used FIS to simulate a sudden surge in traffic by terminating a bunch of instances. Guess what? Their auto-scaling took longer to kick in than they expected, leading to a brief period of performance degradation. Thanks to the experiment, they were able to fix the issue, avoiding real user impact in the future.

My Two Cents (and Maybe a Few More)

I’ve been around the AWS block a few times, and I can tell you that FIS is a game-changer. It’s not just about breaking things; it’s about understanding things. It’s about building systems that are not just robust on paper but resilient in the face of the unpredictable chaos of the real world.

January 18, 2025 by Fernando SRE Cloud stuff DevOps stuff SRE stuff 0

Git branching strategies Merge or Rebase

Picture this: You’re building a magnificent LEGO castle, not alone but with a team. Each of you is crafting different sections, a tower, a wall, maybe a dungeon for the mischievous minifigures. The grand question arises: How do you unite these masterpieces into one glorious fortress?

This is where Git, our trusty version control system, steps in, offering two distinct approaches: Merge and Rebase. Both achieve the same goal, bringing your team’s work together, but they do so with different philosophies and, consequently, different outcomes in your project’s history. So, which path should you choose? Let’s unravel this mystery together!

Merging: The Storyteller

Imagine git merge as a meticulous historian, carefully documenting every step of your castle-building journey. When you merge two branches, Git creates a special “merge commit,” a snapshot that says, “Here’s where we brought these two storylines together.” It’s like adding a chapter to a book that acknowledges the contributions of multiple authors.

# You are on the 'feature' branch 
git checkout main 
git merge feature 

# Result: A new merge commit is created on 'main'

What’s the beauty of this approach?

Preserves History: You get a complete, chronological record of every commit, every twist and turn in your development process. It’s like having a detailed blueprint of how your LEGO castle was built, brick by brick.
Transparency: Everyone on the team can easily see how the project evolved, who made what changes, and when. This is crucial for collaboration and debugging.
Safety Net: If something goes wrong, you can easily trace back the changes and revert to an earlier state. It’s like having a time machine to undo any construction mishaps.

But, there’s a catch (isn’t there always?):

Messy History: Your project’s history can become quite complex, especially with frequent merges. Imagine a book with too many footnotes, it can be a bit overwhelming to follow.

Rebasing: The Time Traveler

Now, git rebase takes a different approach. Think of it as a time traveler who neatly rewrites history. Instead of creating a merge commit, rebase takes your branch’s commits and replants them on top of the target branch, making it appear as if you’d been working directly on that branch all along.

# You are on the 'feature' branch 
git checkout feature 
git rebase main 

# Result: The 'feature' branch's commits are now on top of 'main'

Why would you want to rewrite history?

Clean History: You end up with a linear, streamlined project history, like a well-organized story with a clear narrative flow. It’s easier to read and understand the overall progression of the project.
Simplified View: It can be easier to visualize the project’s development as a single, continuous line, especially in projects with many contributors.

However, there’s a word of caution:

History Alteration: Rebasing rewrites the commit history. This can be problematic if you’re working on a shared branch, as it can lead to confusion and conflicts for other team members. Imagine someone changing the blueprints while you’re still building… chaos.
Potential for Errors: If not done carefully, rebasing can introduce subtle bugs that are hard to track down.

So, Merge or Rebase? The Golden Rule

Here’s the gist, the key takeaway, the rule of thumb you should tattoo on your programmer’s brain (metaphorically, of course):

Use merge for shared or public branches (like main or master). It preserves the true history and keeps everyone on the same page.
Use rebase for your local feature branches before merging them into a shared branch. This keeps your feature branch’s history clean and easy to understand, making the final merge smoother.

Think of it this way: you do your messy experiments and drafts in your private notebook (local branch with rebase), and then you neatly transcribe your final work into the official logbook (shared branch with merge).

Analogy Time!

Let’s say you and your friend are writing a song.

Merge: You each write verses separately. Then, you combine them, creating a new verse that says, “Here’s where Verse 1 and Verse 2 meet.” It’s clear that it was a collaborative effort, and you can still see the individual verses.
Rebase: You write your verse. Then, you take your friend’s verse and rewrite yours as if you had written it after theirs. The song flows seamlessly, but it’s not immediately obvious that two people wrote it.

The Bottom Line

Both merge and rebase are powerful tools. The best choice depends on your specific workflow and your team’s preferences. The most important thing is to understand how each method works and to use them consistently. But always remember the golden rule: merge for shared, rebase for local.

January 8, 2025 by Fernando SRE DevOps stuff Git SRE stuff 0

S3 Access Points explained

Don’t you feel like your data in the cloud is a bit too… exposed? Like you’ve got a treasure chest full of valuable information (your S3 bucket), but it’s just sitting there, practically begging for unwanted attention? You wouldn’t leave your valuables out in the open in the real world, would you? Well, the same logic applies to your data in the cloud.

This is where AWS S3 Access Points come in. They act like bouncers for your data, ensuring only the right people get in. And for those of you with data scattered across the globe, we’ve got something even fancier: Multi-Region Access Points (MRAPs). They’re like the global positioning system for your data, ensuring fast access no matter where you are.

So buckle up, and let’s explore the fascinating world of S3 Access Points and MRAPs. Let’s try to make it fun.

The problem is that your S3 Bucket is wide open (By Default)

Think of an S3 bucket as a giant storage locker in the cloud. When you first create one, it’s like leaving the locker door wide open. Anyone who knows the lockers there can just waltz in and take a peek, or worse, start messing with your stuff.

This might be fine if you’re just storing cat memes, but what if you have sensitive customer data, financial records, or top-secret project files? You need a way to control who gets in and what they can do.

The solution is the Access Points, your data’s bouncers

Imagine Access Points as the bouncers standing guard at the entrance of your storage locker. They check IDs, make sure everyone’s on the guest list, and only let in the people you’ve authorized.

In more technical terms, an Access Point is a unique hostname that you create to enforce distinct permissions and network controls for any request made through it. You can configure each Access Point with its own IAM policy, tailored to specific use cases.

Why you need Access Points. It’s all about control

Here’s the deal:

Granular Access Control: You can create different Access Points for different applications or teams, each with its own set of permissions. Maybe your marketing team only needs read access to product images, while your developers need full read and write access to application logs. Access Points make this a breeze.
Simplified Policy Management: Instead of one giant, complicated bucket policy, you can have smaller, more manageable policies for each Access Point. It’s like having a separate rule book for each group that needs access.
Enhanced Security: By restricting access through specific Access Points, you reduce the risk of accidental data exposure or unauthorized modification. It’s like having multiple layers of security for your precious data.
Compliance Made Easier: Many industries have strict regulations about data access and security (think GDPR, HIPAA). Access Points help you meet these requirements by providing a clear and auditable way to control who can access what.

Let’s get practical with an Access Point policy example

Okay, let’s see how this works in practice. Here’s an example of an Access Point policy that only allows access to a bucket named “pending-documentation” and only permits read and write actions (no deleting!):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/Alice"
      },
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:us-west-2:123456789012:accesspoint/my-access-point/object/pending-documentation/*"
    }
  ]
}

Explanation:

Version: Specifies the policy language version.
Statement: An array of permission statements.
Effect: “Allow” means this statement grants permission.
Principal: This specifies who is granted access. In this case, it’s the IAM user “Alice” (you’d replace this with the actual ARN of your user or role).
Action: The S3 actions allowed. Here, it’s s3:GetObject (read) and s3:PutObject (write).
Resource: This is the crucial part. It specifies the resource the policy applies to. Here, it’s the “pending-documentation” bucket accessed through the “my-access-point” Access Point. The /* at the end means all objects within that bucket path.

Delegating access control to the Access Point (Bucket Policy)

You also need to configure your S3 bucket policy to delegate access control to the Access Point. Here’s an example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*",
      "Condition": {
        "StringEquals": {
          "s3:DataAccessPointArn": "arn:aws:s3:us-west-2:123456789012:accesspoint/my-access-point"
        }
      }
    }
  ]
}

This policy allows any principal (“AWS”: “*”) to perform any S3 action (“s3:*”), but only if the request goes through the specified Access Point ARN.

Taking it global, Multi-Region Access Points (MRAPs)

Now, let’s say your data is spread across multiple AWS regions. Maybe you have users all over the world, and you want them to have fast access to your data, no matter where they are. This is where Multi-Region Access Points (MRAPs) come to the rescue!

Think of an MRAP as a smart global router for your data. It’s a single endpoint that automatically routes requests to the closest copy of your data in one of your S3 buckets across multiple regions.

Why Use MRAPs? Think speed and resilience

Reduced Latency: MRAPs ensure that users are always accessing the data from the nearest region, minimizing latency and improving application performance. It is like having a fast-food in each country, so clients can have their orders faster.
High Availability: If one region becomes unavailable, MRAPs automatically route traffic to another region, ensuring your application stays up and running. It’s like having a backup generator for your data.
Simplified Management: Instead of managing multiple endpoints for different regions, you have one MRAP to rule them all.

MRAPs vs. Regular Access Points, what’s the difference?

While both are about controlling access, MRAPs take it to the next level:

Scope: Regular Access Points are regional; MRAPs are multi-regional.
Focus: Regular Access Points primarily focus on security and access control; MRAPs add performance and availability to the mix.
Complexity: MRAPs are a bit more complex to set up because you’re dealing with multiple regions.

When to unleash the power of Access Points and MRAPs

Data Lakes: Use Access Points to create secure “zones” within your data lake, granting different teams access to only the data they need.
Content Delivery: MRAPs can accelerate content delivery to users around the world by serving data from the nearest region.
Hybrid Cloud: Access Points can help integrate your on-premises applications with your S3 data in a secure and controlled manner.
Compliance: Meeting regulations like GDPR or HIPAA becomes easier with the fine-grained access control provided by Access Points.
Global Applications: If you have a globally distributed application, MRAPs are essential for delivering a seamless user experience.

Lock down your data and speed up access

AWS S3 Access Points and Multi-Region Access Points are powerful tools for managing access to your data in the cloud. They provide the security, control, and performance that modern applications demand.

January 7, 2025 by Fernando SRE Cloud stuff SRE stuff 0

Managing SSL certificates with SNI on AWS ALB and NLB

The challenge of hosting multiple SSL-Secured sites

Let’s talk about security on the web. You want your website to be secure. Of course, you do! That’s where HTTPS and those little SSL/TLS certificates come in. They’re like the secret handshakes of the internet, ensuring that the information flowing between your site and visitors is safe from prying eyes. But here’s the thing: back in the day, if you wanted a bunch of websites, each with its secure certificate, you needed a separate IP address. Imagine having to get a new phone number for every person you wanted to call! It was a real headache and cost a pretty penny, too, especially if you were running a whole bunch of websites.

Defining SNI as a modern SSL/TLS extension

Now, what if I told you there was a clever way around this whole IP address mess? That’s where this little gem called Server Name Indication (SNI) comes in. It’s like a smart little addition to the way websites and browsers talk to each other securely. Think of it this way, your server’s IP address is like a big apartment building, and each website is a different apartment. Without SNI, it’s like visitors can only shout the building’s address (the IP address). The doorman (the server) wouldn’t know which apartment to send them to. SNI fixes that. It lets the visitor whisper both the building address and the apartment number (the website’s name) right at the start. Pretty neat.

Understanding the SNI handshake process

So, how does this SNI thing work? Let’s lift the hood and take a peek at the engine, shall we? It all happens during this little dance called the SSL/TLS handshake, the very beginning of a secure connection.

Client Hello: First, the client (like your web browser) says “Hello!” to the server. But now, thanks to SNI, it also whispers the name of the website it wants to talk to. It is like saying “Hey, I want to connect, and by the way, I’m looking for ‘www.example.com‘”.
Server Selection: The server gets this message and, because it’s a smart cookie, it checks the SNI part. It uses that website name to pick out the right secret handshake (the SSL certificate) from its big box of handshakes.
Server Hello: The server then says “Hello!” back, showing off the certificate it picked.
Secure Connection: The client checks if the handshake looks legit, and if it does, boom! You’ve got yourself a secure connection. It’s like a secret club where everyone knows the password, and they’re all speaking in code so no one else can understand.

AWS load balancers and SNI as a perfect match

Now, let’s bring this into the world of Amazon Web Services (AWS). They’ve got these things called load balancers, which are like traffic cops for websites, directing visitors to the right place. The newer ones, Application Load Balancers (ALB) and Network Load Balancers (NLB) are big fans of SNI. It means you can have a whole bunch of websites, each with its certificate, all hiding behind one of these load balancers. Each of those websites could be running on different computers (EC2 instances, as they call them), but the load balancer, thanks to SNI, knows exactly where to send the visitors.

CloudFront’s adoption of SNI for secure content delivery at scale

And it’s not just load balancers, AWS has this other thing called CloudFront, which is like a super-fast delivery service for websites. It makes sure your website loads quickly for people all over the world. And guess what? CloudFront loves SNI, too. It lets you have different secret handshakes (certificates) for different websites, even if they’re all being delivered through the same CloudFront setup. Just remember, the old-timer, Classic Load Balancer (CLB), doesn’t know this SNI trick. It’s a bit behind the times, so keep that in mind.

Cost savings through optimized resource utilization

Why should you care about all this? Well, for starters, it saves you money! Instead of needing a whole bunch of IP addresses (which cost money), you can use just one with SNI. It is like sharing an office space instead of everyone renting their building.

Simplified management by streamlining certificate handling

And it makes your life a whole lot easier, too. Managing those secret handshakes (certificates) can be a real pain. But with SNI, you can manage them all in one place on your load balancer. It is way simpler than running around to a dozen different offices to update everyone’s secret handshake.

Enhanced scalability for efficient infrastructure growth

And if your website gets popular, no problem, SNI lets you add new websites to your load balancer without breaking a sweat. You don’t have to worry about getting new IP addresses every time you want to launch a new site. It’s like adding new apartments to your building without having to change the building’s address.

Client compatibility to ensure broad support

Now, I have to be honest with you. There might be some really, really old web browsers out there that haven’t heard of SNI. But, honestly, they’re becoming rarer than a dodo bird. Most browsers these days are smart enough to handle SNI, so you don’t have to worry about it.

SNI as a cornerstone of modern Web hosting on AWS

So, there you have it. SNI is like a secret weapon for running websites securely and efficiently on AWS. It’s a clever little trick that saves you money, simplifies your life, and lets your website grow without any headaches. It is proof that even small changes to the way things work on the internet can make a huge difference. When you’re building things on AWS, remember SNI. It’s like having a master key that unlocks a whole bunch of possibilities for a secure and scalable future. It’s a neat piece of engineering if you ask me.

January 5, 2025 by Fernando SRE Cloud stuff DevOps stuff SRE stuff 0

Practical guide to DNS Records in AWS Route 53

Your browser instantly connects you to your desired website when you type in its address and hit enter. It’s a seamless experience we often take for granted. But behind this seemingly simple action lies a complex system that makes it all possible: the Domain Name System (DNS). Think of DNS as the internet’s global directory, translating human-readable domain names into the numerical IP addresses that computers use to communicate. And when managing DNS with reliability and scalability, AWS Route 53 takes center stage. Route 53 is Amazon’s highly available and scalable DNS service, designed to route traffic to your application’s resources with remarkable precision and minimal latency. In this guide, we’ll demystify the most common DNS record types and show you how to use them effectively with Route 53, using practical examples.

Let’s jump into DNS records by breaking them down into simple, relatable examples and exploring real-world use cases. We’ll see how they work together, like a well-orchestrated symphony, to make the internet navigable.

The basics of DNS Records

DNS records are like traffic signs for the internet, directing users to the right destinations. But instead of physical signs, they’re digital entries that guide web browsers and other services. Route 53 makes managing these records straightforward. Here are the most common types:

A Record (Address Record)

Think of an A Record as the street address for your website. It maps a domain name (e.g., example.com) to an IPv4 address (e.g., 192.0.2.1). It’s the most basic thing. It just tells the internet where your website lives.

Purpose: Directs traffic to web servers or other IPv4 resources.
Analogy: Imagine telling a friend to visit you at your home address, that’s what an A Record does for websites. It’s like saying, “Hey, if you’re looking for example.com, it’s over at this IP address.”
Use Case: Hosting a website like example.com on an EC2 instance or an on-premises server.

CNAME Record (Canonical Name)

A CNAME Record is like a nickname for your domain. It maps an alias domain name (e.g., www.example.com) to another “canonical” domain name (e.g., example.com).

Purpose: Simplifies management by allowing multiple domains to point to the same resource. It’s like having various roads leading to the same destination.
Analogy: It’s like calling your friend “Bob” instead of “Robert.” Both names point to the same person.
Use case: Scaling applications by mapping api.example.com to an Application Load Balancer’s DNS name, such as app-load-balancer-456.amazonaws.com. You point your CNAME to the load balancer, and the load balancer handles distributing traffic to your servers.

AAAA Record (Quad A Record)

For the modern internet, AAAA Records map domain names to IPv6 addresses (e.g., 2001:db8::1).

Purpose: Ensures compatibility with IPv6 resources, which is becoming increasingly important as the internet grows.
Analogy: Think of this as an upgrade to a new address system for the internet, ready for the future. It’s like moving from a local phone system to a global one.
Use case: Enabling access to your website via IPv6. This ensures your site is reachable by devices using the newer IPv6 standard.

MX Record (Mail Exchange)

MX Records ensure emails sent to your domain arrive at the correct mail server.

Purpose: Routes emails to the appropriate mail server.
Analogy: Like sorting mail at a post office to send it to the right address. Each piece of mail (email) needs to be directed to the correct recipient (mail server).
Use case: Configuring email for domains with Google Workspace or Microsoft 365. This ensures your emails are handled by the right service.

NS Record (Name Server)

NS Records delegate a domain or subdomain to specific name servers.

Purpose: Specifies which servers are authoritative for answering DNS queries for a domain. In other words, they know all the A records, CNAME records, etc., for that domain.
Analogy: It’s like asking a specific guide for directions within a city. That guide knows the specific area inside and out.
Use case: Delegating subdomains like dev.example.com to a different DNS provider, perhaps for testing purposes.

TXT Record (Text Record)

TXT Records store arbitrary text data, often used for domain verification or email security configurations (e.g., SPF, DKIM).

Purpose: Provides information to external systems.
Analogy: Think of it as posting a sign with instructions outside your door. This sign might say, “To verify you own this house, please show this specific code.”
Use case: Adding SPF, DKIM, and DMARC records to prevent email spoofing and improve email deliverability. This helps ensure your emails don’t end up in spam folders.

Alias Record

Exclusive to AWS, Alias Records map domain names to AWS resources like S3 buckets or CloudFront distributions without needing an IP address.

Purpose: Reduces costs and simplifies DNS management, especially within the AWS ecosystem.
Analogy: A direct shortcut to AWS resources without the extra steps. Think of it as a secret tunnel directly to your destination, bypassing traffic.
Use case: Mapping example.com to a CloudFront distribution for CDN integration. This allows for faster content delivery to users around the world. Or, say you have a static website hosted on S3. An Alias record can point your domain directly to the S3 bucket, without needing a separate web server.

Putting it all together

Let’s look at how these records work in harmony to power your website. See? It’s not so complicated when you break it down. Each record has its job, and they all work together like a well-oiled machine.

Hosting a scalable website

Register your domain: Let’s say you register example.com using Route 53.
Create an A Record: You map example.com to an EC2 instance’s IP address where your website is hosted.
Add a CNAME Record: For www.example.com, you create a CNAME pointing to example.com. This way, both addresses lead to your site.
Utilize Alias Records: To speed up content delivery, you create an Alias record connecting example.com to a CloudFront distribution. This caches your website content at edge locations closer to your users. And shall we use another Alias Record to connect static.example.com to an S3 bucket, to serve your images faster? Why not.
Implement TXT Records: You add TXT records for email authentication (SPF, DKIM) to ensure your emails are trusted and delivered reliably.
Enable health checks: Route 53 can automatically monitor the health of your EC2 instances and route traffic away from unhealthy ones, ensuring your site stays up even if a server has issues. Route 53 can even automatically remove unhealthy instances from your DNS records.

This setup ensures high availability, scalability, and secure communication. But what makes Route 53 special? It’s not just about creating these records; it’s about doing it reliably and efficiently. Route 53 is designed for high availability and low latency. It uses a global network of DNS servers to ensure your website is always reachable, even if one server or region has problems. That means faster loading times for your users, no matter where they are.

Closing thoughts

AWS Route 53 isn’t just about creating DNS records, it’s about building robust, scalable, and secure internet infrastructure. It’s about making sure your website is always available to your users, no matter what. It’s like having a team of incredibly efficient digital postal workers who know exactly how to deliver each data packet to its correct destination. And what’s fascinating is that, like a well-designed metro system, Route 53 operates on multiple levels: it can direct traffic based on latency, geolocation, or even the health status of your services. Consider for a moment the massive scale at which services like Netflix or Amazon operate, keeping their platforms running smoothly with millions of simultaneous users. Part of that magic happens thanks to services like Route 53.
The beauty of it all lies in its apparent simplicity for the end user, everything works seamlessly, but behind the scenes, there’s a complex orchestration of systems working in perfect harmony. It’s like a symphony where each DNS record is a different instrument, and Route 53 is the conductor ensuring everything sounds exactly as it should.

January 3, 2025 by Fernando SRE Cloud stuff SRE stuff 0