HighAvailability

AWS Disaster Recovery simplified for every business

Let’s talk about something really important, even if it’s not always the most glamorous topic: keeping your AWS-based applications running, no matter what. We’re going to explore the world of High Availability (HA) and Disaster Recovery (DR). Think of it as building a castle strong enough to withstand a dragon attack, or, you know, a server outage..

Why all the fuss about Disaster Recovery?

Businesses run on applications. These are the engines that power everything from online shopping to, well, pretty much anything digital. If those engines sputter and die, bad things happen. Money gets lost. Customers get frustrated. Reputations get tarnished. High Availability and Disaster Recovery are all about making sure those engines keep running, even when things go wrong. It’s about resilience.

Before we jump into solutions, we need to understand two key measurements:

Recovery Time Objective (RTO): How long can you afford to be down? Minutes? Hours? Days? This is your RTO.
Recovery Point Objective (RPO): How much data can you afford to lose? The last hour’s worth? The last days? That’s your RPO.

Think of RTO and RPO as your “pain tolerance” levels. A low RTO and RPO mean you need things back up and running fast, with minimal data loss. A higher RTO and RPO mean you can tolerate a bit more downtime and data loss. The correct option will depend on your business needs.

Disaster recovery strategies on AWS, from basic to bulletproof

AWS offers a toolbox of options, from simple backups to fully redundant, multi-region setups. Let’s explore a few common strategies, like choosing the right level of armor for your knight:

Pilot Light: Imagine keeping the pilot light lit on your stove. It’s not doing much, but it’s ready to ignite the main burner at any moment. In AWS terms, this means having the bare minimum running, maybe a database replica syncing data in another region, and your server configurations saved as templates (AMIs). When disaster strikes, you “turn on the gas”, launch those servers, connect them to the database, and you’re back in business.
- Good for: Cost-conscious applications where you can tolerate a few hours of downtime.
- AWS Services: RDS Multi-AZ (for database replication), Amazon S3 cross-region replication, EC2 AMIs.
Warm Standby: This is like having a smaller, backup stove already plugged in and warmed up. It’s not as powerful as your main stove, but it can handle the basic cooking while the main one is being repaired. In AWS, you’d have a scaled-down version of your application running in another region. It’s ready to handle traffic, but you might need to scale it up (add more “burners”) to handle the full load.
- Good for: Applications where you need faster recovery than Pilot Light, but you still want to control costs.
- AWS Services: Auto Scaling (to automatically adjust capacity), Amazon EC2, Amazon RDS.
Active/Active (Multi-Region): This is the “two full kitchens” approach. You have identical setups running in multiple AWS regions simultaneously. If one kitchen goes down, the other one is already cooking, and your customers barely notice a thing. You use AWS Route 53 (think of it as a smart traffic controller) to send users to the closest or healthiest “kitchen.”
- Good for: Mission-critical applications where downtime is simply unacceptable.
- AWS Services: Route 53 (with health checks and failover routing), Amazon EC2, Amazon RDS, DynamoDB global tables.

Picking the right armor, It’s all about trade-offs

There’s no “one-size-fits-all” answer. The best strategy depends on those RTO/RPO targets we talked about, and, of course, your budget.

Here’s a simple way to think about it:

Tight RTO/RPO, Budget No Object? Active/Active is your champion.
Need Fast Recovery, But Watching Costs? Warm Standby is a good compromise.
Can Tolerate Some Downtime, Prioritizing Cost Savings? Pilot Light is your friend.
Minimum RTO/RPO and Minimum Budget? Backups.

The trick is to be honest about your real needs. Don’t build a fortress if a sturdy wall will do.

A quick glimpse at implementation

Let’s say you’re going with the Pilot Light approach. You could:

Set up Amazon S3 Cross-Region Replication to copy your important data to another AWS region.
Create an Amazon Machine Image (AMI) of your application server. This is like a snapshot of your server’s configuration.
Store that AMI in the backup region.

In a disaster scenario, you’d launch EC2 instances from that AMI, connect them to your replicated data, and point your DNS to the new instances.

Tools like AWS Elastic Disaster Recovery (a managed service) or CloudFormation (for infrastructure-as-code) can automate much of this process, making it less of a headache.

Testing, Testing, 1, 2, 3…

You wouldn’t buy a car without a test drive, right? The same goes for disaster recovery. You must test your plan regularly.

Simulate a failure. Shut down resources in your primary region. See how long it takes to recover. Use AWS CloudWatch metrics to measure your actual RTO and RPO. This is how you find the weak spots before a real disaster hits. It’s like fire drills for your application.

The takeaway, be prepared, not scared

Disaster recovery might seem daunting, but it doesn’t have to be. AWS provides the tools, and with a bit of planning and testing, you can build a resilient architecture that can weather the storm. It’s about peace of mind, knowing that your business can keep running, no matter what. Start small, test often, and build up your defenses over time.

Boost Performance and Resilience with AWS EC2 Placement Groups

There’s a hidden art to placing your EC2 instances in AWS. It’s not just about spinning up machines and hoping for the best, where they land in AWS’s vast infrastructure can make all the difference in performance, resilience, and cost. This is where Placement Groups come in.

You might have deployed instances before without worrying about placement, and for many workloads, that’s perfectly fine. But when your application needs lightning-fast communication, fault tolerance, or optimized performance, Placement Groups become a critical tool in your AWS arsenal.

Let’s break it down.

What are Placement Groups?

AWS Placement Groups give you control over how your EC2 instances are positioned within AWS’s data centers. Instead of leaving it to chance, you can specify how close, or how far apart, your instances should be placed. This helps optimize either latency, fault tolerance, or a balance of both.

There are three types of Placement Groups: Cluster, Spread, and Partition. Each serves a different purpose, and choosing the right one depends on your application’s needs.

Types of Placement Groups and when to use them

Cluster Placement Groups for speed over everything

Think of Cluster Placement Groups like a Formula 1 pit crew. Every millisecond counts, and your instances need to communicate at breakneck speeds. AWS achieves this by placing them on the same physical hardware, minimizing latency, and maximizing network throughput.

This is perfect for:
✅ High-performance computing (HPC) clusters
✅ Real-time financial trading systems
✅ Large-scale data processing (big data, AI, and ML workloads)

⚠️ The Trade-off: While these instances talk to each other at lightning speed, they’re all packed together on the same hardware. If that hardware fails, everything inside the Cluster Placement Group goes down with it.

Spread Placement Groups for maximum resilience

Now, imagine you’re managing a set of VIP guests at a high-profile event. Instead of seating them all at the same table (risking one bad spill ruining their night), you spread them out across different areas. That’s what Spread Placement Groups do, they distribute instances across separate physical machines to reduce the impact of hardware failure.

Best suited for:
✅ Mission-critical applications that need high availability
✅ Databases requiring redundancy across multiple nodes
✅ Low-latency, fault-tolerant applications

⚠️ The Limitation: AWS allows only seven instances per Availability Zone in a Spread Placement Group. If your application needs more, you may need to rethink your architecture.

Partition Placement Groups, the best of both worlds approach

Partition Placement Groups work like a warehouse with multiple sections, each with its power supply. If one section loses power, the others keep running. AWS follows the same principle, grouping instances into multiple partitions spread across different racks of hardware. This provides both high performance and resilience, a sweet spot between Cluster and Spread Placement Groups.

Best for:
✅ Distributed databases like Cassandra, HDFS, or Hadoop
✅ Large-scale analytics workloads
✅ Applications needing both performance and fault tolerance

⚠️ AWS’s Partitioning Rule: The number of partitions you can use depends on the AWS Region, and you must carefully plan how instances are distributed.

How to Configure Placement Groups

Setting up a Placement Group is straightforward, and you can do it using the AWS Management Console, AWS CLI, or an SDK.

Example using AWS CLI

Let’s create a Cluster Placement Group:

aws ec2 create-placement-group --group-name my-cluster-group --strategy cluster

Now, launch an instance into the group:

aws ec2 run-instances --image-id ami-12345678 --count 1 --instance-type c5.large --placement GroupName=my-cluster-group

For Spread and Partition Placement Groups, simply change the strategy:

aws ec2 create-placement-group --group-name my-spread-group --strategy spread
aws ec2 create-placement-group --group-name my-partition-group --strategy partition

Best practices for using Placement Groups

🚀 Combine with Multi-AZ Deployments: Placement Groups work within a single Availability Zone, so consider spanning multiple AZs for maximum resilience.

📊 Monitor Network Performance: AWS doesn’t guarantee placement if your instance type isn’t supported or there’s insufficient capacity. Always benchmark your performance after deployment.

💰 Balance Cost and Performance: Cluster Placement Groups give the fastest network speeds, but they also increase failure risk. If high availability is critical, Spread or Partition Groups might be a better fit.

Final thoughts

AWS Placement Groups are a powerful but often overlooked feature. They allow you to maximize performance, minimize downtime, and optimize costs, but only if you choose the right type.

The next time you deploy EC2 instances, don’t just launch them randomly, placement matters. Choose wisely, and your infrastructure will thank you for it.

How to ensure high availability for pods in Kubernetes

I was thinking the other day about these Kubernetes pods, and how they’re like little spaceships floating around in the cluster. But what happens if one of those spaceships suddenly vanishes? Poof! Gone! That’s a real problem. So I started wondering, how can we ensure our pods are always there, ready to do their job, even if things go wrong? It’s like trying to keep a juggling act going while someone’s moving the floor around you…

Let me tell you about this tool called Karpenter. It’s like a super-efficient hotel manager for our Kubernetes worker nodes, always trying to arrange the “guests” (our applications) most cost-effectively. Sometimes, this means moving guests from one room to another to save on operating costs. In Kubernetes terminology, we call this “consolidation.”

The dancing pods challenge

Here’s the thing: We have this wonderful hotel manager (Karpenter) who’s doing a fantastic job, keeping costs down by constantly optimizing room assignments. But what about our guests (the applications)? They might get a bit dizzy with all this moving around, and sometimes, their important work gets disrupted.

So, the question is: How do we keep our applications running smoothly while still allowing Karpenter to do its magic? It’s like trying to keep a circus performance going while the stage crew rearranges the set in the middle of the act.

Understanding the moving parts

Before we explore the solutions, let’s take a peek behind the scenes and see what happens when Karpenter decides to relocate our applications. It’s quite a fascinating process:

First, Karpenter puts up a “Do Not Disturb” sign (technically called a taint) on the node it wants to clear. Then, it finds new accommodations for all the applications. Finally, it carefully moves each application to its new location.

Think of it as a well-choreographed dance where each step must be perfectly timed to avoid any missteps.

The art of high availability

Now, for the exciting part, we have some clever tricks up our sleeves to ensure our applications keep running smoothly:

The buddy system: The first rule of high availability is simple: never go it alone! Instead of running a single instance of your application, run at least two. It’s like having a backup singer, if one voice falters, the show goes on. In Kubernetes, we do this by setting replicas: 2 in our deployment configuration.
Strategic placement: Here’s a neat trick: we can tell Kubernetes to spread our application copies across different physical machines. It’s like not putting all your eggs in one basket. We use something called “Pod Topology Spread Constraints” for this. Here’s how it looks in practice:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: your-app

Setting boundaries: Remember when your parents set rules about how many cookies you could eat? We do something similar in Kubernetes with PodDisruptionBudgets (PDB). We tell Kubernetes, “Hey, you must always keep at least 50% of my application instances running.” This prevents our hotel manager from getting too enthusiastic about rearranging things.
The “Do Not Disturb” sign: For those special cases where we absolutely don’t want an application to be moved, we can put up a permanent “Do Not Disturb” sign using the karpenter.sh/do-not-disrupt: “true” annotation. It’s like having a VIP guest who gets to keep their room no matter what.

The complete picture

The beauty of this system lies in how all the pieces work together. Think of it as a safety net with multiple layers:

Multiple instances ensure basic redundancy.
Strategic placement keeps instances separated.
PodDisruptionBudgets prevent too many moves at once.
And when necessary, we can completely prevent disruption.

A real example

Let me paint you a picture. Imagine you’re running a critical web service. Here’s how you might set it up:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: critical-web-service
spec:
  replicas: 2
  template:
    metadata:
      annotations:
        karpenter.sh/do-not-disrupt: "false"  # We allow movement, but with controls
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: critical-web-service-pdb
spec:
  minAvailable: 50%
  selector:
    matchLabels:
      app: critical-web-service

The result

With these patterns in place, our applications become incredibly resilient. They can handle node failures, scale smoothly, and even survive Karpenter’s optimization efforts without any downtime. It’s like having a self-healing system that keeps your services running no matter what happens behind the scenes.

High availability isn’t just about having multiple copies of our application, it’s about thoughtfully designing how those copies are managed and maintained. By understanding and implementing these patterns, we are not just running applications in Kubernetes; we are crafting reliable, resilient services that can weather any storm.

The next time you deploy an application to Kubernetes, think about these patterns. They might just save you from that dreaded 3 AM wake-up call about your service being down!

December 15, 2024 by Fernando SRE DevOps stuff Kubernetes SRE stuff 0

Building resilient AWS infrastructure

Imagine building a house of cards. One slight bump and the whole structure comes tumbling down. Now, imagine if your AWS infrastructure was like that house of cards, a scary thought, right? That’s exactly what happens when we have single points of failure in our cloud architecture.

We’re setting ourselves up for trouble when we build systems that depend on a single critical component. If that component fails, the entire system can come crashing down like the house of cards. Instead, we want our infrastructure to resemble a well-engineered skyscraper: stable, robust, and designed with the foresight that no one piece can bring everything down. By thinking ahead and using the right tools, we can build systems that are resilient, adaptable, and ready for anything.

Why should you care about high availability?

Let me start with a story. A few years ago, a major e-commerce company lost millions in revenue when its primary database server crashed during Black Friday. The problem? They had no redundancy in place. It was like trying to cross a river with just one bridge, when that bridge failed, they were completely stuck. In the cloud era, having a single point of failure isn’t just risky, it’s entirely avoidable.

AWS provides us with incredible tools to build resilient systems, kind of like having multiple bridges, boats, and even helicopters to cross that river. Let’s explore how to use these tools effectively.

Starting at the edge with DNS and content delivery

Think of DNS as the reception desk of your application. AWS Route 53, its DNS service, is like having multiple receptionists who know exactly where to direct your visitors, even if one of them takes a break. Here’s how to make it bulletproof:

Health checks: Route 53 constantly monitors your endpoints, like a vigilant security guard. If something goes wrong, it automatically redirects traffic to healthy resources.
Multiple routing policies: You can set up different routing rules based on:
- Geolocation: Direct users based on their location.
- Latency: Route traffic to the endpoint that provides the lowest latency.
- Failover: Automatically direct users to a backup endpoint if the primary one fails.
Application recovery controller: Think of this as your disaster recovery command center. It manages complex failover scenarios automatically, giving you the control needed to respond effectively.

But we can make things even better. CloudFront, AWS’s content delivery network, acts like having local stores in every neighborhood instead of one central warehouse. This ensures users get data from the closest location, reducing latency. Add AWS Shield and WAF, and you’ve got bouncers at every door, protecting against DDoS attacks and malicious traffic.

The load balancing dance

Load balancers are like traffic cops directing cars at a busy intersection. The key is choosing the right one:

Application Load Balancer (ALB): Ideal for HTTP/HTTPS traffic, like a sophisticated traffic controller that knows where each type of vehicle needs to go.
Network Load Balancer (NLB): When you need ultra-high performance and static IP addresses, think of it as an express lane for your traffic, ideal for low-latency use cases.
Cross-Zone Load Balancing: This is a feature of both Application Load Balancers (ALB) and Network Load Balancers (NLB). It ensures that even if one availability zone is busier than others, the traffic gets distributed evenly like a good parent sharing cookies equally among children.

The art of auto scaling

Auto Scaling Groups (ASG) are like having a smart hiring manager who knows exactly when to bring in more help and when to reduce staff. Here’s how to make them work effectively:

Multiple Availability Zones: Never put all your eggs in one basket. Spread your instances across different AZs to avoid single points of failure.
Launch Templates: Think of these as detailed job descriptions for your instances. They ensure consistency in the configuration of your resources and make it easy to replicate settings whenever needed.
Scaling Policies: Use CloudWatch alarms to trigger scaling actions based on metrics like CPU usage or request count. For instance, if CPU utilization exceeds 70%, you can automatically add more instances to handle the increased load. This type of setup is known as a Tracking Policy, where scaling actions are determined by monitoring key metrics like CPU utilization.

The backend symphony

Your backend layer needs to be as resilient as the front end. Here’s how to achieve that:

Stateless design: Each server should be like a replaceable worker, able to handle any task without needing to remember previous interactions. Stateless servers make scaling easier and ensure that no single instance becomes critical.
Caching strategy: ElastiCache acts like a team’s shared notebook, frequently needed information is always at hand. This reduces the load on your databases and improves response times.
Message queuing: Services like SQS and MSK ensure that if one part of your system gets overwhelmed, messages wait patiently in line instead of getting lost. This decouples your components, making the whole system more resilient.

The data layer foundation

Your data layer is like the foundation of a building, it needs to be rock solid. Here’s how to achieve that:

RDS Multi-AZ: Your database gets a perfect clone in another availability zone, ready to take over in milliseconds if needed. This provides fault tolerance for critical data.
DynamoDB Global tables: Think of these as synchronized notebooks in different offices around the world. They allow you to read and write data across regions, providing low-latency access and redundancy.
Aurora Global Database: Imagine having multiple synchronized libraries across different continents. With Aurora, you get global resilience with fast failover capabilities that ensure continuity even during regional outages.

Monitoring and management

You need eyes and ears everywhere in your infrastructure. Here’s how to set that up:

AWS Systems Manager: This serves as your central command center for configuration management, enabling you to automate operational tasks across your AWS resources.
CloudWatch: It’s your all-seeing eye for monitoring and alerting. Set alarms for resource usage, errors, and performance metrics to get notified before small issues escalate.
AWS Config: It’s like having a compliance officer, constantly checking that everything in your environment follows the rules and best practices. By the way, Have you wondered how to configure AWS Config if you need to apply its rules across an infrastructure spread over multiple regions? We’ll cover that in another article.

Best practices and common pitfalls

Here are some golden rules to live by:

Regular testing: Don’t wait for a real disaster to test your failover mechanisms. Conduct frequent disaster recovery drills to ensure that your systems and teams are ready for anything.
Documentation: Keep clear runbooks, they’re like instruction manuals for your infrastructure, detailing how to respond to incidents and maintain uptime.
Avoid these common mistakes:
- Forgetting to test failover procedures
- Neglecting to monitor all components, especially those that may seem trivial
- Assuming that AWS services alone guarantee high availability, resilience requires thoughtful architecture

In a Few Words

Building a truly resilient AWS infrastructure is like conducting an orchestra, every component needs to play its part perfectly. But with careful planning and the right use of AWS services, you can create a system that stays running even when things go wrong.

The goal isn’t just to eliminate single points of failure, it’s to build an infrastructure so resilient that your users never even notice when something goes wrong. Because the best high-availability system is one that makes downtime invisible.

Ready to start building your bulletproof infrastructure? Start with one component at a time, test thoroughly, and gradually build up to a fully resilient system. Your future self (and your users) will thank you for it.

Route 53 --> Alias Record --> External Load Balancer --> ASG (Front Layer)
           |                                                     |
           v                                                     v
       CloudFront                               Internal Load Balancer
           |                                                     |
           v                                                     v
  AWS Shield + WAF                                    ASG (Back Layer)
                                                             |
                                                             v
                                              Database (Multi-AZ) or 
                                              DynamoDB Global

November 29, 2024 by Fernando SRE Cloud stuff 0

Types of Failover in Amazon Route 53 Explained Easily

Imagine Amazon Route 53 as a city’s traffic control system that directs cars (internet traffic) to different streets (servers or resources) based on traffic conditions and road health (the health and configuration of your AWS resources).

Active-Active Failover

In an active-active scenario, you have two streets leading to your destination (your website or application), and both are open to traffic all the time. If one street gets blocked (a server fails), traffic simply continues flowing through the other street. This is useful when you want to balance the load between two resources that are always available.

Active-active failover gives you access to all resources during normal operation. In this example, both region 1 and region 2 are active all the time. When a resource becomes unavailable, Route 53 can detect that it’s unhealthy and stop including it when responding to queries.

Active-Passive Failover

In active-passive failover, you have one main street that you prefer all traffic to use (the primary resource) and a secondary street that’s only used if the main one is blocked (the secondary resource is activated only if the primary fails). This method is useful when you have a preferred resource to handle requests but need a backup in case it fails.

Use an active-passive failover configuration when you want a primary resource or group of resources to be available the majority of the time and you want a secondary resource or group of resources to be on standby in case all the primary resources become unavailable.

Configuring Active-Passive Failover with One Primary and One Secondary Resource

This approach is like having one big street and one small street. You use the big street whenever possible because it can handle more traffic or get you to your destination more directly. You only use the small street if there’s construction or a blockage on the big street.

Configuring Active-Passive Failover with Multiple Primary and Secondary Resources

Now imagine you have several big streets and several small streets. All the big ones are your preferred options, and all the small ones are your backup options. Depending on how many big streets are available, you’ll direct traffic to them before considering using the small ones.

Configuring Active-Passive Failover with Weighted Records

This is like having multiple streets leading to your destination, but you give each street a “weight” based on how often you want it used. Some streets (resources) are preferred more than others, and that preference is adjusted by weight. You still have a backup street for when your preferred options aren’t available.

Evaluating Target Health

“Evaluate Target Health” is like having traffic sensors that instantly tell you if a street is blocked. If you’re routing traffic to AWS resources for which you can create alias records, you don’t need to set up separate health checks for those resources. Instead, you enable “Evaluate Target Health” on your alias records, and Route 53 will automatically check the health of those resources. This simplifies setup and keeps your traffic flowing to streets (resources) that are open and healthy without needing additional health configurations.

In short, Amazon Route 53 offers a powerful set of tools that you can use to manage the availability and resilience of your applications through a variety of ways to apply failover configurations. Implementation of such knowledge into the practice of failover strategy will result in keeping your application up and available for the users in cases when any kind of resource fails or gets a downtime outage.

April 1, 2024 by Fernando SRE Cloud stuff SRE stuff 0