EKS

Kubernetes 1.33 practical insights for developers and Ops

As cloud-native technologies rapidly advance, Kubernetes stands out as a fundamental orchestrator, persistently evolving to address the intricate requirements of modern applications. The arrival of version 1.33, codenamed “Octarine” and released on April 23, 2025, marks another significant stride in this journey. With 64 enhancements spanning security, scalability, and the developer experience, this isn’t merely an incremental update; it’s a thoughtful refinement designed to make our digital infrastructure more robust and intuitive. Let’s explore the most impactful upgrades and understand their practical significance for those of us building and maintaining systems in this ecosystem.

Resources adapting seamlessly with In-Place vertical scaling

A notable advancement in Kubernetes 1.33 is the beta introduction of in-place vertical scaling. This allows pods to adjust their CPU and memory allocations without the disruption of a restart. Consider the agility of upgrading your laptop’s RAM or CPU while you’re in the middle of a crucial video conference, with no need to reboot and rejoin. For applications experiencing unpredictable surges or lulls in demand, this capability means resources can be fine-tuned dynamically. The direct benefits are twofold: reduced downtime, ensuring service continuity, and minimized over-provisioning, leading to more efficient resource utilization. This isn’t just about saving resources; it’s about enabling applications to breathe, to adapt instantaneously to real-world demands, ensuring a consistently smooth experience.

To experiment with this, enable the InPlacePodVerticalScaling feature gate and patch a running Pod’s resources.requests to observe Kubernetes manage the change live.

Sidecar containers reliable companions now generally available

Sidecar containers, those trusty auxiliary units that handle tasks like logging, metrics collection, or traffic proxying, have now graduated to General Availability (GA). They are implemented as a specialized class of init containers, distinguished by a restartPolicy: Always, ensuring they remain active throughout the entire lifecycle of the main application pod. Think of a well-designed multitool, always at your belt; it’s not the primary focus, but its various functions are indispensable for the main task to proceed smoothly. This GA status provides a stable, reliable contract for developers building layered functionality, such as service mesh proxies, log shippers, or certificate renewers, without the need for custom lifecycle management scripts. It’s a commitment to dependable, integrated support.

Enhanced control over batch processing with indexed jobs

Indexed Jobs also reach General Availability in this release, bringing two significant improvements for managing large-scale parallel tasks. Firstly, retry limits can now be defined per index. This means each task within a larger job can have its backoffLimit. It’s akin to a relay race where each runner has a personal coach and strategy; if one runner stumbles, their specific recovery plan kicks in without automatically sidelining the entire team. Secondly, custom success policies offer more nuanced control over what constitutes a completed job. For instance, you might define success as “at least 80 percent of tasks must be completed successfully,” or specify that certain critical tasks absolutely must be finished. For complex data pipelines, these enhancements mean they can fail fast where necessary for specific tasks, yet persevere resiliently for others, ultimately saving valuable time and compute resources.

Service account tokens are more secure and informative

In the critical domain of security, ServiceAccount tokens have been enhanced and are now Generally Available. These tokens now embed a unique JTI (JWT ID) and the node identity of the pod from which they originate. This is like upgrading a standard ID card to a modern digital passport, which not only shows your photo but also includes a tamper-proof serial number and details of where it was issued. The practical implication is significant: token leakage becomes easier to detect and problematic tokens can be revoked with greater precision. Furthermore, admission controllers gain richer contextual information, enabling them to make more informed and granular policy decisions, thereby strengthening the overall security posture of the cluster.

Simplified Kubernetes Resource Interaction with Kubectl Subresource Support

Interacting with specific aspects of Kubernetes resources has become more straightforward. The –subresource flag is now fully and generally supported across common kubectl commands such as get, patch, edit, apply, and replace. Previously, managing subresources like status or scale often required more complex commands or direct YAML manipulation. This change is like having a single, intuitive universal remote that seamlessly controls every function of your entertainment system, eliminating the need for a confusing pile of separate remotes. For developers and operators, this means common actions on subresources are simpler and require less bespoke scripting.

Seamless network growth through dynamic service CIDR expansion

Addressing the needs of growing clusters, Kubernetes 1.33 introduces alpha support for the dynamic expansion of Service Classless Inter-Domain Routings (CIDRs). This allows administrators to add new IP address ranges for ClusterIP services by applying new ServiceCIDR objects. Imagine your home Wi-Fi network needing to accommodate a sudden influx of guests and their devices; this feature is like being able to effortlessly add more wireless channels or IP ranges without disrupting anyone already connected. For clusters that risk outgrowing their initial IP address pool, this provides a vital mechanism to scale network capacity without resorting to painful redeployments or risking IP address conflicts.

Stronger tenant isolation using user namespaces for pods

Enhancing security and isolation in multi-tenant environments, beta support for user namespaces in pods is a welcome addition. This feature enables the mapping of container User IDs (UIDs) and Group IDs (GIDs) to a distinct, unprivileged range on the host system. Consider an apartment building where each unit has its own completely separate utility meters and internal wiring, distinct from the building’s main infrastructure. A fault or surge in one apartment doesn’t affect the others. Similarly, user namespaces provide a stronger boundary, reducing the potential blast radius of privilege-escalation exploits by ensuring that even if a process breaks out of its container, it doesn’t gain root privileges on the underlying node.

Simplified tooling and configuration delivery via OCI image mounting

The delivery of tools, configurations, and other artifacts to pods is streamlined with the new alpha capability to mount Open Container Initiative (OCI) images and other OCI artifacts directly as read-only volumes. This is like being able to instantly snap a pre-packaged, specialized toolkit directly onto your workbench exactly when and where you need it, rather than having to unpack a large, cumbersome toolbox and sort through every individual item. This approach allows teams to share binaries, licenses, or even large language models without the need to bake them into base container images or uncompress tarballs at runtime, simplifying workflows and image management.

A more graceful departure with ordered namespace deletion

Managing the lifecycle of resources effectively includes ensuring their clean removal. Kubernetes 1.33 introduces an alpha implementation of a more structured, dependency-aware cleanup sequence when a namespace is deleted. Think of a professional stage crew dismantling a complex concert setup; they don’t just start pulling things down randomly. They follow a precise order, ensuring microphones are unplugged before speakers are moved, and lights are detached before rigging is lowered. This ordered deletion process helps prevent dangling resources, orphaned secrets, and other inconsistencies, contributing to a tidier, more secure, and more reliably managed cluster.

Looking ahead other highlights and farewell to old friends

While we’ve focused on some of the most prominent changes, the “Octarine” release encompasses 64 enhancements in total. Other notable improvements include updates to IPv6 NAT, better Container Runtime Interface (CRI) metrics, and faster kubectl get operations for extensive Custom Resource lists.

As Kubernetes evolves, some features are also retired to make way for more modern, secure, or scalable alternatives:

The Endpoints API is deprecated in favor of the more scalable EndpointSlice API.
The gitRepo volume type has been removed due to security concerns; users are encouraged to use Container Storage Interface (CSI) drivers or other methods for managing code.
hostNetwork support on Windows Pods is being retired due to inconsistencies and the availability of more robust alternative networking solutions.

This process is natural, much like replacing an aging, corded power drill with a more versatile and safer cordless model once superior technology becomes available.

Kubernetes continues its evolution for you

Kubernetes 1.33 “Octarine” clearly demonstrates the project’s ongoing commitment to addressing real-world challenges faced by developers and platform operators. From the operational flexibility of in-place scaling and the robust contract of GA sidecars to smarter batch job processing and thoughtful security reinforcements, these changes collectively pave the way for smoother operations, more resilient applications, and a more empowered development community.

Whether you are managing sprawling production clusters or experimenting in a modest homelab, adopting version 1.33 is less about chasing the newest version number and more about unlocking tangible, practical gains. These are the kinds of improvements that free you up to focus on innovation, ship features with greater confidence, and perhaps even sleep a little easier. The evolution of Kubernetes continues, and it’s a journey that consistently brings valuable enhancements to its vast community of users.

Kubernetes networking bottlenecks explained and resolved

Your Kubernetes cluster is humming along, orchestrating containers like a conductor leading an orchestra. Suddenly, the music falters. Applications lag, connections drop, and users complain. What happened? Often, the culprit hides within the cluster’s intricate communication network, causing frustrating bottlenecks. Just like a city’s traffic can grind to a halt, so too can the data flow within Kubernetes, impacting everything that relies on it.

This article is for you, the DevOps engineer, the cloud architect, the platform administrator, and anyone tasked with keeping Kubernetes clusters running smoothly. We’ll journey into the heart of Kubernetes networking, uncover the common causes of these slowdowns, and equip you with the knowledge to diagnose and resolve them effectively. Let’s turn that traffic jam back into a superhighway.

The unseen traffic control, Kubernetes networking, and CNI

At the core of every Kubernetes cluster’s communication lies the Container Network Interface, or CNI. Think of it as the sophisticated traffic management system for your digital city. It’s a crucial plugin responsible for assigning IP addresses to pods and managing how data packets navigate the complex connections between them. Without a well-functioning CNI, pods can’t talk, services become unreachable, and the system stutters.

Several CNI plugins act as different traffic control strategies:

Flannel: Often seen as a straightforward starting point, using overlay networks. Good for simpler setups, but can sometimes act like a narrow road during rush hour.
Calico: A more advanced option, known for high performance and robust network policy enforcement. Think of it as having dedicated express lanes and smart traffic signals.
Weave Net: A user-friendly choice for multi-host networking, though its ease of use might come with a bit more overhead, like adding extra toll booths.

Choosing the right CNI isn’t just a technical detail; it’s fundamental to your cluster’s health, depending on your needs for speed, complexity, and security.

Spotting the digital traffic jam, recognizing network bottlenecks

How do you know if your Kubernetes network is experiencing a slowdown? The signs are usually clear, though sometimes subtle:

Increased latency: Applications take noticeably longer to respond. Simple requests feel sluggish.
Reduced throughput: Data transfer speeds drop, even if your nodes seem to have plenty of CPU and memory.
Connection issues: You might see frequent timeouts, dropped connections, or intermittent failures for pods trying to communicate.

It feels like hitting an unexpected gridlock on what should be an open road. Even minor network hiccups can cascade, causing significant delays across your applications.

Unmasking the culprits, common causes for Bottlenecks

Network bottlenecks don’t appear out of thin air. They often stem from a few common culprits lurking within your cluster’s configuration or resources:

CNI Plugin performance limits: Some CNIs, especially simpler overlay-based ones like Flannel in certain configurations, have inherent throughput limitations. Pushing too much traffic through them is like forcing rush hour traffic onto a single-lane country road.
Node resource starvation: Packet processing requires CPU and memory on the worker nodes. If nodes are starved for these resources, the CNI can’t handle packets efficiently, much like an underpowered truck struggling to climb a steep hill.
Configuration glitches: Incorrect CNI settings, mismatched MTU (Maximum Transmission Unit) sizes between nodes and pods, or poorly configured routing rules can create significant inefficiencies. It’s like having traffic lights horribly out of sync, causing jams instead of flow.
Scalability hurdles: As your cluster grows, especially with high pod density per node, you might exhaust available IP addresses or simply overwhelm the network paths. This is akin to a city intersection completely gridlocked because far too many vehicles are trying to pass through simultaneously.

The investigation, diagnosing the problem systematically

Finding the exact cause requires a methodical approach, like a detective gathering clues at a crime scene. Don’t just guess; investigate:

Start with kubectl: This is your primary investigation tool. Examine pod statuses (kubectl get pods -o wide), check logs (kubectl logs <pod-name>), and test basic connectivity between pods (kubectl exec <pod-name> — ping <other-pod-ip>). Are pods running? Can they reach each other? Are there revealing error messages?
Deploy monitoring tools: You need visibility. Tools like Prometheus for metrics collection and Grafana for visualization are invaluable. They act as your network’s vital signs monitor.
Track key network metrics: Focus on the critical indicators:
- Latency: How long does communication take between pods, nodes, and external services?
- Packet loss: Are data packets getting dropped during transmission?
- Throughput: How much data is flowing through the network compared to expectations?
- Error rates: Are network interfaces reporting errors?

Analyzing these metrics helps pinpoint whether the issue is widespread or localized, related to specific nodes or applications. It’s like a mechanic testing different systems in a car to isolate the fault.

Paving the way for speed, proven solutions, and best practices

Once you’ve diagnosed the bottleneck, it’s time to implement solutions. Think of this as upgrading the city’s infrastructure to handle more traffic smoothly:

Choose the right CNI for the job: If your current CNI is hitting limits, consider migrating to a higher-performance option better suited for heavy workloads or complex network policies, such as Calico or Cilium. Evaluate based on benchmarks and your specific traffic patterns.
Optimize CNI configuration: Dive into the settings. Ensure the MTU is configured correctly across your infrastructure (nodes, pods, underlying network). Fine-tune routing configurations specific to your CNI (e.g., BGP peering with Calico). Small tweaks here can yield significant improvements.
Scale your infrastructure wisely: Sometimes, the nodes themselves are the bottleneck. Add more CPU or memory to existing nodes, or scale out by adding more nodes to distribute the load. Ensure your underlying network infrastructure can handle the increased traffic.
Tune overlay networks or go direct: If using overlay networks (like VXLAN used by Flannel or some Calico modes), ensure they are tuned correctly. For maximum performance, explore direct routing options where packets go directly between nodes without encapsulation, if supported by your CNI and infrastructure (e.g., Calico with BGP).

Success stories, from gridlock to open roads

The theory is good, but the results matter. Consider a team battling high latency in a densely packed cluster running Flannel. Diagnosis pointed to overlay network limitations. By migrating to Calico configured with BGP for more direct routing, they drastically cut latency and improved application responsiveness.

In another case, intermittent connectivity issues plagued a cluster during peak loads. Monitoring revealed CPU saturation on specific worker nodes handling heavy network traffic. Upgrading these nodes with more CPU resources immediately stabilized connectivity and boosted packet processing speeds. These examples show that targeted fixes, guided by proper diagnosis, deliver real performance gains.

Keeping the traffic flowing

Tackling Kubernetes networking bottlenecks isn’t just about fixing a technical problem; it’s about ensuring the reliability and scalability of the critical services your cluster hosts. A smooth, efficient network is the foundation upon which robust applications are built.

Just as a well-managed city anticipates and manages traffic flow, proactively monitoring and optimizing your Kubernetes network ensures your digital services remain responsive and ready to scale. Keep exploring, and keep learning, advanced CNI features and network policies offer even more avenues for optimization. Remember, a healthy Kubernetes network paves the way for digital innovation.

May 4, 2025 by Fernando SRE DevOps stuff Kubernetes SRE stuff

Why simplicity wins when you pick AWS ECS Fargate instead of EKS

Selecting the right tools often feels like navigating a crossroads. Consider planning a significant project, like building a custom home workshop. You could opt for a complex setup with specialized, industrial-grade machinery (powerful, flexible, demanding maintenance and expertise). Or, you might choose high-quality, standard power tools that handle 90% of your needs reliably and with far less fuss. Development teams deploying containers on AWS face a similar decision. The powerful, industry-standard Kubernetes via Elastic Kubernetes Service (EKS) beckons, but is it always the necessary path? Often, the streamlined native solution, Elastic Container Service (ECS) paired with its serverless Fargate launch type, offers a smarter, more efficient route.

AWS presents these two primary highways for container orchestration. EKS delivers managed Kubernetes, bringing its vast ecosystem and flexibility. It frequently dominates discussions and is hailed in the DevOps world. But then there’s ECS, AWS’s own mature and deeply integrated orchestrator. This article explores the compelling scenarios where choosing the apparent simplicity of ECS, particularly with Fargate, isn’t just easier; it’s strategically better.

Getting to know your AWS container tools

Before charting a course, let’s clarify what each service offers.

ECS (Elastic Container Service): Think of ECS as the well-designed, built-in toolkit that comes standard with your AWS environment. It’s AWS’s native container orchestrator, designed for seamless integration. ECS offers two ways to run your containers:

EC2 launch type: You manage the underlying EC2 virtual machine instances yourself. This gives you granular control over the instance type (perhaps you need specific GPUs or network configurations) but brings back the responsibility of patching, scaling, and managing those servers.
Fargate launch type: This is the serverless approach. You define your container needs, and Fargate runs them without you ever touching, or even seeing, the underlying server infrastructure.

Fargate: This is where serverless container execution truly shines. It’s like setting your high-end camera to an intelligent ‘auto’ mode. You focus on the shot (your application), and the camera (Fargate) expertly handles the complex interplay of aperture, shutter speed, and ISO (server provisioning, scaling, patching). You simply run containers.

EKS (Elastic Kubernetes Service): EKS is AWS’s managed offering for the Kubernetes platform. It’s akin to installing a professional-grade, multi-component software suite onto your operating system. It provides immense power, conforms to the Kubernetes standard loved by many, and grants access to its sprawling ecosystem of tools and extensions. However, even with AWS managing the control plane’s availability, you still need to understand and configure Kubernetes concepts, manage worker nodes (unless using Fargate with EKS, which adds its own considerations), and handle integrations.

The power of keeping things simple with ECS Fargate

So, what makes this simpler path with ECS Fargate so appealing? Several key advantages stand out.

Reduced operational overhead: This is often the most significant win. Consider the sheer liberation Fargate offers: it completely removes the burden of managing the underlying servers. Forget patching operating systems at 2 AM or figuring out complex scaling policies for your EC2 fleet. It’s the difference between owning a car, with all its maintenance chores, oil changes, tire rotations, and unexpected repairs, and using a seamless rental or subscription service where the vehicle is just there when you need it, ready to drive. You focus purely on the journey (your application), not the engine maintenance (the infrastructure).

Faster learning curve and easier management: ECS generally presents a gentler learning curve than the multifaceted world of Kubernetes. For teams already comfortable within the AWS ecosystem, ECS concepts feel intuitive and familiar. Managing task definitions, services, and clusters in ECS is often more straightforward than navigating Kubernetes deployments, services, pods, and the YAML complexities involved. This translates to faster onboarding and less time spent wrestling with the orchestrator itself. Furthermore, EKS carries an hourly cost for its control plane (though free tiers exist), an expense absent in the standard ECS setup.

Seamless AWS integration: ECS was born within AWS, and it shows. Its integration with other AWS services is typically tighter and simpler to configure than with EKS. Assigning IAM roles directly to ECS tasks for granular permissions, for instance, is remarkably straightforward compared to setting up Kubernetes Service Accounts and configuring IAM Roles for Service Accounts (IRSA) with an OIDC provider in EKS. Connecting to Application Load Balancers, registering targets, and pushing logs and metrics to CloudWatch often requires less configuration boilerplate with ECS/Fargate. It’s like your home’s electrical system being designed for standard plugs, appliances just work without needing special adapters or wiring.

True serverless container experience (Fargate): With Fargate, you pay for the vCPU and memory resources your containerized application requests, consumed only while it’s running. You aren’t paying for idle virtual machines waiting for work. This model is incredibly cost-effective for applications with variable loads, APIs that scale on demand, or batch jobs that run periodically.

Finding your route when ECS Fargate is the best fit

Knowing these advantages, let’s pinpoint the specific road signs indicating ECS/Fargate is the right direction for your team and application.

Teams prioritizing simplicity and velocity: If your primary goal is to ship features quickly and minimize the time spent on infrastructure management, ECS/Fargate is a strong contender. It allows developers to focus more on code and less on orchestration intricacies. It’s like choosing a reliable microwave and stove for everyday cooking; they get the job done efficiently without the complexity of a commercial kitchen setup.

Standard microservices or web applications: Many common workloads, like stateless web applications, APIs, or backend microservices, don’t require the advanced orchestration features or the specific tooling found only in the Kubernetes ecosystem. For these, ECS/Fargate provides robust, scalable, and reliable hosting without unnecessary complexity.

Deep reliance on the AWS ecosystem: If your application heavily leverages other AWS services (like DynamoDB, SQS, Lambda, RDS) and multi-cloud portability isn’t an immediate strategic requirement, ECS/Fargate’s native integration offers tangible benefits in ease of use and configuration.

Serverless-First architectures: For teams embracing a serverless mindset for event-driven processing, data pipelines, or API backends, Fargate fits perfectly. Its pay-per-use model and elimination of server management align directly with serverless principles.

Operational cost sensitivity: When evaluating the total cost of ownership, factor in the human effort. The reduced operational burden of ECS/Fargate can lead to significant savings in staff time and effort, potentially outweighing any differences in direct compute costs or the EKS control plane fee.

Acknowledging the alternative when EKS remains the champion

Of course, EKS exists for good reasons, and it remains the superior choice in certain contexts. Let’s be clear about when you need that powerful, customizable machinery.

Need for Kubernetes Standard/API: If your team requires the full Kubernetes API, needs specific Custom Resource Definitions (CRDs), operators, or advanced scheduling capabilities inherent to Kubernetes, EKS is the way to go.

Leveraging the vast Kubernetes ecosystem: Planning to use popular Kubernetes-native tools like Helm for packaging, Argo CD for GitOps, Istio or Linkerd for a service mesh, or specific monitoring agents designed for Kubernetes? EKS provides the standard platform these tools expect.

Existing Kubernetes expertise or workloads: If your team is already proficient in Kubernetes or you’re migrating existing Kubernetes applications to AWS, sticking with EKS leverages that investment and knowledge, ensuring consistency.

Hybrid or Multi-Cloud strategy: When running workloads across different cloud providers or in hybrid on-premises/cloud environments, Kubernetes (and thus EKS on AWS) provides a consistent orchestration layer, crucial for portability and operational uniformity.

Highly complex orchestration needs: For applications demanding intricate network policies (e.g., using Calico), complex stateful set management, or very specific affinity/anti-affinity rules that might be more mature or flexible in Kubernetes, EKS offers greater depth.

Think of EKS as that specialized, heavy-duty truck. It’s indispensable when you need to haul unique, heavy loads (complex apps), attach specialized equipment (ecosystem tools), modify the engine extensively (custom controllers), or drive consistently across varied terrains (multi-cloud).

Choosing your lane ECS Fargate or EKS

The key insight here isn’t about crowning one service as universally “better.” It’s about recognizing that the AWS container landscape offers different tools meticulously designed for different journeys. ECS with Fargate stands as a powerful, mature, and often much simpler alternative, decisively challenging the notion that Kubernetes via EKS should be the default starting point for every containerized application on AWS.

Before committing, honestly assess your application’s real complexity, your team’s operational capacity, and existing expertise, your reliance on the broader AWS vs. Kubernetes ecosystems, and your strategic goals regarding portability. It’s like packing for a trip: you wouldn’t haul mountaineering equipment for a relaxing beach holiday. Choose the toolset that minimizes friction, maximizes your team’s velocity, and keeps your journey smooth. Choose wisely.

April 28, 2025 by Fernando SRE Cloud stuff DevOps stuff Kubernetes SRE stuff

How to ensure high availability for pods in Kubernetes

I was thinking the other day about these Kubernetes pods, and how they’re like little spaceships floating around in the cluster. But what happens if one of those spaceships suddenly vanishes? Poof! Gone! That’s a real problem. So I started wondering, how can we ensure our pods are always there, ready to do their job, even if things go wrong? It’s like trying to keep a juggling act going while someone’s moving the floor around you…

Let me tell you about this tool called Karpenter. It’s like a super-efficient hotel manager for our Kubernetes worker nodes, always trying to arrange the “guests” (our applications) most cost-effectively. Sometimes, this means moving guests from one room to another to save on operating costs. In Kubernetes terminology, we call this “consolidation.”

The dancing pods challenge

Here’s the thing: We have this wonderful hotel manager (Karpenter) who’s doing a fantastic job, keeping costs down by constantly optimizing room assignments. But what about our guests (the applications)? They might get a bit dizzy with all this moving around, and sometimes, their important work gets disrupted.

So, the question is: How do we keep our applications running smoothly while still allowing Karpenter to do its magic? It’s like trying to keep a circus performance going while the stage crew rearranges the set in the middle of the act.

Understanding the moving parts

Before we explore the solutions, let’s take a peek behind the scenes and see what happens when Karpenter decides to relocate our applications. It’s quite a fascinating process:

First, Karpenter puts up a “Do Not Disturb” sign (technically called a taint) on the node it wants to clear. Then, it finds new accommodations for all the applications. Finally, it carefully moves each application to its new location.

Think of it as a well-choreographed dance where each step must be perfectly timed to avoid any missteps.

The art of high availability

Now, for the exciting part, we have some clever tricks up our sleeves to ensure our applications keep running smoothly:

The buddy system: The first rule of high availability is simple: never go it alone! Instead of running a single instance of your application, run at least two. It’s like having a backup singer, if one voice falters, the show goes on. In Kubernetes, we do this by setting replicas: 2 in our deployment configuration.
Strategic placement: Here’s a neat trick: we can tell Kubernetes to spread our application copies across different physical machines. It’s like not putting all your eggs in one basket. We use something called “Pod Topology Spread Constraints” for this. Here’s how it looks in practice:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: your-app

Setting boundaries: Remember when your parents set rules about how many cookies you could eat? We do something similar in Kubernetes with PodDisruptionBudgets (PDB). We tell Kubernetes, “Hey, you must always keep at least 50% of my application instances running.” This prevents our hotel manager from getting too enthusiastic about rearranging things.
The “Do Not Disturb” sign: For those special cases where we absolutely don’t want an application to be moved, we can put up a permanent “Do Not Disturb” sign using the karpenter.sh/do-not-disrupt: “true” annotation. It’s like having a VIP guest who gets to keep their room no matter what.

The complete picture

The beauty of this system lies in how all the pieces work together. Think of it as a safety net with multiple layers:

Multiple instances ensure basic redundancy.
Strategic placement keeps instances separated.
PodDisruptionBudgets prevent too many moves at once.
And when necessary, we can completely prevent disruption.

A real example

Let me paint you a picture. Imagine you’re running a critical web service. Here’s how you might set it up:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: critical-web-service
spec:
  replicas: 2
  template:
    metadata:
      annotations:
        karpenter.sh/do-not-disrupt: "false"  # We allow movement, but with controls
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: critical-web-service-pdb
spec:
  minAvailable: 50%
  selector:
    matchLabels:
      app: critical-web-service

The result

With these patterns in place, our applications become incredibly resilient. They can handle node failures, scale smoothly, and even survive Karpenter’s optimization efforts without any downtime. It’s like having a self-healing system that keeps your services running no matter what happens behind the scenes.

High availability isn’t just about having multiple copies of our application, it’s about thoughtfully designing how those copies are managed and maintained. By understanding and implementing these patterns, we are not just running applications in Kubernetes; we are crafting reliable, resilient services that can weather any storm.

The next time you deploy an application to Kubernetes, think about these patterns. They might just save you from that dreaded 3 AM wake-up call about your service being down!

December 15, 2024 by Fernando SRE DevOps stuff Kubernetes SRE stuff

How to mount AWS EFS on EKS for scalable storage solutions

Suppose you need multiple applications to share files seamlessly, without worrying about running out of storage space or struggling with complex configurations. That’s where AWS Elastic File System (EFS) comes in. EFS is a fully managed, scalable file system that multiple AWS services or containers can access. In this guide, we’ll take a simple yet comprehensive journey through the process of mounting AWS EFS to an Amazon Elastic Kubernetes Service (EKS) cluster. I’ll make sure to keep it straightforward, so you can follow along regardless of your Kubernetes experience.

Why use EFS with EKS?

Before we go into the details, let’s consider why using EFS in a Kubernetes environment is beneficial. Imagine you have multiple applications (pods) that all need to access the same data—like a shared directory of documents. Instead of replicating data for each application, EFS provides a centralized storage solution that can be accessed by all pods, regardless of which node they’re running on.

Here’s what makes EFS a great choice for EKS:

Shared Storage: Multiple pods across different nodes can access the same files at the same time, making it perfect for workloads that require shared access.
Scalability: EFS automatically scales up or down as your data needs change, so you never have to worry about manually managing storage limits.
Durability and Availability: AWS ensures that your data is highly durable and accessible across multiple Availability Zones (AZs), which means your applications stay resilient even if there are hardware failures.

Typical use cases for using EFS with EKS include machine learning workloads, content management systems, or shared file storage for collaborative environments like JupyterHub.

Prerequisites

Before we start, make sure you have the following:

EKS Cluster: You need a running EKS cluster, and kubectl should be configured to access it.
EFS File System: An existing EFS file system in the same AWS region as your EKS cluster.
IAM Roles: Correct IAM roles and policies for your EKS nodes to interact with EFS.
Amazon EFS CSI Driver: This driver must be installed in your EKS cluster.

How to mount AWS EFS on EKS

Let’s take it step by step, so by the end, you’ll have a working setup where your Kubernetes pods can use EFS for shared, scalable storage.

Create an EFS file system

To begin, navigate to the EFS Management Console:

Create a New File System: Select the appropriate VPC and subnets—they should be in the same region as your EKS cluster.
File System ID: Note the File System ID; you’ll use it later.
Networking: Ensure that your security group allows inbound traffic from the EKS worker nodes. Think of this as permitting EKS to access your storage safely.

Set up IAM role for the EFS CSI driver

The Amazon EFS CSI driver manages the integration between EFS and Kubernetes. For this driver to work, you need to create an IAM role. It’s a bit like giving the CSI driver its set of keys to interact with EFS securely.

To create the role:

Log in to the AWS Management Console and navigate to IAM.
Create a new role and set up a custom trust policy:

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Principal": {
               "Federated": "arn:aws:iam::<account-id>:oidc-provider/oidc.eks.<region>.amazonaws.com/id/<oidc-provider-id>"
           },
           "Action": "sts:AssumeRoleWithWebIdentity",
           "Condition": {
               "StringLike": {
                   "oidc.eks.<region>.amazonaws.com/id/<oidc-provider-id>:sub": "system:serviceaccount:kube-system:efs-csi-*"
               }
           }
       }
   ]
}

Make sure to attach the AmazonEFSCSIDriverPolicy to this role. This step ensures that the CSI driver has the necessary permissions to manage EFS volumes.

Install the Amazon EFS CSI driver

You can install the EFS CSI driver using either the EKS Add-ons feature or via Helm charts. I recommend the EKS Add-on method because it’s easier to manage and stays updated automatically.

Attach the IAM role you created to the EFS CSI add-on in your cluster.

(Optional) Create an EFS access point

Access points provide a way to manage and segregate access within an EFS file system. It’s like having different doors to different parts of the same warehouse, each with its key and permissions.

Go to the EFS Console and select your file system.
Create a new Access Point and note its ID for use in upcoming steps.

Configure an IAM Policy for worker nodes

To make sure your EKS worker nodes can access EFS, attach an IAM policy to their role. Here’s an example policy:

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "elasticfilesystem:DescribeAccessPoints",
               "elasticfilesystem:DescribeFileSystems",
               "elasticfilesystem:ClientMount",
               "elasticfilesystem:ClientWrite"
           ],
           "Resource": "*"
       }
   ]
}

This ensures your nodes can create and interact with the necessary resources.

Create a storage class for EFS

Next, create a Kubernetes StorageClass to provision Persistent Volumes (PVs) dynamically. Here’s an example YAML file:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  fileSystemId: <file-system-id>
  directoryPerms: "700"
  basePath: "/dynamic_provisioning"
  ensureUniqueDirectory: "true"

Replace <file-system-id> with your EFS File System ID.

Apply the file:

kubectl apply -f efs-storage-class.yaml

Create a persistent volume claim (PVC)

Now, let’s request some storage by creating a PersistentVolumeClaim (PVC):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: efs-sc

Apply the PVC:

kubectl apply -f efs-pvc.yaml

Use the EFS PVC in a pod

With the PVC created, you can now mount the EFS storage into a pod. Here’s a sample pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: efs-app
spec:
  containers:
  - name: app
    image: nginx
    volumeMounts:
    - mountPath: "/data"
      name: efs-volume
  volumes:
  - name: efs-volume
    persistentVolumeClaim:
      claimName: efs-pvc

Apply the configuration:

kubectl apply -f efs-pod.yaml

You can verify the setup by checking if the pod can access the mounted storage:

kubectl exec -it efs-app -- ls /data

A note on direct EFS mounting

You can mount EFS directly into pods without using a Persistent Volume (PV) or Persistent Volume Claim (PVC) by referencing the EFS file system directly in the pod’s configuration. This approach simplifies the setup but offers less flexibility compared to using dynamic provisioning with a StorageClass. Here’s how you can do it:

apiVersion: v1
kind: Pod
metadata:
  name: efs-mounted-app
  labels:
    app: efs-example
spec:
  containers:
  - name: nginx-container
    image: nginx:latest
    volumeMounts:
    - name: efs-storage
      mountPath: "/shared-data"
  volumes:
  - name: efs-storage
    csi:
      driver: efs.csi.aws.com
      volumeHandle: <file-system-id>
      readOnly: false

Replace <file-system-id> with your EFS File System ID. This method works well for simpler scenarios where direct access is all you need.

Final remarks

Mounting EFS to an EKS cluster gives you a powerful, shared storage solution for Kubernetes workloads. By following these steps, you can ensure that your applications have access to scalable, durable, and highly available storage without needing to worry about complex management or capacity issues.

As you can see, EFS acts like a giant, shared repository that all your applications can tap into. Whether you’re working on machine learning projects, collaborative tools, or any workload needing shared data, EFS and EKS together simplify the whole process.

Now that you’ve walked through mounting EFS on EKS, think about what other applications could benefit from this setup. It’s always fascinating to see how managed services can help reduce the time you spend on the nitty-gritty details, letting you focus on building great solutions.

December 5, 2024 by Fernando SRE Cloud stuff DevOps stuff Kubernetes SRE stuff

How many pods fit on an AWS EKS node?

Managing Kubernetes workloads on AWS EKS (Elastic Kubernetes Service) is much like managing a city, you need to know how many “tenants” (Pods) you can fit into your “buildings” (EC2 instances). This might sound straightforward, but a bit more is happening behind the scenes. Each type of instance has its characteristics, and understanding the limits is key to optimizing your deployments and avoiding resource headaches.

Why Is there a pod limit per node in AWS EKS?

Imagine you want to deploy several applications as Pods across several instances in AWS EKS. You might think, “Why not cram as many as possible onto each node?” Well, there’s a catch. Every EC2 instance in AWS has a limit on networking resources, which ultimately determines how many Pods it can support.

Each EC2 instance has a certain number of Elastic Network Interfaces (ENIs), and each ENI can hold a certain number of IPv4 addresses. But not all these IP addresses are available for Pods, AWS reserves some for essential services like the AWS CNI (Container Network Interface) and kube-proxy, which helps maintain connectivity and communication across your cluster.

Think of each ENI like an apartment building, and the IPv4 addresses as individual apartments. Not every apartment is available to your “tenants” (Pods), because AWS keeps some for maintenance. So, when calculating the maximum number of Pods for a specific instance type, you need to take this into account.

For example, a t3.medium instance has a maximum capacity of 17 Pods. A slightly bigger t3.large can handle up to 35 Pods. The difference depends on the number of ENIs and how many apartments (IPv4 addresses) each ENI can hold.

Formula to calculate Max pods per EC2 instance

To determine the maximum number of Pods that an instance type can support, you can use the following formula:

Max Pods = (Number of ENIs × IPv4 addresses per ENI) – Reserved IPs

Let’s apply this to a t2.medium instance:

Number of ENIs: 3
IPv4 addresses per ENI: 6

Using these values, we get:

Max Pods = (3 × 6) – 1

Max Pods = 18 – 1

Max Pods = 17

So, a t2.medium instance in EKS can support up to 17 Pods. It’s important to understand that this number isn’t arbitrary, it reflects the way AWS manages networking to keep your cluster running smoothly.

Why does this matter?

Knowing the limits of your EC2 instances can be crucial when planning your Kubernetes workloads. If you exceed the maximum number of Pods, some of your applications might fail to deploy, leading to errors and downtime. On the other hand, choosing an instance that’s too large might waste resources, costing you more than necessary.

Suppose you’re running a city, and you need to decide how many tenants each building can support comfortably. You don’t want buildings overcrowded with tenants, nor do you want them half-empty. Similarly, you need to find the sweet spot in AWS EKS, enough Pods to maximize efficiency, but not so many that your node runs out of resources.

The apartment analogy

Consider an m5.large instance. Let’s say it has 4 ENIs, and each ENI can support 10 IP addresses. But, AWS reserves a few apartments (IPv4 addresses) in each building (ENI) for maintenance staff (essential services). Using our formula, we can estimate how many Pods (tenants) we can fit.

Number of ENIs: 4
IPv4 addresses per ENI: 10

Max Pods = (4 × 10) – 1

Max Pods = 40 – 1

Max Pods = 39

So, an m5.large can support 39 Pods. This limit helps ensure that the building (instance) doesn’t get overwhelmed and that the essential services can function without issues.

Automating the Calculation

Manually calculating these limits can be tedious, especially if you’re managing multiple instance types or scaling dynamically. Thankfully, AWS provides tools and scripts to help automate these calculations. You can use the kubectl describe node command to get insights into your node’s capacity or refer to AWS documentation for Pod limits by instance type. Automating this step saves time and helps you avoid deployment issues.

Best practices for scaling

When planning the architecture of your EKS cluster, consider these best practices:

Match instance type to workload needs: If your application requires many Pods, opt for an instance type with more ENIs and IPv4 capacity.
Consider cost efficiency: Sometimes, using fewer large instances can be more cost-effective than using many smaller ones, depending on your workload.
Leverage autoscaling: AWS allows you to set up autoscaling for both your Pods and your nodes. This can help ensure that you have the right amount of capacity during peak and off-peak times without manual intervention.

Key takeaways

Understanding the Pod limits per EC2 instance in AWS EKS is more than just a calculation, it’s about ensuring your Kubernetes workloads run smoothly and efficiently. By thinking of ENIs as buildings and IP addresses as apartments, you can simplify the complexity of AWS networking and better plan your deployments.

Like any good city planner, you want to make sure there’s enough room for everyone, but not so much that you’re wasting space. AWS gives you the tools, you just need to know how to use them.

November 24, 2024 by Fernando SRE Cloud stuff DevOps stuff Kubernetes SRE stuff

Container deployment in AWS with ECS, EKS, and Fargate

How do the apps you use daily get built, shipped, and scaled so smoothly? A lot of it has to do with the magic of containers. Think of containers like neat little LEGO blocks, self-contained, portable, and ready to snap together to build something awesome. In the tech world, these blocks hold all the essential bits and pieces of an application, making it super easy to move them around and run them anywhere.

Imagine you’ve got a bunch of these LEGO blocks, each representing a different part of your app. You’ll need a good way to organize them, right? That’s where container orchestration comes in. It’s like having a master builder who knows how to put those blocks together, make sure they’re all playing nicely, and even create more blocks when things get busy.

And guess what? AWS, the cloud superhero, has a whole toolkit to help you with this container adventure.

AWS container services toolkit

AWS offers a variety of services that work together like a well-oiled machine to help you build, deploy, and manage your containerized applications.

Amazon Elastic Container Registry (ECR) – Your container garage

Think of ECR as your very own garage for storing container images. It’s a fully managed service that allows you to store, share, and deploy your container images securely. ECR is like a safe and organized space where you keep all your valuable LEGO creations. You can easily control who has access to your images, making sure only the right people can use them. Plus, it integrates seamlessly with other AWS services, making it a breeze to include in your workflows.

Amazon Elastic Container Service (ECS) – Your container playground

Once you’ve got your container images stored safely in ECR, what’s next? Meet ECS, your container playground! ECS is a highly scalable and high-performance container orchestration service that allows you to run and manage your containers on a cluster of Amazon EC2 instances. It’s like having a dedicated play area where you can arrange your LEGO blocks, build amazing structures, and even add or remove blocks as needed. ECS takes care of all the heavy lifting, so you can focus on what matters most, building awesome applications.

Amazon Elastic Kubernetes Service (EKS) – Your Kubernetes command center

For those of you who prefer the Kubernetes way of doing things, AWS has you covered with EKS. It’s a managed Kubernetes service that makes it easy to run Kubernetes on AWS without having to worry about managing the underlying infrastructure. Kubernetes is like a super-sophisticated set of instructions for building complex LEGO structures. EKS takes care of all the complexities of managing Kubernetes so that you can focus on building and deploying your applications.

EC2 vs. Fargate – Choosing your foundation

Now, let’s talk about the foundation of your container playground. You have two main options: EC2 and Fargate.

EC2-based container deployment – The DIY approach

With EC2, you get full control over the underlying infrastructure. It’s like building your own LEGO table from scratch. You choose the size, shape, and color of the table, and you’re responsible for keeping it clean and tidy. This gives you a lot of flexibility, but it also means you have more responsibilities.

AWS Fargate – The hassle-free option

Fargate, on the other hand, is like having a magical LEGO table that appears whenever you need it. You don’t have to worry about building or maintaining the table; you just focus on playing with your LEGOs. Fargate is a serverless compute engine for containers, meaning you don’t have to manage any servers. It’s a great option if you want to simplify your operations and reduce your overhead.

Making the right choice

So, which option is right for you? Well, it depends on your specific needs and preferences. If you need full control over your infrastructure and want to optimize costs by managing your own servers, EC2 might be a good choice. But if you prefer a serverless approach and want to avoid the hassle of managing servers, Fargate is the way to go.

AWS Container Services Compared

To make things easier, here’s a quick comparison of ECS, EKS, and Fargate:

Service	Description	Use Case
ECS	Managed container orchestration for EC2 instances	Great for full control over infrastructure
EKS	Managed Kubernetes service	Ideal for teams with Kubernetes expertise
Fargate	Serverless compute engine for ECS or EKS	Simplifies operations, no infrastructure management

Best practices and security for building a secure and reliable playground

Just like any playground, your container environment needs to be safe and secure. AWS provides a range of tools and best practices to help you build a reliable and secure container playground.

Security best practices for keeping your LEGOs safe

AWS offers a variety of security features to help you protect your container environment. You can use IAM to control access to your resources, implement network security measures (like Security Groups and NACLs) to protect your containers from unauthorized access, and scan your container images for vulnerabilities with tools like Amazon Inspector.

High availability for ensuring your playground is always open

To ensure your applications are always available, you can use AWS’s high-availability features. This includes deploying your containers across multiple availability zones, configuring load balancing to distribute traffic across your containers, and implementing disaster recovery measures to protect your applications from unexpected events.

Monitoring and troubleshooting for keeping an eye on your playground

AWS provides comprehensive monitoring and troubleshooting tools to help you keep your container environment running smoothly. You can use CloudWatch to monitor your containers’ performance, set up detailed alarms to catch issues before they escalate, and use CloudWatch Logs to dive deep into the activity of your applications. Additionally, AWS X-Ray helps you trace requests as they travel through your application, giving you a granular view of where bottlenecks or failures may occur. These tools together allow for proactive monitoring, quick detection of anomalies, and effective root-cause analysis, ensuring that your container environment is always optimized and functioning properly.

DevOps integration for automating your LEGO creations

AWS container services integrate seamlessly with your DevOps workflows, allowing you to automate deployments, ensure consistent environments, and streamline the entire development lifecycle. By integrating services like CodeBuild, CodeDeploy, and CodePipeline, AWS enables you to create end-to-end CI/CD pipelines that automate testing, building, and releasing your containerized applications. This integration helps teams release features faster, reduce errors due to manual processes, and maintain a high level of consistency across different environments.

CI/CD pipeline integration for building and deploying automatically

You can use AWS CodePipeline to create a continuous integration and continuous delivery (CI/CD) pipeline that automatically builds, tests, and deploys your containerized applications. This allows you to release new features and updates quickly and efficiently. Imagine using CodePipeline as an automated assembly line for your LEGO creations.

Cost optimization for saving money on your LEGOs

AWS offers a variety of cost optimization tools to help you save money on your container deployments. You can use ECR lifecycle policies to manage your container images efficiently, choose the right instance types for your workloads, and leverage AWS’s pricing models to optimize your costs. Additionally, AWS provides Savings Plans and Spot Instances, which allow you to significantly reduce costs when running containerized workloads with flexible scheduling. Utilizing the AWS Compute Optimizer can also help identify opportunities to downsize or modify your infrastructure to be more cost-effective, ensuring you’re always operating in a lean and optimized manner.

Real-world implementation for bringing your LEGO creations to life

Deploying containerized applications in a production environment requires careful planning and execution. This involves assessing your infrastructure, understanding resource requirements, and preparing for potential scaling needs. AWS provides a range of tools and best practices, such as infrastructure templates, automated deployment scripts, and monitoring solutions, to help ensure that your applications are deployed successfully. Additionally, AWS recommends using blue-green deployments to minimize downtime and risk, as well as leveraging autoscaling to maintain performance under varying loads.

Production deployment checklist for your Pre-flight check

Before deploying your applications, it’s important to consider a few key factors, such as your application’s requirements, your infrastructure needs, and your security and compliance requirements. AWS provides a comprehensive checklist to help you ensure your applications are ready for production.

Common challenges and solutions for troubleshooting your LEGO creations

Deploying and managing containerized applications can present some challenges, such as dealing with scaling complexities, managing network configurations, or troubleshooting performance bottlenecks. However, AWS provides a wealth of resources and support to help you overcome these challenges. You can find solutions to common problems, troubleshooting tips, and best practices in the AWS documentation, community forums, and even through AWS Support Plans, which offer access to technical experts. Additionally, tools like AWS Trusted Advisor can help identify potential issues before they impact your applications, while AWS Well-Architected Framework guides optimizing your container deployments for reliability, performance, and cost-efficiency.

Choosing the right tools for the job

AWS offers a comprehensive suite of container services to help you build, deploy, and manage your applications. By understanding the different services and their capabilities, you can choose the right tools for your specific needs and build a secure, reliable, and cost-effective container environment.

The key is to choose the right tools for the job and follow best practices to ensure your applications are secure, reliable, and scalable.

November 19, 2024 by Fernando SRE Cloud stuff Kubernetes