DistributedSystems

Kubernetes leases or the art of waiting for the bathroom

If you looked inside a running Kubernetes cluster with a microscope, you would not see a perfectly choreographed ballet of binary code. You would see a frantic, crowded open-plan office staffed by thousands of employees who have consumed dangerous amounts of espresso. You have schedulers, controllers, and kubelets all sprinting around, frantically trying to update databases and move containers without crashing into each other.

It is a miracle that the whole thing does not collapse into a pile of digital rubble within seconds. Most human organizations of this size descend into bureaucratic infighting before lunch. Yet, somehow, Kubernetes keeps this digital circus from turning into a riot.

You might assume that the mechanism preventing this chaos is a highly sophisticated, cryptographic algorithm forged in the fires of advanced mathematics. It is not. The thing that keeps your cluster from eating itself is the distributed systems equivalent of a sticky note on a door. It is called a Lease.

And without this primitive, slightly passive-aggressive little object, your entire cloud infrastructure would descend into anarchy faster than you can type kubectl delete namespace.

The sticky note of power

To understand why a Lease is necessary, we have to look at the psychology of a Kubernetes controller. These components are, by design, incredibly anxious. They want to ensure that the desired state of the world matches the actual state.

The problem arises when you want high availability. You cannot just have one controller running because if it dies, your cluster stops working. So you run three replicas. But now you have a new problem. If all three replicas try to update the same routing table or create the same pod at the exact same moment, you get a “split-brain” scenario. This is the technical term for a psychiatric emergency where the left hand deletes what the right hand just created.

Kubernetes solves this with the Lease object. Technically, it is an API resource in the coordination.k8s.io group. Spiritually, it is a “Do Not Disturb” sign hung on a doorknob.

If you look at the YAML definition of a Lease, it is almost insultingly simple. It does not ask for a security clearance or a biometric scan. It essentially asks three questions:

HolderIdentity: Who are you?
LeaseDurationSeconds: How long are you going to be in there?
RenewTime: When was the last time you shouted that you are still alive?

Here is what one looks like in the wild:

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: cluster-coordination-lock
  namespace: kube-system
spec:
  holderIdentity: "controller-pod-beta-09"
  leaseDurationSeconds: 15
  renewTime: "2023-10-27T10:04:05.000000Z"

In plain English, this document says: “Controller Beta-09 is holding the steering wheel. It has fifteen seconds to prove it has not died of a heart attack. If it stays silent for sixteen seconds, we are legally allowed to pry the wheel from its cold, dead fingers.”

An awkward social experiment

To really grasp the beauty of this system, we need to leave the server room and enter a shared apartment with a terrible design flaw. There is only one bathroom, the lock is broken, and there are five roommates who all drank too much water.

The bathroom is the “critical resource.” In a computerized world without Leases, everyone would just barge in whenever they felt the urge. This leads to what engineers call a “race condition” and what normal people call “an extremely embarrassing encounter.”

Since we cannot fix the lock, we install a whiteboard on the door. This is the Lease.

The rules of this apartment are strict but effective. When you walk up to the door, you write your name and the current time on the board. You have now acquired the lock. As long as your name is there and the timestamp is fresh, the other roommates will stand in the hallway, crossing their legs and waiting politely.

But here is where it gets stressful. You cannot just write your name and fall asleep in the tub. The system requires constant anxiety. Every few seconds, you have to crack the door open, reach out with a marker, and update the timestamp. This is the “heartbeat.” It tells the people waiting outside that you are still conscious and haven’t slipped in the shower.

If you faint, or if the WiFi cuts out and you cannot reach the whiteboard, you stop updating the time. The roommates outside watch the clock. Ten seconds pass. Fifteen seconds. At sixteen seconds, they do not knock to see if you are okay. They assume you are gone forever, wipe your name off the board, write their own, and barge in.

It is ruthless, but it ensures that the bathroom is never left empty just because the previous occupant vanished into the void.

The paranoia of leader election

The most critical use of this bathroom logic is something called Leader Election. This is the mechanism that keeps your kube-controller-manager and kube scheduler from turning into a bar fight.

You typically run multiple copies of these control plane components for redundancy. However, you absolutely cannot have five different schedulers trying to assign the same pod to five different nodes simultaneously. That would be like having five conductors trying to lead the same orchestra. You do not get music; you get noise and a lot of angry musicians.

So, the replicas hold an election. But it is not a democratic vote with speeches and ballots. It is a race to grab the marker.

The moment the controllers start up, they all rush toward the Lease object. The first one to write its name in the holderIdentity field becomes the Leader. The others, the candidates, do not go home. They stand in the corner, staring at the Lease, refreshing the page every two seconds, waiting for the Leader to fail.

There is something deeply human about this setup. The backup replicas are not “supporting” the leader. They are jealous understudies watching the lead actor, hoping he breaks a leg so they can take center stage.

If the Leader crashes or simply gets stuck in a network traffic jam, the renewTime stops updating. The lease expires. Immediately, the backups scramble to write their own name. The winner takes over the cluster duties instantly. It is seamless, automated, and driven entirely by the assumption that everyone else is unreliable.

Reducing the noise pollution

In the early days of Kubernetes, things were even messier. Nodes, the servers doing the actual work, had to prove they were alive by sending a massive status report to the API server every few seconds.

Imagine a receptionist who has to process a ten-page medical history form from every single employee every ten seconds, just to confirm they are at their desks. It was exhausting. The API server spent so much time reading these reports that it barely had time to do anything else.

Today, Kubernetes uses Leases for node heartbeats, too. Instead of the full medical report, the node just updates a Lease object. It is a quick, lightweight ping.

“I’m here.”

“Good.”

“Still here.”

“Great.”

This change reduced the computational cost of staying alive significantly. The API server no longer needs to know your blood pressure and cholesterol levels every ten seconds; it just needs to know you are breathing. It turns a bureaucratic nightmare into a simple check-in.

How to play with fire

The beauty of the Lease system is that it is just a standard Kubernetes object. You can see these invisible sticky notes right now. If you list the leases in the system namespace, you will see the invisible machinery that keeps the lights on:

kubectl get leases -n kube-system

You will see entries for the controller manager, the scheduler, and probably one for every node in your cluster. If you want to see who the current boss is, you can describe the lease:

kubectl describe lease kube-scheduler -n kube-system

You will see the holderIdentity. That is the name of the replica currently running the show.

Now, if you are feeling particularly chaotic, or if you just want to see the world burn, you can delete a Lease manually.

kubectl delete lease kube-scheduler -n kube-system

Please do not do this in production unless you enjoy panic attacks.

Deleting an active Lease is like ripping the “Occupied” sign off the bathroom door while someone is inside. You are effectively lying to the system. You are telling the backup controllers, “The leader is dead! Long live the new leader!”

The backups will rush in and elect a new leader. But the old leader, who was effectively just sitting there minding its own business, is still running. Suddenly, it realizes it has been fired without notice. Ideally, it steps down gracefully. But in the split second before it realizes what happened, you might have two controllers giving orders.

The system will heal itself, usually within seconds, but those few seconds are a period of profound confusion for everyone involved.

The survival of the loudest

Leases are the unsung heroes of the cloud native world. We like to talk about Service Meshes and eBPF and other shiny, complex technologies. But at the bottom of the stack, keeping the whole thing from exploding, is a mechanism as simple as a name on a whiteboard.

It works because it accepts a fundamental truth about distributed systems: nothing is reliable, everyone is going to crash eventually, and the only way to maintain order is to force components to shout “I am alive!” every few seconds.

Next time your cluster survives a node failure or a controller restart without you even noticing, spare a thought for the humble Lease. It is out there in the void, frantically renewing timestamps, protecting you from the chaos of a split-brain scenario. And that is frankly better than a lock on a bathroom door any day.

Building living systems with WebSockets

For the longest time, communication on the web has been a painfully formal affair. It’s like sending a letter. Your browser meticulously writes a request, sends it off via the postal service (HTTP), and then waits. And waits. Eventually, the server might write back with an answer. If you want to know if anything has changed five seconds later, you have to send another letter. It’s slow, it’s inefficient, and frankly, the postman is starting to give you funny looks.

This constant pestering, “Anything new? How about now? Now?”, is the digital equivalent of a child on a road trip asking, “Are we there yet?” It’s called polling, and it’s the clumsy foundation upon which much of the old web was built. For applications that need to feel alive, this just won’t do.

What if, instead of sending a flurry of letters, we could just open a phone line directly to the server? A dedicated, always-on connection where both sides can just shout information at each other the moment it happens. That, in a nutshell, is the beautiful, chaotic, and nonstop chatter of WebSockets. It’s the technology that finally gave our distributed systems a voice.

The secret handshake that starts the party

So how do you get access to this exclusive, real-time conversation? You can’t just barge in. You have to know the secret handshake.

The process starts innocently enough, with a standard HTTP request. It looks like any other request, but it carries a special, almost magical, header: Upgrade: websocket. This is the client subtly asking the server, “Hey, this letter-writing thing is a drag. Can we switch to a private line?”

If the server is cool, and equipped for a real conversation, it responds with a special status code, 101 Switching Protocols. This isn’t just an acknowledgment; it’s an agreement. The server is saying, “Heck yes. The formal dinner party is over. Welcome to the after-party.” At that moment, the clumsy, transactional nature of HTTP is shed like a heavy coat, and the connection transforms into a sleek, persistent, two-way WebSocket tunnel. The phone line is now open.

So what can we do with all this chatter?

Once you have this open line, the possibilities become far more interesting than just fetching web pages. You can build systems that breathe.

The art of financial eavesdropping

Think of a stock trading platform. With HTTP, you’d be that sweaty-palmed investor hitting refresh every two seconds, hoping to catch a price change. With WebSockets, the server just whispers the new prices in your ear the microsecond they change. It’s the difference between reading yesterday’s newspaper and having a live feed from the trading floor piped directly into your brain.

Keeping everyone on the same page literally

Remember the horror of emailing different versions of a document back and forth? “Report_Final_v2_Johns_Edits_Final_FINAL.docx”. Collaborative tools like Google Docs killed that nightmare, and WebSockets were the murder weapon. When you type, your keystrokes are streamed to everyone else in the document instantly. It’s a seamless, shared consciousness, not a series of disjointed monologues.

Where in the world is my taxi

Ride-sharing apps like Uber would be a farce without a live map. You don’t want a “snapshot” of where your driver was 30 seconds ago; you want to see that little car icon gliding smoothly toward you. WebSockets provide that constant stream of location data, turning a map from a static picture into a living, moving window.

When the conversation gets too loud

Of course, hosting a party where a million people are all talking at once isn’t exactly a walk in the park. This is where our brilliant WebSocket-powered dream can turn into a bit of a logistical headache.

A server that could happily handle thousands of brief HTTP requests might suddenly break into a cold sweat when asked to keep tens of thousands of WebSocket phone lines open simultaneously. Each connection consumes memory and resources. It’s like being a party host who promised to have a deep, meaningful conversation with every single guest, all at the same time. Eventually, you’re just going to collapse from exhaustion.

And what happens if the line goes dead? A phone can be hung up, but a digital connection can just… fade into the void. Is the client still there, listening quietly? Or did their Wi-Fi die mid-sentence? To avoid talking to a ghost, servers have to periodically poke the client with a ping message. If they get a pong back, all is well. If not, the server sadly hangs up, freeing the line for someone who actually wants to talk.

How to be a good conversation host

Taming this beast requires a bit of cleverness. You can’t just throw one server at the problem and hope for the best.

Load balancers become crucial, but they need to be smarter. A simple load balancer that just throws requests at any available server is a disaster for WebSockets. It’s like trying to continue a phone conversation while the operator keeps switching you to a different person who has no idea what you were talking about. You need “sticky sessions,” which is a fancy way of saying the load balancer is smart enough to remember which server you were talking to and keeps you connected to it.

Security also gets a fun new twist. An always-on connection is a wonderfully persistent doorway into your system. If you’re not careful about who you’re talking to and what they’re saying (WSS, the secure version, is non-negotiable), you might find you’ve invited a Trojan horse to your party.

A world that talks back

So, no, WebSockets aren’t just another tool in the shed. They represent a philosophical shift. It’s the moment we stopped treating the web like a library of dusty, static books and started treating it like a bustling, chaotic city square. We traded the polite, predictable, and frankly boring exchange of letters for the glorious, unpredictable mess of a real-time human conversation.

It means our applications can now have a pulse. They can be surprised, they can interrupt, and they can react with the immediacy of a startled cat. Building these living systems is certainly noisier and requires a different kind of host, one who’s part traffic cop and part group therapist. But by embracing the chaos, we create experiences that don’t just respond; they engage, they live. And isn’t building a world that actually talks back infinitely more fun?

October 1, 2025 by Fernando SRE Cloud stuff Computer Science stuff

Edge computing reshapes DevOps for the real-time era

A new frontier at your doorstep

When Amazon started placing delivery lockers in neighborhoods, packages arrived faster and more reliably. Edge computing follows a similar logic, bringing computational power closer to the user. Instead of sending data halfway around the world, edge computing processes it locally, dramatically reducing latency, enhancing privacy, and maintaining autonomy.

For DevOps teams, this shift isn’t trivial. Like switching from central mail hubs to neighborhood lockers, it demands new strategies and skills.

CI/CD faces a new reality

Classic cloud pipelines are centralized, much like a single distribution center. Edge computing flips that model upside-down, scattering deployments across numerous tiny locations. Deploying updates to thousands of edge devices isn’t the same as updating a handful of cloud servers.

DevOps teams now battle version drift, a scenario similar to managing software on thousands of smartphones with different versions. The solutions? Smaller, incremental updates and lightweight build artifacts, ensuring that pushing changes doesn’t overwhelm limited network bandwidth or hardware resources.

Designing for when things go dark

Planning a family dinner knowing there’s a possibility of a power outage means stocking up on candles and sandwiches. Similarly, edge devices must be designed for disconnection, ensuring operations continue uninterrupted during network downtime.

Offline-first architectures become critical here. Techniques like local queuing and eventual data reconciliation help edge applications function seamlessly, even if connectivity is lost for hours or days. Managing schema migrations carefully is crucial; it’s akin to updating recipes without knowing if family members received the memo.

Keeping data consistently in sync

Imagine organizing a city-wide neighborhood watch: push notifications ensure quick alerts, while pull mechanisms periodically fetch updates. Edge deployments use similar synchronization tactics.

Techniques such as Conflict-Free Replicated Data Types (CRDTs) help manage data consistency, even when devices are offline or slow to respond. DevOps engineers also need to factor in bandwidth budgeting, using intelligent compression and prioritizing data to ensure crucial information reaches its destination promptly.

Observability without seeing everything

Monitoring edge deployments is like managing a fleet of food trucks spread across the city. You can’t constantly keep an eye on every truck. Instead, you rely on periodic check-ins and key signals.

Telemetry sampling, data aggregation at the edge, and effective back-pressure management prevent network floods. Selecting a few meaningful metrics, like checking a truck’s gas gauge rather than tracking every sandwich sold, helps quickly pinpoint issues without drowning in data.

Incident response across the edge

Responding to issues at thousands of remote locations is challenging, like troubleshooting vending machines scattered nationwide without direct access.

Edge incident response leverages runbook templates, policy-as-code, and remote diagnostics tools. Because traditional SSH access isn’t always viable, tactics like automated self-healing and structured escalation paths blending central SRE teams with local staff become indispensable.

Bridging cloud and edge

Integrating IoT devices into your infrastructure is similar to securely registering visitors at a large event, you need clear identification, managed credentials, and accurate headcounts.

Edge computing uses secure onboarding, rotating credentials, and message brokers that maintain state coherence across the network. Digital twins represent device states virtually, helping maintain consistent and accurate information between edge and cloud environments. Cost-effective strategies determine whether workloads run locally or in centralized clouds.

Preparing for what’s next

Edge computing evolves rapidly, with emerging standards like WebAssembly (WASM) running applications directly at the edge, and maturing tools like OpenTelemetry simplifying observability.

DevOps teams should embrace these changes early. Developing skills in hardware awareness and basic radio frequency (RF) knowledge becomes increasingly valuable. Experimenting now, rigorously measuring results, and sharing insights ensures teams stay ahead.

Innovate and adapt for the road ahead

Edge computing is reshaping DevOps in real-time. Thriving in this era requires adapting practices, tooling, and mindset. Bring your computational lockers closer to home, plan proactively for network disruptions, streamline synchronization, enhance remote observability, and respond intelligently to incidents.

By preparing today, your DevOps team can confidently navigate tomorrow’s distributed landscape. Embracing edge computing means more than just keeping pace with technology; it positions your team to deliver faster, more reliable services, capitalize on emerging business opportunities, and maintain a competitive advantage. Investing now in the right tools, processes, and skills not only safeguards against future challenges but also unlocks potential for innovation, growth, and sustained success in a rapidly evolving technological world.

In short, the future belongs to those who embrace change and adapt quickly; let your team be among them.

July 3, 2025 by Fernando SRE Cloud stuff DevOps stuff SRE stuff

What are cloud operating systems?

You know your computer, right? That trusty machine, maybe running Windows, macOS, or perhaps a flavor of Linux like my buddy Fernando rocks with his Ubuntu setup. It has an Operating System. Its job? To manage the guts of that one machine, the processor, the memory, the storage, making sure your apps can run, your files are saved. It’s the conductor of a small, personal orchestra.

Now… zoom out. Way out.

Imagine not one computer but thousands. Tens of thousands. Maybe millions. Housed in colossal buildings we call data centers, spread across the globe, all interconnected. A sprawling, humming galaxy of computation.

How do you manage that? You can’t just install Windows on the entire internet! That’s like trying to run a city using the rules of a single household. It just doesn’t scale.

Meet the Cloud Operating System.

Now, hold on, don’t picture a single piece of software called “CloudOS” that you download. It’s more fundamental, more… cosmic in its scope. Think of it less as the OS on a single server in the cloud (that’s often still Linux or Windows), and more like the overarching intelligence, the distributed brain managing the entire fleet, the whole data center, maybe even multiple data centers as one cohesive entity.

What does this cosmic brain do? It performs a symphony of coordination on a scale that would make your desktop OS blush:

It Abstracts the Hardware: It takes all those individual servers, storage racks, networking gear, the raw physical stuff, and throws a kind of “invisibility cloak” over it. It presents it all as a unified, seemingly infinite pool of resources. You ask for processing power, memory, storage, and the Cloud OS figures out where in that vast physical infrastructure to get it from, without you needing to know or care about the specific box. It’s like asking for “water” and the system handles whether it comes from this reservoir or that aquifer.
It Orchestrates Resources: Need to spin up a thousand virtual servers for a massive calculation? Boom. The Cloud OS handles the provisioning, allocation, and networking. Need to automatically scale your website’s capacity because you just went viral? The Cloud OS is the maestro making that happen seamlessly. It’s the ultimate traffic controller, resource allocator, and taskmaster for the entire digital city.
It Manages Virtualization: This is key. Cloud OSes are masters of virtualization, carving up physical machines into multiple virtual ones (VMs) or pooling resources to make many machines act as one giant one. It’s about turning rigid hardware into a flexible, fluid resource.
It Provides Essential Services: Think scheduling (what runs where and when), storage management (replicating data for safety, moving it for speed), network management (directing traffic flow), fault tolerance (if one server fails, the system barely notices), and massive automation (because no army of humans could manage this manually).

So, can you point to one specific “Cloud Operating System”? Well, it’s complicated. The giants, Amazon AWS, Microsoft Azure, and Google Cloud Platform, have built their own incredibly sophisticated, largely proprietary systems that act as the planet-scale operating systems for their clouds. Projects like OpenStack aim to provide an open-source framework to build this kind of cloud management system. And technologies like Kubernetes, while often called a “container orchestrator,” are essentially performing many of the distributed operating system functions at the application layer within the cloud.

Why is this disruptive? Because it fundamentally broke the old model of computing. We went from being limited by the box on our desk to tapping into near-limitless resources on demand. The Cloud OS is the unsung hero behind this revolution, the invisible intelligence weaving together the fabric of the modern digital world. It’s not just managing silicon and wires; it’s managing possibility on an unprecedented scale.

Think about that the next time you access a file from anywhere or watch a video streamed from the ether. You’re witnessing the silent, elegant dance orchestrated by a Cloud Operating System.

Hope that expands your view of the computational cosmos! Keep looking up… and into the cloud.

March 30, 2025 by Fernando SRE Cloud stuff Computer Science stuff

Observability of Distributed Applications, Beyond the Logs

A Journey into Modern Monitoring

In the world of software, we’ve witnessed a fascinating evolution. Applications have transformed from monolithic giants into nimble constellations of microservices. This shift, while empowering, has brought forth a new challenge: the overwhelming deluge of data generated by these distributed systems. Traditional logging, once our trusty guide, now feels like trying to assemble a puzzle with pieces scattered across a vast landscape.

The Puzzle of Modern Applications

Imagine a bustling city. Each microservice is like a building, each with its own story. Logs are akin to the whispers within those walls, offering glimpses into individual activities. But what if we want to understand the city as a whole? How do we grasp the flow of traffic, the interconnectedness of services, and the subtle signs of trouble brewing beneath the surface?

This is where the concept of “observability” shines. It’s more than just collecting logs; it’s about understanding our complex systems holistically. It’s about peering beyond the individual whispers and seeing the symphony of interactions.

Beyond Logs: Metrics and Traces

To truly embrace observability, we must expand our toolkit. Alongside logs, we need two more powerful allies:

Metrics: These are the vital signs of our applications, the pulse rate, blood pressure, and temperature. Metrics provide quantitative data like CPU usage, request latency, and error rates. They give us a real-time snapshot of system health, allowing us to detect anomalies and trends. As the saying goes, “Metrics tell us when something went wrong.“
Traces: Think of these as the GPS trackers of our requests. As a request journeys through our microservices, traces capture its path, the time spent at each stop, and any bottlenecks encountered. This helps us pinpoint the root cause of issues and optimize performance. In essence, “Traces tell us where something went wrong.“

The Power of Correlation

But the true magic of observability lies in the correlation of these three pillars. We gain a multi-dimensional view of our systems by weaving together logs, metrics, and traces. When an alert is triggered based on unusual metrics, we can investigate the corresponding traces to see exactly which requests were affected. From there, we can examine the logs of the relevant microservices to understand precisely what went wrong.

This correlation is the key to rapid troubleshooting and proactive problem-solving. It empowers us to move beyond reactive firefighting and into a realm of continuous improvement.

The Observability Toolbox. Prometheus, Grafana, Jaeger and Loki

Now, let’s equip ourselves with the tools of the trade:

Prometheus: This is our trusty data collector, like a diligent census taker. It goes from microservice to microservice, gathering up those vital signs – the metrics – and storing them neatly. But it’s more than just a collector; it’s a clever analyst too. It gives us a special language to ask questions about our data and to see patterns and trends emerging from the numbers.
Grafana: Imagine a grand control room, with screens glowing with information. That’s Grafana. It takes the raw data, those metrics, and logs, and turns them into beautiful pictures, like a painter turning a blank canvas into a masterpiece. We can see the rise and fall of CPU usage, and the dance of network traffic, all laid out before our eyes.
Jaeger: This is our detective’s toolkit, the magnifying glass and fingerprint powder. It follows the trails of requests as they wander through our city of microservices. It shows us where they get stuck, and where they take unexpected turns. By working together with our log collector, it helps us match up those trails with the clues hidden in the logs.
Loki: If logs are the whispers of our city, Loki is our trusty stenographer. It captures and stores those whispers, those tiny details that might seem insignificant on their own. But when we correlate them with our metrics and traces, they reveal the secrets of how our city truly functions. Loki is like a time machine for our logs, letting us rewind and replay events to understand what went wrong.

With these four tools in our hands, we become not just architects of our systems, but explorers and detectives. We can see the hidden connections, diagnose the ailments, and ultimately, make our city of microservices run smoother, faster, and more reliably.

The Power of Observability

By adopting observability, we unlock a new level of understanding. We can:

Diagnose issues faster: Instead of sifting through endless logs, we can quickly identify the root cause of problems using metrics and traces.
Optimize performance: By analyzing the flow of requests, we can pinpoint bottlenecks and fine-tune our systems for optimal efficiency.
Proactive monitoring: With real-time alerts based on metrics, we can detect anomalies before they escalate into major incidents.
Data-driven decisions: Observability data provides invaluable insights for capacity planning, resource allocation, and architectural improvements.

The Journey Continues

The world of distributed applications is ever-evolving. New technologies and challenges will emerge. But armed with the principles of observability and the right tools, we can navigate this landscape with confidence. We can build systems that are not only resilient and scalable but also deeply understood.

Observability is not a destination; it’s a journey of continuous discovery. By adopting it, we embark on a path of greater insight, better performance, and ultimately, more reliable and user-friendly applications.

August 5, 2024 by Fernando SRE DevOps stuff SRE stuff