All systems operational. Good luck signing in

The customer buys a waffle iron, pays for next-day delivery, and gets a receipt before the kettle boils. She closes the tab. As far as she is concerned, somebody in a warehouse is about to go looking for a waffle iron.

Maya, the engineer on call, has reason to doubt this. Her pager says the oldest order waiting to reach the warehouse is getting older. She opens the production console, taps her security key, and watches the sign-in page reappear. She taps the key again, with the slightly firmer touch people reserve for lift buttons that have already lit up.

Another order comes in. Checkout is green.

What the green light actually bought you

The checkout probe has done an honest morning’s work. It submitted a test order, got an order number back, and recorded a success. A background worker handles the handoff to the warehouse. Separating those jobs was a reasonable design decision. A slow warehouse connection needn’t make a customer stand around waiting to pay.

That separation also buys the team a little time. Orders can sit in the queue while somebody finds out why the worker has stopped taking them. The design quietly assumes that somebody can get in.

Follow Maya for a minute. The alert reaches her phone. The runbook opens on her laptop because yesterday’s browser session is still valid. The link to the worker logs asks her to sign in again. So does the console where she can request permission to restart it. Two tabs, two different company logos, one corporate sign-in service having a difficult morning.

Meanwhile, every successful checkout adds an order to the queue. The business can carry on taking money for rather longer than it can carry on keeping promises.

The spare key is in the locked office

There is an emergency account. Maya remembers it from the last access review. It has the right permissions, a named owner, and a reassuring entry in a spreadsheet. The runbook points her to the company password vault. The vault offers a familiar blue button marked “Continue with single sign-on.”

The arrangement would work perfectly well if Maya had lost access to her individual cloud account. Today the component that vouches for everyone is unavailable. Both accounts are fine. Neither is much use to her.

This is a good question to ask of any emergency route. Which failure can it survive? Trace one named failure all the way through. Where are the instructions? Who can approve access? How does that person sign in? A second administrator can be a useful safeguard. If the approval request is sitting in a chat room that neither administrator can open, you now have two qualified people available to admire the problem.

When sign-in recovers, Maya reads the worker log, finds the stalled warehouse connection, and restarts the worker. Orders begin to move. The restart takes seconds. Getting to the point where she can decide whether a restart is sensible takes eighteen minutes.

Hand the runbook to someone who didn’t write it

At the next scheduled drill, Maya hands the pager to a colleague. They leave production traffic alone and agree to treat the usual sign-in service as unavailable. Her colleague starts with a fresh browser profile and no active cloud credentials. Maya keeps her hands off the keyboard.

The assignment is small. Use the emergency route to find the affected queue and report when a worker last successfully handed an order to the warehouse. No restart required. It’s a read-only task with an answer the team can verify, and it takes the responder one useful step beyond a welcome screen. A role that can open the console and can’t read the logs is quite capable of passing a login test.

Start the clock when the alert arrives. Leave it running while the responder looks for the instructions, reaches the approver, and works out which production account contains the queue. Those minutes count on Sundays too. If the vault sends them back to the unavailable sign-in service, record where the exercise stopped. Borrowing Maya’s open browser tab would make for a shorter drill and a rather longer Sunday. When the route works, check that somebody else saw the emergency access event and that the temporary access expires as intended.

Pay attention to the moment the author of the runbook feels an overwhelming urge to point at the screen. That’s usually where the next edit belongs.

Eighteen minutes out of twenty

At this shop, orders can wait about twenty minutes before the delay starts to threaten the promised dispatch time. Eighteen minutes spent getting access leaves two to investigate, act, and see whether the queue is draining. That’s a fairly demanding performance target for whoever happens to be holding the pager. It also gives the team a reason to fix the vault dependency before spending another afternoon tuning an alert that arrived on time.

At your next architecture review, pick one action a responder would need to take while customers can still buy the waffle iron. Start with a fresh session. How long does the person on call need to take that first useful action, and when did you last time it?

Share