DevOps

Sometimes, you’re working with Kubernetes, orchestrating your containers like a maestro, and suddenly, one of your Pods throws a tantrum. It enters the dreaded CrashLoopBackOff state. You check the logs, hoping for a clue, a breadcrumb trail leading to the culprit, but… nothing. Silence. It feels like the Pod is crashing so fast it doesn’t even have time to whisper why. Frustrating, right? Many of us in the DevOps, SRE, and development world have been there. It’s like trying to solve a mystery where the main witness disappears before saying a word.

But don’t despair! This CrashLoopBackOff status isn’t just Kubernetes being difficult. It’s a signal. It tells us Kubernetes is trying to run your container, but the container keeps stopping almost immediately after starting. Kubernetes, being persistent, waits a bit (that’s the “BackOff” part) and tries again, entering a loop of crash-wait-restart-crash. Our job is to break this loop by figuring out why the container won’t stay running. Let’s put on our detective hats and explore the common reasons and how to investigate them.

Starting the investigation. What Kubernetes tells us

Before diving deep, let’s ask Kubernetes itself what it knows. The describe command is often our first and most valuable tool. It gives us a broader picture than just the logs.

kubectl describe pod <your-pod-name> -n <your-namespace>

Don’t just glance at the output. Look closely at these sections:

• State: It will likely show Waiting with the reason CrashLoopBackOff. But look at the Last State. What was the state before it crashed? Did it have an Exit Code? This code is a crucial clue! We’ll talk more about specific codes soon.
• Restart Count: A high number confirms the container is stuck in the crash loop.
• Events: This section is pure gold. Scroll down and read the events chronologically. Kubernetes logs significant happenings here. You might see errors pulling the image (ErrImagePull, ImagePullBackOff), problems mounting volumes, failures in scheduling, or maybe even messages about health checks failing. Sometimes, the reason is right there in the events!

Chasing ghosts. Checking previous logs

Okay, so the current logs are empty. But what about the logs from the previous attempt just before it crashed? If the container managed to run for even a fraction of a second and log something, we might catch it using the –previous flag.

kubectl logs <your-pod-name> -n <your-namespace> --previous

It’s a long shot sometimes, especially if the crash is instantaneous, but it costs nothing to try and can occasionally yield the exact error message you need.

Are the health checks too healthy?

Liveness and Readiness probes are fantastic tools. They help Kubernetes know if your application is truly ready to serve traffic or if it’s become unresponsive and needs a restart. But what if the probes themselves are the problem?

• Too Aggressive: Maybe the initialDelaySeconds is too short, and the probe checks before your app is even initialized, causing Kubernetes to kill it prematurely.
• Wrong Endpoint or Port: A simple typo in the path or port means the probe will always fail.
• Resource Starvation: If the probe endpoint requires significant resources to respond, and the container is resource-constrained, the probe might time out.

Check your Deployment or Pod definition YAML for livenessProbe and readinessProbe sections.

# Example Probe Definition
livenessProbe:
  httpGet:
    path: /heaalth # Is this path correct?
    port: 8780     # Is this the right port?
  initialDelaySeconds: 15 # Is this long enough for startup?
  periodSeconds: 10
  timeoutSeconds: 3     # Is the app responding within 3 seconds?
  failureThreshold: 3

If you suspect the probes, a good debugging step is to temporarily remove or comment them out.

• Edit the deployment:

kubectl edit deployment <your-deployment-name> -n <your-namespace>

• Find the livenessProbe and readinessProbe sections within the container spec and comment them out (add # at the beginning of each line) or delete them.
• Save and close the editor. Kubernetes will trigger a rolling update.

Observe the new Pods. If they run without crashing now, you’ve found your culprit! Now you need to fix the probe configuration (adjust delays, timeouts, paths, ports) or figure out why your application isn’t responding correctly to the probes and then re-enable them. Don’t leave probes disabled in production!

Decoding the Exit codes reveals the container’s last words

Remember the exit code we saw in kubectl? Can you describe the pod under Last State? These numbers aren’t random; they often tell a story. Here are some common ones:

• Exit Code 0: Everything finished successfully. You usually won’t see this with CrashLoopBackOff, as that implies failure. If you do, it might mean your container’s main process finished its job and exited, but Kubernetes expected it to keep running (like a web server). Maybe you need a different kind of workload (like a Job) or need to adjust your container’s command to keep it running.
• Exit Code 1: A generic, unspecified application error. This usually means the application itself caught an error and decided to terminate. You’ll need to look inside the application’s code or logic.
• Exit Code 137 (128 + 9): This often means the container was killed by the system due to using too much memory (OOMKilled – Out Of Memory). The operating system sends a SIGKILL signal (which is signal number 9).
• Exit Code 139 (128 + 11): Segmentation Fault. The container tried to access memory it shouldn’t have. This is usually a bug within the application itself or its dependencies.
• Exit Code 143 (128 + 15): The container received a SIGTERM signal (signal 15) and terminated gracefully. This might happen during a normal shutdown process initiated by Kubernetes, but if it leads to CrashLoopBackOff, perhaps the application isn’t handling SIGTERM correctly or something external is repeatedly telling it to stop.
• Exit Code 255: An exit status outside the standard 0-254 range, often indicating an application error occurred before it could even set a specific exit code.

Exit Code 137 is particularly common in CrashLoopBackOff scenarios. Let’s look closer at that.

Running out of breath resource limits

Modern applications can be memory-hungry. Kubernetes allows you to set resource requests (what the Pod wants) and limits (the absolute maximum it can use). If your container tries to exceed its memory limit, the Linux kernel’s OOM Killer steps in and terminates the process, resulting in that Exit Code 137.

Check the resources section in your Pod/Deployment definition:

# Example Resource Definition
resources:
  requests:
    memory: "128Mi" # How much memory it asks for initially
    cpu: "250m"     # How much CPU it asks for initially (m = millicores)
  limits:
    memory: "256Mi" # The maximum memory it's allowed to use
    cpu: "500m"     # The maximum CPU it's allowed to use

If you suspect an OOM kill (Exit Code 137 or events mentioning OOMKilled):

1. Check Limits: Are the limits set too low for what the application actually needs?
2. Increase Limits: Try carefully increasing the memory limit. Edit the deployment (kubectl edit deployment…) and raise the limits. Observe if the crashes stop. Be mindful not to set limits too high across many pods, as this can exhaust node resources.
3. Profile Application: The long-term solution might be to profile your application to understand its memory usage and optimize it or fix memory leaks.

Insufficient CPU limits can also cause problems (like extreme slowness leading to probe timeouts), but memory limits are a more frequent direct cause of crashes via OOMKilled.

Is the recipe wrong? Image and configuration issues

Sometimes, the problem happens before the application code even starts running.

• Bad Image: Is the container image name and tag correct? Does the image exist in the registry? Is it built for the correct architecture (e.g., trying to run an amd64 image on an arm64 node)? Check the Events in kubectl describe pod for image-related errors (ErrImagePull, ImagePullBackOff). Try pulling and running the image locally to verify:

docker pull <your-image-name>:<tag>
docker run --rm <your-image-name>:<tag>

• Configuration Errors: Modern apps rely heavily on configuration passed via environment variables or mounted files (ConfigMaps, Secrets).

.- Is a critical environment variable missing or incorrect?

.- Is the application trying to read a file from a ConfigMap or Secret volume that doesn’t exist or hasn’t been mounted correctly?

.- Are file permissions preventing the container user from reading necessary config files?

Check your deployment YAML for env, envFrom, volumeMounts, and volumes sections. Ensure referenced ConfigMaps and Secrets exist in the correct namespace (kubectl get configmap <map-name> -n <namespace>, kubectl get secret <secret-name> -n <namespace>).

Keeping the container alive for questioning

What if the container crashes so fast that none of the above helps? We need a way to keep it alive long enough to poke around inside. We can tell Kubernetes to run a different command when the container starts, overriding its default entrypoint/command with something that doesn’t exit, like sleep.

• Edit your deployment:

kubectl edit deployment <your-deployment-name> -n <your-namespace>

• Find the containers section and add a command and args field to override the container’s default startup process:

# Inside the containers: array
- name: <your-container-name>
  image: <your-image-name>:<tag>
  # Add these lines:
  command: [ "sleep" ]
  args: [ "infinity" ] # Or "3600" for an hour, etc.
  # ... rest of your container spec (ports, env, resources, volumeMounts)

(Note: Some base images might not have sleep infinity; you might need sleep 3600 or similar)

• Save the changes. A new Pod should start. Since it’s just sleeping, it shouldn’t crash.

Now that the container is running (even if it’s doing nothing useful), you can use kubectl exec to get a shell inside it:

kubectl exec -it <your-new-pod-name> -n <your-namespace> -- /bin/sh
# Or maybe /bin/bash if sh isn't available

Once inside:

• Check Environment: Run env to see all environment variables. Are they correct?
• Check Files: Navigate (cd, ls) to where config files should be mounted. Are they there? Can you read them (cat <filename>)? Check permissions (ls -l).
• Manual Startup: Try to run the application’s original startup command manually from the shell. Observe the output directly. Does it print an error message now? This is often the most direct way to find the root cause.

Remember to remove the command and args override from your deployment once you’ve finished debugging!

The power of kubectl debug

There’s an even more modern way to achieve something similar without modifying the deployment directly: kubectl debug. This command can create a copy of your crashing pod or attach a new “ephemeral” container to the running (or even failed) pod’s node, sharing its process namespace.

A common use case is to create a copy of the pod but override its command, similar to the sleep trick:

kubectl debug pod/<your-pod-name> -n <your-namespace> --copy-to=debug-pod --set-image='*' --share-processes -- /bin/sh
# This creates a new pod named 'debug-pod', using the same spec but running sh instead of the original command

Or you can attach a debugging container (like busybox, which has lots of utilities) to the node where your pod is running, allowing you to inspect the environment from the outside:

kubectl debug node/<node-name-where-pod-runs> -it --image=busybox
# Once attached to the node, you might need tools like 'crictl' to inspect containers directly

kubectl debug is powerful and flexible, definitely worth exploring in the Kubernetes documentation.

Don’t forget the basics node and cluster health

While less common, sometimes the issue isn’t the Pod itself but the underlying infrastructure.

• Node Health: Is the node where the Pod is scheduled healthy?
  kubectl get nodes

# Check the STATUS. Is it 'Ready'?
kubectl describe node <node-name>
# Look for Conditions (like MemoryPressure, DiskPressure) and Events at the node level.

• Cluster Events: Are there broader cluster issues happening?
  kubectl get events -n <your-namespace>

kubectl get events --all-namespaces # Check everywhere

Wrapping up the investigation

Dealing with CrashLoopBackOff without logs can feel like navigating in the dark, but it’s usually solvable with a systematic approach. Start with kubectl describe, check previous logs, scrutinize your probes and configuration, understand the exit codes (especially OOM kills), and don’t hesitate to use techniques like overriding the entrypoint or kubectl debug to get inside the container for a closer look.

Most often, the culprit is a configuration error, a resource limit that’s too tight, a faulty health check, or simply an application bug that manifests immediately on startup. By patiently working through these possibilities, you can unravel the mystery and get your Pods back to a healthy, running state.