Kubernetes Troubleshooting

Kubernetes CrashLoopBackOff Troubleshooting

A production-focused workflow for diagnosing a Kubernetes container that repeatedly starts, crashes, and gets restarted. Use pod state, exit codes, logs, events, probes, configuration, and resource evidence to identify the actual root cause.

What is CrashLoopBackOff?

CrashLoopBackOff is a Kubernetes status that indicates a container has repeatedly terminated and Kubernetes is backing off before attempting another restart.

The key troubleshooting principle is: CrashLoopBackOff is a symptom, not the root cause. Your job is to find out why the container is terminating.

Think like an SRE

Do not immediately delete or restart the pod. First capture evidence, determine customer impact, identify the failing layer, apply the smallest safe change, and then verify that the service remains healthy.

Production Troubleshooting Flow

Follow this order during an incident. It keeps the investigation evidence-driven instead of changing multiple things at once.

Confirm the symptom

kubectl get pods -n <namespace>

Is the pod repeatedly restarting, and is the restart count increasing?

Capture Kubernetes evidence

kubectl describe pod <pod-name> -n <namespace>

What are State, Last State, Reason, Exit Code, probes, and Events telling you?

Capture application evidence

kubectl logs <pod-name> -n <namespace> --previous

What happened immediately before the previous container exited?

Classify the failure

kubectl get events -n <namespace> --sort-by=.lastTimestamp

Is this an application, configuration, probe, resource, mount, or platform issue?

Apply the smallest safe fix

kubectl rollout status deployment/<deployment-name> -n <namespace>

Can you correct the underlying cause without making unrelated production changes?

Verify recovery

kubectl get pods -n <namespace>

Is the pod Ready, stable, and no longer accumulating restarts?

Step 1: Confirm the Pod Status

Start by identifying the affected pod and checking whether the restart count is increasing.

kubectl get pods -n <namespace>

NAME                         READY   STATUS             RESTARTS
my-app-7d8f9c6b7d-abc12     0/1     CrashLoopBackOff   5

A pod in CrashLoopBackOff is not necessarily telling you why the process failed. Treat STATUS as the starting signal and continue to the container state, logs, and events.

Step 2: Inspect State, Last State, Exit Code, and Events

kubectl describe pod <pod-name> -n <namespace>

State

Shows the current container state. It helps establish what is happening right now.

Last State

Often more useful for CrashLoopBackOff because it describes the previous terminated container instance.

Reason

Look for evidence such as OOMKilled or another termination reason. Do not infer the cause from the CrashLoopBackOff label alone.

Exit Code

Provides a clue about how the process terminated. Always correlate the exit code with logs and the termination reason.

Step 3: Check Current and Previous Logs

Application logs are often the fastest route to the actual failure.

kubectl logs <pod-name> -n <namespace>

Most important command for a crash loop

kubectl logs <pod-name> -n <namespace> --previous

If the container has already restarted, the previous instance may contain the final application error that caused the termination.

What to search for in logs

• Configuration or environment-variable errors
• Connection or dependency failures
• Authentication or certificate errors
• Missing files or directories
• Application startup exceptions
• Explicit non-zero process termination

Step 4: Check Kubernetes Events

Events can expose Kubernetes-level problems that do not appear in application logs.

kubectl get events -n <namespace> --sort-by=.lastTimestamp

Also check the Events section returned by:

kubectl describe pod <pod-name> -n <namespace>

Common CrashLoopBackOff Causes

Application crash

The process starts and exits because of an application exception, invalid startup configuration, missing dependency, or an intentional non-zero exit.

Evidence

kubectl logs or kubectl logs --previous usually contains the application error.

Bad configuration

Environment variables, command-line arguments, ConfigMaps, Secrets, or mounted configuration files contain invalid or missing values.

Evidence

Logs show configuration errors, or describe reveals incorrect environment/configuration references.

Liveness or startup probe failure

Kubernetes can restart a container when its liveness probe repeatedly fails. A startup probe can also prevent normal probe evaluation until startup succeeds.

Evidence

describe pod and Events commonly show repeated probe failures.

OOMKilled / memory pressure

The container exceeded its memory limit and was terminated. This is different from simply seeing a generic CrashLoopBackOff status.

Evidence

describe pod shows a previous termination reason such as OOMKilled.

Incorrect command or arguments

The image starts with a command or argument that does not exist, is invalid, or causes the process to terminate immediately.

Evidence

Container logs, pod configuration, and the image's expected startup command are the first places to investigate.

Missing files, mounts, or permissions

The application expects a file, volume, Secret, ConfigMap, certificate, or permission that is unavailable inside the container.

Evidence

Application logs, describe output, and mount-related events can reveal the missing dependency.

Understanding Exit Codes

Exit codes are clues, not standalone root-cause answers. Always correlate them with Last State, Reason, logs, and events.

Exit 0

The process exited successfully. If the workload is expected to stay running, investigate why the application exits instead of remaining a long-running process.

Exit 1

A generic application error. Treat the application logs as the primary evidence rather than assuming Kubernetes itself caused the failure.

Exit 137

The process was terminated with SIGKILL (128 + 9). In Kubernetes, check Last State and the termination reason for evidence of OOMKilled before concluding that memory pressure was the cause.

Exit 143

The process was terminated with SIGTERM (128 + 15). Investigate why termination was requested, including workload lifecycle events and graceful shutdown behavior.

Probe Failures: Startup vs Liveness

A common production mistake is changing probe settings without first checking whether the application actually starts correctly.

Startup probe

Gives a slow-starting application time to initialize before normal liveness/readiness behavior becomes relevant.

Investigate repeated startup failures and whether the configured startup window matches the application's actual startup time.

Liveness probe

Helps Kubernetes determine whether a running container should be restarted.

If liveness failures appear in Events, verify the endpoint, port, timing, and application's real health behavior before changing thresholds.

kubectl describe pod <pod-name> -n <namespace>

OOMKilled: Do Not Confuse It With CrashLoopBackOff

OOMKilled describes a memory-related termination. CrashLoopBackOff describes the resulting repeated restart condition. A pod can show CrashLoopBackOff after a container is repeatedly terminated for memory reasons.

kubectl describe pod <pod-name> -n <namespace>

Check the container's Last State and Reason. If the evidence shows OOMKilled, investigate memory limits, actual workload usage, recent application changes, and whether the process has a memory-growth problem before simply increasing the limit.

Configuration, ConfigMap, and Secret Checks

If logs point to missing or invalid configuration, inspect the workload configuration instead of repeatedly restarting the pod.

kubectl get deployment <deployment-name> -n <namespace> -o yaml

kubectl get configmap <configmap-name> -n <namespace> -o yaml

kubectl get secret <secret-name> -n <namespace> -o yaml

Avoid exposing Secret values in tickets, screenshots, terminal captures, or incident documentation.

Common Fixes — Based on Evidence

✓Correct the application configuration or startup error identified in the logs.
✓Fix invalid environment variables, ConfigMap values, or Secret references.
✓Correct the container command or arguments if the process is exiting immediately.
✓Adjust incorrect startup or liveness probes after validating the application's actual startup behavior.
✓Investigate OOMKilled by reviewing memory usage, limits, application behavior, and recent workload changes.
✓Restore missing volumes, files, certificates, or permissions required by the application.
✓Roll back a bad workload revision when the evidence points to a recent deployment change.

Production warning

Avoid using pod deletion or arbitrary restarts as the first troubleshooting step. Restarting can destroy useful evidence and may temporarily hide the underlying problem.

Verify the Fix

Recovery is not complete just because the pod briefly becomes Running. Verify readiness, restart stability, workload rollout, and recent events.

kubectl get pods -n <namespace>

kubectl describe pod <pod-name> -n <namespace>

kubectl logs <pod-name> -n <namespace> --tail=100

kubectl get events -n <namespace> --sort-by=.lastTimestamp

kubectl rollout status deployment/<deployment-name> -n <namespace>

Recovery criteria

✓ Pod reaches Ready state.
✓ Restart count stops increasing.
✓ Application logs no longer show the failure.
✓ Recent events do not show the same warning repeatedly.
✓ Deployment rollout completes successfully.

SRE Incident Checklist

□Record the affected namespace, workload, pod name, start time, and customer impact before making changes.
□Capture current and previous container logs before restarting or deleting the pod.
□Inspect Last State, Reason, Exit Code, and Events rather than treating CrashLoopBackOff as the root cause.
□Check whether one pod is failing or whether multiple replicas are showing the same behavior.
□Validate ConfigMaps, Secrets, mounted files, commands, arguments, probes, and resource limits.
□After the fix, verify readiness and stability rather than stopping when the pod changes from CrashLoopBackOff to Running.

Quick CrashLoopBackOff Checklist

1. Confirm

kubectl get pods

2. Describe

State + Last State + Reason + Exit Code + Events

3. Logs

kubectl logs --previous

4. Classify

App / Config / Probe / Resource / Mount / Platform

5. Fix

Change the underlying cause, not just the symptom

6. Verify

Ready + stable restarts + healthy rollout

Frequently Asked Questions

Is CrashLoopBackOff the root cause?

Usually no. CrashLoopBackOff describes a repeated restart condition and back-off behavior. The underlying cause may be an application error, configuration problem, failed probe, resource termination, or another container startup issue.

What command should I run first?

Start with kubectl get pods -n <namespace> to confirm the affected pod and restart count. Then use kubectl describe pod and kubectl logs --previous to collect evidence.

Why is kubectl logs --previous important?

A container in CrashLoopBackOff may terminate and restart quickly. --previous lets you inspect logs from the previous container instance, which can contain the failure that caused the restart.

Does CrashLoopBackOff always mean OOMKilled?

No. OOMKilled is only one possible cause. Check Last State and Reason in kubectl describe pod and correlate them with logs and events.