DevOps • SRE • Production Support

DevOps Troubleshooting Guide

Troubleshoot common DevOps and production problems using practical commands, evidence-first investigation and structured incident workflows across Linux, Kubernetes, Docker, Jenkins, Terraform, networking and monitoring.

KubernetesLinuxDockerJenkinsTerraformNetworkingMonitoringProduction Support
18
Troubleshooting areas
78
Practical commands
8
Incident investigation steps

What are you troubleshooting?

Search by symptom, technology, command or production problem.

78 troubleshooting commands

Linux CPU High

top

Find processes consuming CPU in real time.

ps aux --sort=-%cpu | head

Show the highest CPU-consuming processes.

uptime

Check system load averages and system pressure.

pidstat 1

Monitor CPU usage by process over time.

Linux Memory High

free -h

Check total, used and available memory.

ps aux --sort=-%mem | head

Find processes consuming the most memory.

vmstat 1

Inspect memory, processes, paging and CPU activity.

dmesg | grep -i oom

Look for out-of-memory killer events.

Disk 100 Percent

df -h

Find filesystems that are running out of space.

df -i

Check inode consumption.

du -xh / | sort -h | tail -20

Find large directories and files.

journalctl --disk-usage

Check how much disk space systemd journals use.

Linux Service Not Starting

systemctl status nginx

Check the current service state and recent errors.

journalctl -u nginx -n 100

Read recent logs for the service.

systemctl cat nginx

Inspect the service unit configuration.

systemctl daemon-reload

Reload systemd after changing unit files.

Port Already In Use

ss -lntp

List listening TCP ports and owning processes.

ss -lntup

List listening TCP and UDP ports.

lsof -i :8080

Find the process using port 8080.

fuser -v 8080/tcp

Identify processes using a TCP port.

SSH Troubleshooting

ssh -vvv user@server

Enable verbose SSH debugging.

nc -vz server 22

Check whether TCP port 22 is reachable.

systemctl status ssh

Check the SSH server service.

ss -lntp | grep :22

Verify that SSH is listening.

DNS Troubleshooting

dig example.com

Inspect DNS resolution details.

dig +short example.com

Return a concise DNS answer.

nslookup example.com

Perform a basic DNS lookup.

cat /etc/resolv.conf

Check configured DNS resolvers on Linux.

Network Connectivity

ping -c 4 8.8.8.8

Check basic IP connectivity.

curl -I https://example.com

Check HTTP connectivity and response headers.

traceroute example.com

Inspect the network path toward a destination.

ip route

Inspect the local routing table.

Docker Container Crashing

docker ps -a

Check running and stopped containers.

docker logs <container>

Read container application logs.

docker inspect <container>

Inspect container configuration and state.

docker stats

Check container CPU and memory usage.

Docker Image Problems

docker images

List locally available images.

docker history <image>

Inspect image layers.

docker image inspect <image>

Inspect image metadata and configuration.

docker system df

Check Docker disk consumption.

Kubernetes CrashLoopBackOff

kubectl get pods

Identify pods that are repeatedly restarting.

kubectl describe pod <pod>

Inspect events, container state and configuration.

kubectl logs <pod> --previous

Read logs from the previous crashed container.

kubectl get events --sort-by=.lastTimestamp

Review recent Kubernetes events.

Kubernetes ImagePullBackOff

kubectl describe pod <pod>

Check image pull errors and related events.

kubectl get secret

Check whether image pull secrets exist.

kubectl get pod <pod> -o yaml

Inspect the pod image and imagePullSecrets configuration.

kubectl get events --sort-by=.lastTimestamp

Find registry authentication or image errors.

Kubernetes Pod Pending

kubectl get pods -o wide

Check where pending pods are scheduled or waiting.

kubectl describe pod <pod>

Inspect scheduler events and resource constraints.

kubectl get nodes

Check available cluster nodes.

kubectl describe node <node>

Inspect node capacity, conditions and taints.

Kubernetes Service Not Reachable

kubectl get svc

Check service configuration and ports.

kubectl get endpoints <service>

Verify that the service has backend endpoints.

kubectl get pods --show-labels

Compare pod labels with the service selector.

kubectl describe svc <service>

Inspect service selectors, ports and endpoints.

Kubernetes Node NotReady

kubectl get nodes

Identify nodes reporting NotReady.

kubectl describe node <node>

Inspect node conditions and recent events.

kubectl get pods -A -o wide

Check workloads running on affected nodes.

kubectl get events -A --sort-by=.lastTimestamp

Review cluster-wide events.

Jenkins Build Failure

docker logs <jenkins-container>

Inspect Jenkins container logs when Jenkins runs in Docker.

java -version

Verify the Java runtime used by Jenkins.

df -h

Check whether disk exhaustion is affecting builds.

free -h

Check available memory on the Jenkins host.

Terraform State Problems

terraform state list

List resources tracked in Terraform state.

terraform state show <resource>

Inspect a resource stored in state.

terraform plan

Compare configuration against current state and infrastructure.

terraform refresh

Refresh state from infrastructure when supported by the workflow.

Prometheus Target Down

curl http://localhost:9090/-/healthy

Check Prometheus health.

curl http://localhost:9090/api/v1/targets

Inspect target states through the Prometheus API.

curl http://target:9100/metrics

Check whether an exporter endpoint responds.

promtool check config prometheus.yml

Validate Prometheus configuration.

Production Incident Workflow

date

Record the current time when beginning incident investigation.

uptime

Check host uptime and system load.

df -h

Check for disk exhaustion.

free -h

Check memory pressure.

ss -lntup

Inspect listening network services.

journalctl -p err -n 100

Review recent high-priority system errors.

Production Troubleshooting Method

Troubleshoot from symptom to verified recovery

Production troubleshooting is more than finding a command that returns an error. First establish the symptom and impact, then collect evidence, isolate the failure domain, apply a controlled remediation and verify the result.

1

Confirm the symptom

Define exactly what is failing, when it started and which users, services or hosts are affected.

2

Scope the impact

Determine whether the problem affects one process, host, pod, service, region or the wider platform.

3

Check recent changes

Review deployments, configuration changes, infrastructure updates, patches and scheduled jobs.

4

Collect evidence

Use logs, metrics, events, process state, network checks and configuration inspection.

5

Form a hypothesis

Connect the observed evidence to a likely failure domain before changing the environment.

6

Apply a controlled fix

Choose the smallest safe remediation and understand its expected impact before execution.

7

Verify recovery

Confirm service health, application behaviour, error rates and dependent systems after the change.

8

Prevent recurrence

Document the root cause, improve monitoring or automation and capture the permanent corrective action.

SRE Mindset

Principles for safer production troubleshooting

Good incident investigation reduces uncertainty before making changes to a live system.

Evidence before action

Capture useful logs, metrics, events and system state before restarting, deleting or changing configuration.

Separate symptom from cause

A restart may restore service without explaining why the application failed. Treat recovery and root-cause analysis as separate steps.

Check recent changes

Deployment, configuration, infrastructure and dependency changes are important signals during incident investigation.

Change one thing at a time

Controlled changes make it easier to determine which action affected the system and reduce troubleshooting noise.

Explore

Troubleshoot by Technology

Start with the technology closest to the problem and work through the relevant checks.

Kubernetes

Investigate pods, scheduling, services, images, nodes, events and cluster failures.

Troubleshooting reference

Linux

Troubleshoot CPU, memory, disk, services, ports, SSH and operating-system problems.

Troubleshooting reference

Docker

Investigate containers, images, logs, resources, runtime state and startup failures.

Troubleshooting reference

Jenkins

Investigate CI/CD build failures, Java compatibility, agents, workspace and host resources.

Troubleshooting reference

Terraform

Investigate state, plans, infrastructure drift and infrastructure deployment problems.

Troubleshooting reference

Networking

Trace DNS, ports, routes, connectivity, SSH and HTTP communication problems.

Troubleshooting reference

Monitoring

Investigate Prometheus health, targets, exporters and monitoring signal failures.

Troubleshooting reference

Production Support

Follow an evidence-first workflow for incidents affecting applications and infrastructure.

Troubleshooting reference

Common Production Problems

Start with a Common Failure Symptom

Use these problem areas to quickly locate commands and investigation checks relevant to an incident.

Evidence Collection

What evidence should you collect during an incident?

Before changing a production system, capture enough information to understand the failure and preserve useful evidence for later root cause analysis.

Logs

Application, system, container and platform logs around the failure window.

Metrics

CPU, memory, disk, latency, throughput, errors and saturation signals.

Events

Kubernetes events, deployment events, alerts and infrastructure changes.

Changes

Deployments, configuration updates, patches, releases and scheduled jobs.

Avoid blind remediation

Restarting a service, deleting a pod, increasing resources or rolling back a deployment may restore availability, but those actions can also remove evidence. Capture the relevant state first whenever the incident allows it.

Command Reference

Practical DevOps Troubleshooting Commands

Searchable commands grouped by the production problems they help investigate. Use the commands to collect evidence and narrow the failure domain before applying remediation.

78 troubleshooting commands

Linux CPU High

top

Find processes consuming CPU in real time.

ps aux --sort=-%cpu | head

Show the highest CPU-consuming processes.

uptime

Check system load averages and system pressure.

pidstat 1

Monitor CPU usage by process over time.

Linux Memory High

free -h

Check total, used and available memory.

ps aux --sort=-%mem | head

Find processes consuming the most memory.

vmstat 1

Inspect memory, processes, paging and CPU activity.

dmesg | grep -i oom

Look for out-of-memory killer events.

Disk 100 Percent

df -h

Find filesystems that are running out of space.

df -i

Check inode consumption.

du -xh / | sort -h | tail -20

Find large directories and files.

journalctl --disk-usage

Check how much disk space systemd journals use.

Linux Service Not Starting

systemctl status nginx

Check the current service state and recent errors.

journalctl -u nginx -n 100

Read recent logs for the service.

systemctl cat nginx

Inspect the service unit configuration.

systemctl daemon-reload

Reload systemd after changing unit files.

Port Already In Use

ss -lntp

List listening TCP ports and owning processes.

ss -lntup

List listening TCP and UDP ports.

lsof -i :8080

Find the process using port 8080.

fuser -v 8080/tcp

Identify processes using a TCP port.

SSH Troubleshooting

ssh -vvv user@server

Enable verbose SSH debugging.

nc -vz server 22

Check whether TCP port 22 is reachable.

systemctl status ssh

Check the SSH server service.

ss -lntp | grep :22

Verify that SSH is listening.

DNS Troubleshooting

dig example.com

Inspect DNS resolution details.

dig +short example.com

Return a concise DNS answer.

nslookup example.com

Perform a basic DNS lookup.

cat /etc/resolv.conf

Check configured DNS resolvers on Linux.

Network Connectivity

ping -c 4 8.8.8.8

Check basic IP connectivity.

curl -I https://example.com

Check HTTP connectivity and response headers.

traceroute example.com

Inspect the network path toward a destination.

ip route

Inspect the local routing table.

Docker Container Crashing

docker ps -a

Check running and stopped containers.

docker logs <container>

Read container application logs.

docker inspect <container>

Inspect container configuration and state.

docker stats

Check container CPU and memory usage.

Docker Image Problems

docker images

List locally available images.

docker history <image>

Inspect image layers.

docker image inspect <image>

Inspect image metadata and configuration.

docker system df

Check Docker disk consumption.

Kubernetes CrashLoopBackOff

kubectl get pods

Identify pods that are repeatedly restarting.

kubectl describe pod <pod>

Inspect events, container state and configuration.

kubectl logs <pod> --previous

Read logs from the previous crashed container.

kubectl get events --sort-by=.lastTimestamp

Review recent Kubernetes events.

Kubernetes ImagePullBackOff

kubectl describe pod <pod>

Check image pull errors and related events.

kubectl get secret

Check whether image pull secrets exist.

kubectl get pod <pod> -o yaml

Inspect the pod image and imagePullSecrets configuration.

kubectl get events --sort-by=.lastTimestamp

Find registry authentication or image errors.

Kubernetes Pod Pending

kubectl get pods -o wide

Check where pending pods are scheduled or waiting.

kubectl describe pod <pod>

Inspect scheduler events and resource constraints.

kubectl get nodes

Check available cluster nodes.

kubectl describe node <node>

Inspect node capacity, conditions and taints.

Kubernetes Service Not Reachable

kubectl get svc

Check service configuration and ports.

kubectl get endpoints <service>

Verify that the service has backend endpoints.

kubectl get pods --show-labels

Compare pod labels with the service selector.

kubectl describe svc <service>

Inspect service selectors, ports and endpoints.

Kubernetes Node NotReady

kubectl get nodes

Identify nodes reporting NotReady.

kubectl describe node <node>

Inspect node conditions and recent events.

kubectl get pods -A -o wide

Check workloads running on affected nodes.

kubectl get events -A --sort-by=.lastTimestamp

Review cluster-wide events.

Jenkins Build Failure

docker logs <jenkins-container>

Inspect Jenkins container logs when Jenkins runs in Docker.

java -version

Verify the Java runtime used by Jenkins.

df -h

Check whether disk exhaustion is affecting builds.

free -h

Check available memory on the Jenkins host.

Terraform State Problems

terraform state list

List resources tracked in Terraform state.

terraform state show <resource>

Inspect a resource stored in state.

terraform plan

Compare configuration against current state and infrastructure.

terraform refresh

Refresh state from infrastructure when supported by the workflow.

Prometheus Target Down

curl http://localhost:9090/-/healthy

Check Prometheus health.

curl http://localhost:9090/api/v1/targets

Inspect target states through the Prometheus API.

curl http://target:9100/metrics

Check whether an exporter endpoint responds.

promtool check config prometheus.yml

Validate Prometheus configuration.

Production Incident Workflow

date

Record the current time when beginning incident investigation.

uptime

Check host uptime and system load.

df -h

Check for disk exhaustion.

free -h

Check memory pressure.

ss -lntup

Inspect listening network services.

journalctl -p err -n 100

Review recent high-priority system errors.

Troubleshooting Guide

Common DevOps troubleshooting questions

How should I troubleshoot a production issue?

Start by confirming the symptom and impact, check recent changes, collect logs and metrics, isolate the failure domain, apply a controlled remediation and verify that the service has recovered.

What should I check first when a server is slow?

Check CPU, load average, memory, disk usage, I/O, processes and network activity. Then correlate those signals with application logs and recent changes.

How do I troubleshoot Kubernetes pod failures?

Start with pod status, describe the pod, inspect current and previous container logs, review Kubernetes events and then investigate configuration, probes, resources, scheduling and dependencies.

What is the difference between troubleshooting and RCA?

Troubleshooting focuses on identifying and resolving the immediate failure. Root cause analysis goes further by establishing why the failure occurred and what permanent corrective or preventive actions are required.

Need the command, not the explanation?

Explore the DevOps command references for Kubernetes, Linux, Docker, AWS, Terraform, Ansible, Jenkins and more.

Explore DevOps Commands →