SRE • Monitoring • Prometheus

Prometheus Commands & PromQL Cheat Sheet

Practical Prometheus commands, promtool utilities, PromQL queries, monitoring, alerting, recording rules, configuration, Kubernetes monitoring and production troubleshooting.

What is Prometheus?

Prometheus is a monitoring and alerting system that collects time-series metrics from monitored targets. PromQL is its query language for selecting, aggregating and analyzing those metrics.

Showing 147 commands.

Prometheus Setup and Version

prometheus --version

Display the installed Prometheus version.

prometheus --help

Display Prometheus command-line help.

prometheus --help-long

Display detailed Prometheus command-line options.

prometheus --config.file=prometheus.yml

Start Prometheus using a specified configuration file.

prometheus --web.listen-address=0.0.0.0:9090

Configure the address and port used by the Prometheus web server.

prometheus --log.level=debug

Start Prometheus with debug-level logging.

Prometheus Docker

docker pull prom/prometheus

Download the official Prometheus container image.

docker run -p 9090:9090 prom/prometheus

Run Prometheus in Docker and expose port 9090.

docker run -p 9090:9090 -v ./prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Run Prometheus with a custom configuration file.

docker run -d --name prometheus -p 9090:9090 prom/prometheus

Run Prometheus as a detached Docker container.

docker logs prometheus

View Prometheus container logs.

docker restart prometheus

Restart the Prometheus container.

Prometheus Configuration

prometheus --config.file=prometheus.yml

Specify the Prometheus configuration file.

promtool check config prometheus.yml

Validate a Prometheus configuration file.

promtool check config prometheus.yml --lint=all

Validate configuration and apply available lint checks.

prometheus --config.file=prometheus.yml --config.auto-reload

Enable automatic configuration reload behavior.

curl -X POST http://localhost:9090/-/reload

Trigger a configuration reload when the lifecycle endpoint is enabled.

kill -HUP <prometheus-pid>

Reload Prometheus configuration by sending SIGHUP on supported systems.

Prometheus Targets and Scraping

curl http://localhost:9090/api/v1/targets

Query Prometheus target information through the HTTP API.

curl http://localhost:9090/api/v1/targets?state=active

Query active Prometheus targets.

curl http://localhost:9090/metrics

View Prometheus's own exposed metrics.

curl http://localhost:9100/metrics

View metrics exposed by a Node Exporter endpoint.

promtool check service-discovery prometheus.yml <job>

Inspect service discovery and relabeling for a configured job.

PromQL Basics

up

Return the health status of scraped targets.

up{job="node"}

Select target health metrics for the node job.

node_cpu_seconds_total

Query the Node Exporter CPU time metric.

node_memory_MemAvailable_bytes

Query available system memory exposed by Node Exporter.

node_filesystem_avail_bytes

Query available filesystem space.

http_requests_total

Query an HTTP request counter metric.

PromQL Label Selectors

up{job="node"}

Select series where the job label equals node.

up{job!="node"}

Select series where the job label is not node.

up{job=~"node|api"}

Select series whose job label matches a regular expression.

up{job!~"test.*"}

Exclude series whose job label matches a regular expression.

http_requests_total{status="500"}

Select HTTP requests with a 500 status label.

http_requests_total{method="GET",status="200"}

Select HTTP requests matching multiple labels.

PromQL Aggregation

sum(up)

Calculate the sum of selected series.

avg(up)

Calculate the average value across series.

min(up)

Return the minimum value across series.

max(up)

Return the maximum value across series.

count(up)

Count the number of returned series.

sum by (job) (up)

Aggregate target status by job.

sum by (instance) (up)

Aggregate target status by instance.

sum without (instance) (up)

Aggregate while excluding the instance label from grouping.

PromQL Rates and Counters

rate(http_requests_total[5m])

Calculate the per-second average increase of a counter over five minutes.

irate(http_requests_total[5m])

Calculate the per-second rate using the most recent samples.

increase(http_requests_total[1h])

Calculate the total counter increase over one hour.

sum(rate(http_requests_total[5m]))

Calculate the total request rate across selected series.

sum by (status) (rate(http_requests_total[5m]))

Calculate request rate grouped by HTTP status.

PromQL CPU and Memory

100 * (1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])))

Estimate overall CPU utilization from idle CPU time.

100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

Calculate memory utilization percentage.

sum by (instance) (rate(node_cpu_seconds_total[5m]))

Calculate CPU time rate grouped by instance.

node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes

Calculate used memory in bytes.

PromQL Disk and Filesystem

node_filesystem_avail_bytes{fstype!="tmpfs"}

Query available filesystem space while excluding tmpfs.

100 * (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes)

Calculate filesystem usage percentage.

node_filesystem_readonly == 1

Find filesystems mounted as read-only.

node_filesystem_avail_bytes / 1024 / 1024 / 1024

Convert available filesystem space from bytes to GiB.

PromQL HTTP and Application Monitoring

sum(rate(http_requests_total[5m]))

Calculate application request rate.

sum(rate(http_requests_total{status=~"5.."}[5m]))

Calculate HTTP 5xx error rate.

100 * sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

Calculate the percentage of requests returning HTTP 5xx responses.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Calculate the 95th percentile request latency from a classic histogram.

PromQL Histograms

rate(http_request_duration_seconds_bucket[5m])

Calculate the per-second rate for histogram buckets.

sum by (le) (rate(http_request_duration_seconds_bucket[5m]))

Aggregate histogram bucket rates.

histogram_quantile(0.50, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Calculate the 50th percentile latency.

histogram_quantile(0.90, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Calculate the 90th percentile latency.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Calculate the 95th percentile latency.

histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Calculate the 99th percentile latency.

PromQL Operators

node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes

Subtract available memory from total memory.

node_filesystem_avail_bytes / node_filesystem_size_bytes

Calculate the filesystem availability ratio.

rate(http_requests_total[5m]) * 60

Convert a per-second rate into an approximate per-minute rate.

up == 0

Select targets whose up value is zero.

node_filesystem_avail_bytes < 10737418240

Find filesystems with less than approximately 10 GiB available.

Recording Rules

promtool check rules rules.yml

Validate Prometheus recording and alerting rules.

promtool test rules test.yml

Run unit tests for Prometheus rules.

record: job:http_requests:rate5m

Example recording rule name for a reusable five-minute request rate.

expr: sum by (job) (rate(http_requests_total[5m]))

Example PromQL expression used by a recording rule.

Alerting Rules

promtool check rules alerts.yml

Validate alerting rules before deployment.

promtool test rules alerts-test.yml

Unit test alerting rules.

alert: HighCPUUsage

Example alert name for high CPU utilization.

expr: cpu_usage > 80

Example alert expression for CPU utilization.

for: 5m

Require an alert expression to remain active for a specified duration.

Promtool Configuration Validation

promtool --help

Display promtool command-line help.

promtool check config prometheus.yml

Validate Prometheus configuration.

promtool check rules rules.yml

Validate rule files.

promtool check web-config web.yml

Validate Prometheus web configuration.

promtool check metrics

Validate metrics supplied through standard input.

curl -s http://localhost:9090/metrics | promtool check metrics

Validate metrics exposed by a running Prometheus server.

Promtool Health Checks

promtool check healthy

Check whether the Prometheus server is healthy.

promtool check ready

Check whether the Prometheus server is ready.

promtool check healthy --url=http://localhost:9090

Check Prometheus health at a specific URL.

promtool check ready --url=http://localhost:9090

Check Prometheus readiness at a specific URL.

Promtool Querying

promtool query instant http://localhost:9090 up

Run an instant PromQL query against Prometheus.

promtool query range http://localhost:9090 'rate(http_requests_total[5m])'

Run a range query against Prometheus.

promtool query series http://localhost:9090

Query series information from Prometheus.

promtool query labels http://localhost:9090 __name__

Query label values from Prometheus.

Promtool Debugging

promtool debug metrics http://localhost:9090

Fetch Prometheus metrics debugging information.

promtool debug pprof http://localhost:9090

Fetch profiling information from Prometheus.

promtool debug all http://localhost:9090

Fetch available Prometheus debug information.

Promtool Rule Testing

promtool test rules test.yml

Run Prometheus rule unit tests.

promtool test rules test.yml --run <test-name>

Run selected rule test groups.

promtool test rules test.yml --junit results.xml

Write rule test results in JUnit XML format.

Prometheus HTTP API

curl http://localhost:9090/api/v1/query?query=up

Execute an instant PromQL query through the HTTP API.

curl 'http://localhost:9090/api/v1/query_range?query=up&start=<start>&end=<end>&step=15s'

Execute a range PromQL query through the HTTP API.

curl http://localhost:9090/api/v1/targets

Retrieve target information through the API.

curl http://localhost:9090/api/v1/rules

Retrieve configured recording and alerting rules.

curl http://localhost:9090/api/v1/alerts

Retrieve currently active alerts.

curl http://localhost:9090/api/v1/label/__name__/values

Retrieve metric name label values.

Prometheus Runtime and Reload

curl http://localhost:9090/-/healthy

Check the Prometheus HTTP health endpoint.

curl http://localhost:9090/-/ready

Check the Prometheus HTTP readiness endpoint.

curl -X POST http://localhost:9090/-/reload

Reload configuration when lifecycle management is enabled.

curl http://localhost:9090/api/v1/status/config

Retrieve the current Prometheus configuration.

curl http://localhost:9090/api/v1/status/runtimeinfo

Retrieve Prometheus runtime information.

curl http://localhost:9090/api/v1/status/flags

Retrieve Prometheus runtime flag values.

Prometheus Storage

prometheus --storage.tsdb.path=/prometheus

Configure the local TSDB storage path.

prometheus --storage.tsdb.retention.time=15d

Configure time-based local data retention.

prometheus --storage.tsdb.retention.size=20GB

Configure a size-based local data retention limit.

prometheus --storage.tsdb.wal-compression

Enable WAL compression.

Prometheus Query Performance

prometheus --query.max-concurrency=20

Configure the maximum number of concurrent queries.

prometheus --query.max-samples=50000000

Limit the number of samples a query can load.

prometheus --query.timeout=2m

Configure the maximum execution time for queries.

Kubernetes Monitoring

kubectl get pods -n monitoring

List monitoring namespace pods.

kubectl get servicemonitors -A

List ServiceMonitor resources across namespaces.

kubectl get prometheusrules -A

List PrometheusRule resources across namespaces.

kubectl logs -n monitoring <prometheus-pod>

View Prometheus pod logs.

kubectl port-forward -n monitoring svc/prometheus 9090:9090

Forward Prometheus web access to local port 9090.

kubectl describe pod -n monitoring <prometheus-pod>

Inspect the Prometheus pod when troubleshooting Kubernetes deployment issues.

Production Troubleshooting

promtool check config prometheus.yml

Validate configuration before restarting Prometheus.

promtool check rules rules.yml

Validate recording and alerting rules.

promtool check healthy

Check Prometheus health.

promtool check ready

Check Prometheus readiness.

curl http://localhost:9090/api/v1/targets

Inspect target health and scrape status.

curl http://localhost:9090/api/v1/alerts

Inspect active alerts.

curl http://localhost:9090/api/v1/status/runtimeinfo

Inspect Prometheus runtime information.

curl http://localhost:9090/api/v1/status/tsdb

Inspect TSDB statistics and storage information.

CI/CD Prometheus Workflow

promtool check config prometheus.yml

Validate Prometheus configuration in CI.

promtool check rules rules.yml

Validate alerting and recording rules in CI.

promtool test rules tests.yml

Run Prometheus rule tests in CI.

promtool check metrics < metrics.prom

Validate exported metrics during automated testing.

docker build -t prometheus-monitoring:latest .

Build a custom Prometheus monitoring image.

docker run -d --name prometheus -p 9090:9090 prometheus-monitoring:latest

Run the validated Prometheus image.

Recommended Prometheus Workflow

prometheus --version

Confirm the installed Prometheus version.

promtool check config prometheus.yml

Validate the Prometheus configuration.

promtool check rules rules.yml

Validate recording and alerting rules.

promtool test rules tests.yml

Run rule unit tests.

prometheus --config.file=prometheus.yml

Start Prometheus using the validated configuration.

promtool check ready

Confirm that Prometheus is ready.

curl http://localhost:9090/api/v1/targets

Verify target discovery and scrape status.

curl http://localhost:9090/api/v1/alerts

Verify active alert state.

Common Prometheus Workflow

prometheus --versionpromtool check config prometheus.ymlpromtool check rules rules.ymlpromtool test rules tests.ymlprometheus --config.file=prometheus.ymlpromtool check readycurl http://localhost:9090/api/v1/targets