SRE • Monitoring • Prometheus

Prometheus Commands & PromQL Cheat Sheet

Practical Prometheus commands, promtool utilities, PromQL queries, monitoring, alerting, recording rules, configuration, Kubernetes monitoring and production troubleshooting.

What is Prometheus?

Prometheus is a monitoring and alerting system that collects time-series metrics from monitored targets. PromQL is its query language for selecting, aggregating and analyzing those metrics.

Prometheus Command Categories

Jump directly to the Prometheus and PromQL commands you need.

Prometheus Setup and Version

Prometheus commands for prometheus setup and version.

View commands →

Prometheus Docker

Prometheus commands for prometheus docker.

View commands →

Prometheus Configuration

Prometheus commands for prometheus configuration.

View commands →

Prometheus Targets and Scraping

Prometheus commands for prometheus targets and scraping.

View commands →

PromQL Basics

Prometheus commands for promql basics.

View commands →

PromQL Label Selectors

Prometheus commands for promql label selectors.

View commands →

PromQL Aggregation

Prometheus commands for promql aggregation.

View commands →

PromQL Rates and Counters

Prometheus commands for promql rates and counters.

View commands →

PromQL CPU and Memory

Prometheus commands for promql cpu and memory.

View commands →

PromQL Disk and Filesystem

Prometheus commands for promql disk and filesystem.

View commands →

PromQL HTTP and Application Monitoring

Prometheus commands for promql http and application monitoring.

View commands →

PromQL Histograms

Prometheus commands for promql histograms.

View commands →

PromQL Operators

Prometheus commands for promql operators.

View commands →

Recording Rules

Prometheus commands for recording rules.

View commands →

Alerting Rules

Prometheus commands for alerting rules.

View commands →

Promtool Configuration Validation

Prometheus commands for promtool configuration validation.

View commands →

Promtool Health Checks

Prometheus commands for promtool health checks.

View commands →

Promtool Querying

Prometheus commands for promtool querying.

View commands →

Promtool Debugging

Prometheus commands for promtool debugging.

View commands →

Promtool Rule Testing

Prometheus commands for promtool rule testing.

View commands →

Prometheus HTTP API

Prometheus commands for prometheus http api.

View commands →

Prometheus Runtime and Reload

Prometheus commands for prometheus runtime and reload.

View commands →

Prometheus Storage

Prometheus commands for prometheus storage.

View commands →

Prometheus Query Performance

Prometheus commands for prometheus query performance.

View commands →

Kubernetes Monitoring

Prometheus commands for kubernetes monitoring.

View commands →

Production Troubleshooting

Prometheus commands for production troubleshooting.

View commands →

CI/CD Prometheus Workflow

Prometheus commands for ci/cd prometheus workflow.

View commands →

Recommended Prometheus Workflow

Prometheus commands for recommended prometheus workflow.

View commands →

Showing 147 commands across 28 categories.

Prometheus Setup and Version

6 commands

prometheus --version

Display the installed Prometheus version.

prometheus --help

Display Prometheus command-line help.

prometheus --help-long

Display detailed Prometheus command-line options.

prometheus --config.file=prometheus.yml

Start Prometheus using a specified configuration file.

prometheus --web.listen-address=0.0.0.0:9090

Configure the address and port used by the Prometheus web server.

prometheus --log.level=debug

Start Prometheus with debug-level logging.

Prometheus Docker

6 commands

docker pull prom/prometheus

Download the official Prometheus container image.

docker run -p 9090:9090 prom/prometheus

Run Prometheus in Docker and expose port 9090.

docker run -p 9090:9090 -v ./prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Run Prometheus with a custom configuration file.

docker run -d --name prometheus -p 9090:9090 prom/prometheus

Run Prometheus as a detached Docker container.

docker logs prometheus

View Prometheus container logs.

docker restart prometheus

Restart the Prometheus container.

Prometheus Configuration

6 commands

prometheus --config.file=prometheus.yml

Specify the Prometheus configuration file.

promtool check config prometheus.yml

Validate a Prometheus configuration file.

promtool check config prometheus.yml --lint=all

Validate configuration and apply available lint checks.

prometheus --config.file=prometheus.yml --config.auto-reload

Enable automatic configuration reload behavior.

curl -X POST http://localhost:9090/-/reload

Trigger a configuration reload when the lifecycle endpoint is enabled.

kill -HUP <prometheus-pid>

Reload Prometheus configuration by sending SIGHUP on supported systems.

Prometheus Targets and Scraping

5 commands

curl http://localhost:9090/api/v1/targets

Query Prometheus target information through the HTTP API.

curl http://localhost:9090/api/v1/targets?state=active

Query active Prometheus targets.

curl http://localhost:9090/metrics

View Prometheus's own exposed metrics.

curl http://localhost:9100/metrics

View metrics exposed by a Node Exporter endpoint.

promtool check service-discovery prometheus.yml <job>

Inspect service discovery and relabeling for a configured job.

PromQL Basics

6 commands

up

Return the health status of scraped targets.

up{job="node"}

Select target health metrics for the node job.

node_cpu_seconds_total

Query the Node Exporter CPU time metric.

node_memory_MemAvailable_bytes

Query available system memory exposed by Node Exporter.

node_filesystem_avail_bytes

Query available filesystem space.

http_requests_total

Query an HTTP request counter metric.

PromQL Label Selectors

6 commands

up{job="node"}

Select series where the job label equals node.

up{job!="node"}

Select series where the job label is not node.

up{job=~"node|api"}

Select series whose job label matches a regular expression.

up{job!~"test.*"}

Exclude series whose job label matches a regular expression.

http_requests_total{status="500"}

Select HTTP requests with a 500 status label.

http_requests_total{method="GET",status="200"}

Select HTTP requests matching multiple labels.

PromQL Aggregation

8 commands

sum(up)

Calculate the sum of selected series.

avg(up)

Calculate the average value across series.

min(up)

Return the minimum value across series.

max(up)

Return the maximum value across series.

count(up)

Count the number of returned series.

sum by (job) (up)

Aggregate target status by job.

sum by (instance) (up)

Aggregate target status by instance.

sum without (instance) (up)

Aggregate while excluding the instance label from grouping.

PromQL Rates and Counters

5 commands

rate(http_requests_total[5m])

Calculate the per-second average increase of a counter over five minutes.

irate(http_requests_total[5m])

Calculate the per-second rate using the most recent samples.

increase(http_requests_total[1h])

Calculate the total counter increase over one hour.

sum(rate(http_requests_total[5m]))

Calculate the total request rate across selected series.

sum by (status) (rate(http_requests_total[5m]))

Calculate request rate grouped by HTTP status.

PromQL CPU and Memory

4 commands

100 * (1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])))

Estimate overall CPU utilization from idle CPU time.

100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

Calculate memory utilization percentage.

sum by (instance) (rate(node_cpu_seconds_total[5m]))

Calculate CPU time rate grouped by instance.

node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes

Calculate used memory in bytes.

PromQL Disk and Filesystem

4 commands

node_filesystem_avail_bytes{fstype!="tmpfs"}

Query available filesystem space while excluding tmpfs.

100 * (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes)

Calculate filesystem usage percentage.

node_filesystem_readonly == 1

Find filesystems mounted as read-only.

node_filesystem_avail_bytes / 1024 / 1024 / 1024

Convert available filesystem space from bytes to GiB.

PromQL HTTP and Application Monitoring

4 commands

sum(rate(http_requests_total[5m]))

Calculate application request rate.

sum(rate(http_requests_total{status=~"5.."}[5m]))

Calculate HTTP 5xx error rate.

100 * sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

Calculate the percentage of requests returning HTTP 5xx responses.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Calculate the 95th percentile request latency from a classic histogram.

PromQL Histograms

6 commands

rate(http_request_duration_seconds_bucket[5m])

Calculate the per-second rate for histogram buckets.

sum by (le) (rate(http_request_duration_seconds_bucket[5m]))

Aggregate histogram bucket rates.

histogram_quantile(0.50, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Calculate the 50th percentile latency.

histogram_quantile(0.90, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Calculate the 90th percentile latency.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Calculate the 95th percentile latency.

histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Calculate the 99th percentile latency.

PromQL Operators

5 commands

node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes

Subtract available memory from total memory.

node_filesystem_avail_bytes / node_filesystem_size_bytes

Calculate the filesystem availability ratio.

rate(http_requests_total[5m]) * 60

Convert a per-second rate into an approximate per-minute rate.

up == 0

Select targets whose up value is zero.

node_filesystem_avail_bytes < 10737418240

Find filesystems with less than approximately 10 GiB available.

Recording Rules

4 commands

promtool check rules rules.yml

Validate Prometheus recording and alerting rules.

promtool test rules test.yml

Run unit tests for Prometheus rules.

record: job:http_requests:rate5m

Example recording rule name for a reusable five-minute request rate.

expr: sum by (job) (rate(http_requests_total[5m]))

Example PromQL expression used by a recording rule.

Alerting Rules

5 commands

promtool check rules alerts.yml

Validate alerting rules before deployment.

promtool test rules alerts-test.yml

Unit test alerting rules.

alert: HighCPUUsage

Example alert name for high CPU utilization.

expr: cpu_usage > 80

Example alert expression for CPU utilization.

for: 5m

Require an alert expression to remain active for a specified duration.

Promtool Configuration Validation

6 commands

promtool --help

Display promtool command-line help.

promtool check config prometheus.yml

Validate Prometheus configuration.

promtool check rules rules.yml

Validate rule files.

promtool check web-config web.yml

Validate Prometheus web configuration.

promtool check metrics

Validate metrics supplied through standard input.

curl -s http://localhost:9090/metrics | promtool check metrics

Validate metrics exposed by a running Prometheus server.

Promtool Health Checks

4 commands

promtool check healthy

Check whether the Prometheus server is healthy.

promtool check ready

Check whether the Prometheus server is ready.

promtool check healthy --url=http://localhost:9090

Check Prometheus health at a specific URL.

promtool check ready --url=http://localhost:9090

Check Prometheus readiness at a specific URL.

Promtool Querying

4 commands

promtool query instant http://localhost:9090 up

Run an instant PromQL query against Prometheus.

promtool query range http://localhost:9090 'rate(http_requests_total[5m])'

Run a range query against Prometheus.

promtool query series http://localhost:9090

Query series information from Prometheus.

promtool query labels http://localhost:9090 __name__

Query label values from Prometheus.

Promtool Debugging

3 commands

promtool debug metrics http://localhost:9090

Fetch Prometheus metrics debugging information.

promtool debug pprof http://localhost:9090

Fetch profiling information from Prometheus.

promtool debug all http://localhost:9090

Fetch available Prometheus debug information.

Promtool Rule Testing

3 commands

promtool test rules test.yml

Run Prometheus rule unit tests.

promtool test rules test.yml --run <test-name>

Run selected rule test groups.

promtool test rules test.yml --junit results.xml

Write rule test results in JUnit XML format.

Prometheus HTTP API

6 commands

curl http://localhost:9090/api/v1/query?query=up

Execute an instant PromQL query through the HTTP API.

curl 'http://localhost:9090/api/v1/query_range?query=up&start=<start>&end=<end>&step=15s'

Execute a range PromQL query through the HTTP API.

curl http://localhost:9090/api/v1/targets

Retrieve target information through the API.

curl http://localhost:9090/api/v1/rules

Retrieve configured recording and alerting rules.

curl http://localhost:9090/api/v1/alerts

Retrieve currently active alerts.

curl http://localhost:9090/api/v1/label/__name__/values

Retrieve metric name label values.

Prometheus Runtime and Reload

6 commands

curl http://localhost:9090/-/healthy

Check the Prometheus HTTP health endpoint.

curl http://localhost:9090/-/ready

Check the Prometheus HTTP readiness endpoint.

curl -X POST http://localhost:9090/-/reload

Reload configuration when lifecycle management is enabled.

curl http://localhost:9090/api/v1/status/config

Retrieve the current Prometheus configuration.

curl http://localhost:9090/api/v1/status/runtimeinfo

Retrieve Prometheus runtime information.

curl http://localhost:9090/api/v1/status/flags

Retrieve Prometheus runtime flag values.

Prometheus Storage

4 commands

prometheus --storage.tsdb.path=/prometheus

Configure the local TSDB storage path.

prometheus --storage.tsdb.retention.time=15d

Configure time-based local data retention.

prometheus --storage.tsdb.retention.size=20GB

Configure a size-based local data retention limit.

prometheus --storage.tsdb.wal-compression

Enable WAL compression.

Prometheus Query Performance

3 commands

prometheus --query.max-concurrency=20

Configure the maximum number of concurrent queries.

prometheus --query.max-samples=50000000

Limit the number of samples a query can load.

prometheus --query.timeout=2m

Configure the maximum execution time for queries.

Kubernetes Monitoring

6 commands

kubectl get pods -n monitoring

List monitoring namespace pods.

kubectl get servicemonitors -A

List ServiceMonitor resources across namespaces.

kubectl get prometheusrules -A

List PrometheusRule resources across namespaces.

kubectl logs -n monitoring <prometheus-pod>

View Prometheus pod logs.

kubectl port-forward -n monitoring svc/prometheus 9090:9090

Forward Prometheus web access to local port 9090.

kubectl describe pod -n monitoring <prometheus-pod>

Inspect the Prometheus pod when troubleshooting Kubernetes deployment issues.

Production Troubleshooting

8 commands

promtool check config prometheus.yml

Validate configuration before restarting Prometheus.

promtool check rules rules.yml

Validate recording and alerting rules.

promtool check healthy

Check Prometheus health.

promtool check ready

Check Prometheus readiness.

curl http://localhost:9090/api/v1/targets

Inspect target health and scrape status.

curl http://localhost:9090/api/v1/alerts

Inspect active alerts.

curl http://localhost:9090/api/v1/status/runtimeinfo

Inspect Prometheus runtime information.

curl http://localhost:9090/api/v1/status/tsdb

Inspect TSDB statistics and storage information.

CI/CD Prometheus Workflow

6 commands

promtool check config prometheus.yml

Validate Prometheus configuration in CI.

promtool check rules rules.yml

Validate alerting and recording rules in CI.

promtool test rules tests.yml

Run Prometheus rule tests in CI.

promtool check metrics < metrics.prom

Validate exported metrics during automated testing.

docker build -t prometheus-monitoring:latest .

Build a custom Prometheus monitoring image.

docker run -d --name prometheus -p 9090:9090 prometheus-monitoring:latest

Run the validated Prometheus image.

Recommended Prometheus Workflow

8 commands

prometheus --version

Confirm the installed Prometheus version.

promtool check config prometheus.yml

Validate the Prometheus configuration.

promtool check rules rules.yml

Validate recording and alerting rules.

promtool test rules tests.yml

Run rule unit tests.

prometheus --config.file=prometheus.yml

Start Prometheus using the validated configuration.

promtool check ready

Confirm that Prometheus is ready.

curl http://localhost:9090/api/v1/targets

Verify target discovery and scrape status.

curl http://localhost:9090/api/v1/alerts

Verify active alert state.

Common Prometheus Workflow

prometheus --versionpromtool check config prometheus.ymlpromtool check rules rules.ymlpromtool test rules tests.ymlprometheus --config.file=prometheus.ymlpromtool check readycurl http://localhost:9090/api/v1/targets