Ever opened htop on machine after machine just to figure out why a server feels slow? That used to be me. Every time something looked off, I would SSH into the wrong box first, exit, then try the next one. Annoying.
In this article (and video), we build a monitoring stack: Prometheus collects metrics from all your machines, Grafana draws them as graphs, and Alertmanager tells you when something goes wrong, like a disk almost full. Every component runs as a Docker container on one machine, while each monitored machine only needs a lightweight exporter.
Bonus: we also pull metrics straight from the Proxmox VE API using PVE Exporter, so VMs and Proxmox nodes all show up in one dashboard.
Versions used here: Prometheus v3.15.1 and Grafana OSS (latest). When newer versions come out, just change the numbers in the compose file.
What We Are Building #
Six containers on one Docker network:
| Container | Job | Port |
|---|---|---|
| prometheus | Collects and stores metrics | 9090 |
| grafana | Dashboards and graphs | 3000 |
| node-exporter | Hardware and OS metrics | 9100 |
| cadvisor | Per-container Docker metrics | 8080 |
| alertmanager | Sends alert notifications | 9093 |
| pve-exporter | Proxmox VE metrics via API | 9221 |
How it works, briefly. Prometheus uses a pull model: every 15 seconds it visits the /metrics URL of each target and stores the numbers it finds. An exporter is a small program whose job is serving those numbers. Grafana stores nothing itself, it only displays query results from Prometheus.
Prerequisites #
- One Linux machine with Docker and Docker Compose plugin v2 installed. 2 CPU cores and 4 GB RAM is comfortable.
- SSH access to that machine, plus to any other machine you want to monitor.
- At least one Proxmox VE node if you want the PVE part. Web UI access is enough, no Proxmox CLI needed.
Step 1 📁 Project Folder Structure #
Create the project directory and subfolders to store configuration files:
mkdir -p monitoring-stack/prometheus monitoring-stack/alertmanager
cd monitoring-stackThese two folders will hold our config files. Everything lives outside the containers via volume bind mounts on purpose, so you can edit them with any editor and reload without rebuilding anything.
Step 2 🐳 Create the Docker Compose File #
Create compose.yml in the project root folder (monitoring-stack/compose.yml):
services:
prometheus:
image: prom/prometheus:v3.15.1
container_name: prometheus
volumes:
- ./prometheus:/etc/prometheus
- prometheus-data:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.retention.time=15d"
- "--web.enable-lifecycle"
ports:
- "9090:9090"
restart: unless-stopped
grafana:
image: grafana/grafana-oss:latest
container_name: grafana
volumes:
- grafana-data:/var/lib/grafana
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=changeme_now
restart: unless-stopped
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
pid: host
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
command:
- "--path.procfs=/host/proc"
- "--path.sysfs=/host/sys"
- "--path.rootfs=/rootfs"
ports:
- "9100:9100"
restart: unless-stopped
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
ports:
- "8080:8080"
restart: unless-stopped
alertmanager:
image: prom/alertmanager:latest
container_name: alertmanager
volumes:
- ./alertmanager:/etc/alertmanager
ports:
- "9093:9093"
restart: unless-stopped
pve-exporter:
image: prompve/prometheus-pve-exporter:latest
container_name: pve-exporter
volumes:
- ./pve.yml:/etc/prometheus/pve.yml
ports:
- "9221:9221"
environment:
- PROXMOX_VE_USERNAME=grafana@pam
- PROXMOX_VE_TOKEN_NAME=grafana
- PROXMOX_VE_TOKEN_VALUE=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
restart: unless-stopped
volumes:
prometheus-data:
grafana-data:What must be replaced: GF_SECURITY_ADMIN_PASSWORD and PROXMOX_VE_TOKEN_VALUE (match the token generated in the next step). The --web.enable-lifecycle flag on Prometheus enables reloading configs on the fly without restarting the container.
Do not run compose yet—we will prepare all configuration files first to avoid crash loops or missing volume errors.
Step 3 🔑 Proxmox VE Configuration & pve.yml
#
If you use Proxmox VE, two preparation steps are required on the Proxmox side before the exporter can query cluster metrics.
1. Install Node Exporter on Each Proxmox Node #
SSH into each Proxmox node and run:
apt update
apt install prometheus-node-exporterThis package automatically runs as a systemd service and exposes host metrics on port 9100. The native Debian package is lightweight and does not require Docker on Proxmox.
2. Create User, API Token, and pve.yml
#
PVE Exporter reads metrics via Proxmox REST API, requiring a token with read-only permissions:
- In the Proxmox Web UI, navigate to Datacenter > Permissions > Users and click Add. Set username
grafana, realm Linux PAM (grafana@pam). - Go to Datacenter > Permissions > API Tokens, click Add. Select user
grafana@pam, Token IDgrafana, and uncheck Privilege separation so the token inherits the user permissions. - Copy the token Secret (UUID string) immediately. It will not be shown again once closed.
- Go to Datacenter > Permissions > Add > User Permission. Choose path
/, usergrafana@pam, and role PVEAuditor.
Permissions at path
/ apply across all nodes if your Proxmox hosts belong to a cluster. If your nodes are unclustered standalone hosts, repeat the user and token setup on each node.
Now switch back to your Docker host and create pve.yml in the project root directory (monitoring-stack/pve.yml):
default:
username: grafana@pam
token_name: grafana
token_value: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
verify_ssl: falseReplace token_value with your copied UUID. verify_ssl: false is typically required in homelabs because Proxmox SSL certificates are self-signed.
Step 4 📝 Prometheus Configuration (prometheus/prometheus.yml)
#
Create prometheus/prometheus.yml:
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets: ["alertmanager:9093"]
rule_files:
- "alert_rules.yml"
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "node-exporter"
static_configs:
- targets: ["node-exporter:9100", "192.168.100.10:9100", "192.168.100.20:9100", "192.168.100.60:9100"]
- job_name: "cadvisor"
static_configs:
- targets: ["cadvisor:8080"]
- job_name: "pve"
static_configs:
- targets:
- 192.168.100.10 # Proxmox VE node 1
- 192.168.100.20 # Proxmox VE node 2
metrics_path: /pve
params:
module: [default]
cluster: ['1']
node: ['1']
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: pve-exporter:9221 # PVE exporterKey points:
node-exporter:9100is the container inside our compose stack, monitoring the primary Docker host.- IP targets (
192.168.100.x:9100) point to remote machines or Proxmox nodes running node-exporter. Adjust these to match your LAN IPs. - On the
pvejob, relabeling passes the target IP topve-exporter:9221. One single exporter container handles all your Proxmox nodes.
Step 5 🚨 Alertmanager, ntfy & Alert Rules #
Set up the alerting system before bringing up the containers.
1. File alertmanager/alertmanager.yml
#
Create alertmanager/alertmanager.yml:
route:
group_by: ["alertname"]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: ntfy-notifications
receivers:
# Option 1: Public / No Password (ntfy.sh or self-hosted without auth)
- name: ntfy-notifications
webhook_configs:
- url: "https://ntfy.sh/your-secret-topic-name"
send_resolved: true
# Option 2: With Password / Token (Self-hosted with auth protection)
# - name: ntfy-notifications
# webhook_configs:
# - url: "https://ntfy.yourdomain.com/alerts"
# send_resolved: true
# # Method A: Username & Password (Basic Auth)
# http_config:
# basic_auth:
# username: "admin"
# password: "your_secret_password"
# # Method B: Access Token (Bearer Auth)
# # http_config:
# # authorization:
# # type: Bearer
# # credentials: "tk_xxxxxxxxxxxxxxxxxxxxxx"The route block controls alert grouping (group_by), a 30-second initial wait, and a 4-hour repeat interval for unresolved alerts. Alerts route to ntfy via webhook. Choose between public unauthenticated endpoints (use an unguessable topic name) or protected private endpoints with Basic Auth or Bearer tokens.
2. File prometheus/alert_rules.yml
#
Create prometheus/alert_rules.yml:
groups:
- name: host-alerts
rules:
- alert: HighCPUUsage
expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU usage on {{ $labels.instance }}"
description: "CPU usage has been above 85% for 5 minutes on {{ $labels.instance }}."
- alert: LowDiskSpace
expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
for: 10m
labels:
severity: critical
annotations:
summary: "Low disk space on {{ $labels.instance }}"
description: "Root filesystem free space is below 10% on {{ $labels.instance }}."HighCPUUsage: triggers when average CPU across 5 minutes exceeds 85%. Thefor: 5mcondition avoids noisy alerts from brief CPU spikes.LowDiskSpace: tracks root filesystem (/) free space and raises a critical alert when free space drops below 10% for 10 minutes.
Step 6 🚀 Launch the Docker Compose Stack #
All configuration files (compose.yml, pve.yml, prometheus.yml, alert_rules.yml, and alertmanager.yml) are in place. Now start the stack:
docker compose up -d
docker compose psAll six containers should show state Up. If a container shows Restarting, verify all volume-mounted files exist on the host (most commonly: a missing pve.yml).
Step 7 ✅ Verify Targets & Alert Rules in Prometheus #
Open your browser and navigate to: http://<HOST-IP>:9090/targets
Verify that all endpoints show a green state (UP):
prometheus(localhost:9090)node-exporter(local host and remote machines)cadvisor(local container metrics)pve(all registered Proxmox nodes)
If any target is DOWN, check whether the IP is reachable, port 9100 is allowed by your firewall, or the exporter service is running on the target machine.
Next, open the Alerts tab at http://<HOST-IP>:9090/alerts. Both rules (HighCPUUsage and LowDiskSpace) should be listed with state Inactive (normal state).
Step 8 📊 Connect Grafana to Prometheus #
- Open Grafana in your browser:
http://<HOST-IP>:3000 - Log in with user
adminand the password configured incompose.yml. - Navigate to Connections > Data sources > Add data source, then select Prometheus.
- In the Prometheus server URL field, enter:
http://prometheus:9090 - Click Save & test.
The data source URL must be
http://prometheus:9090, NOT localhost. Grafana runs in its own Docker container, so localhost refers to itself. Always use the Docker service name.
Step 9 📈 Add Dashboards #
Community dashboards provide battle-tested monitoring layouts out of the box. Go to Dashboards > New > Import and enter these IDs:
- 1860, Node Exporter Full: comprehensive hardware and system metrics for a single host. Select hosts via the Host dropdown.
- 25823, cAdvisor - Containers, Host & Runtime: real-time per-container CPU, RAM, and network metrics.
- 10347, Proxmox via Prometheus: cluster overview, node health, ZFS/LVM storage, and VM/LXC resource consumption.
🎁 Bonus: Custom “Multi Server Overview” Dashboard #
If you monitor multiple servers at once (e.g., your primary Docker host, a few VMs, and Proxmox nodes), constantly toggling host dropdowns on standard dashboards gets tedious.
I built a custom dashboard (Multi Server Overview) designed to give you a fleet-wide birds-eye view of all your servers in one single screen:
- Fleet Health Summary: online vs offline host counts, actively firing alerts, cluster-wide average CPU & RAM, and kernel OOM kill counters.
- Server Summary Table: a single row per server displaying UP/DOWN status, CPU %, RAM %, root disk usage, Load 1m, Uptime, CPU cores, and Kernel release. Clicking any instance takes you directly to its full detailed Node Exporter Full view.
- Disk Capacity Forecasting: a linear projection gauge estimating days until the root filesystem (
/) runs full based on 24-hour consumption trends. - Fleet-Wide Time Series: side-by-side metric graphs comparing CPU usage, RAM (used vs total), Load average per core, Network bandwidth (in/out), Disk I/O, hardware temperatures, clock drift, and active TCP connections across all hosts.
How to Install:
- In Grafana, navigate to Dashboards > New > Import.
- Copy the entire JSON snippet below and paste it into the Import via panel json box.
- Click Load, select your Prometheus data source, then click Import.
Click here to view & copy the Dashboard JSON code
{
"id": null,
"uid": "multi-server-overview",
"title": "Multi Server Overview",
"description": "Fleet-wide overview of all servers monitored with node_exporter. Click an instance for detailed metrics.",
"tags": [
"node-exporter",
"prometheus",
"multi-server",
"overview"
],
"timezone": "browser",
"editable": true,
"graphTooltip": 1,
"schemaVersion": 39,
"version": 2,
"refresh": "30s",
"time": {
"from": "now-6h",
"to": "now"
},
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"links": [
{
"title": "Node Exporter Full",
"type": "link",
"icon": "dashboard",
"url": "/d/rYdddlPWk/node-exporter-full",
"targetBlank": true,
"includeVars": false,
"keepTime": true,
"tags": [],
"asDropdown": false,
"tooltip": ""
}
],
"templating": {
"list": [
{
"name": "ds_prometheus",
"label": "Datasource",
"type": "datasource",
"query": "prometheus",
"refresh": 1,
"hide": 0,
"includeAll": false,
"multi": false,
"options": [],
"current": {},
"regex": ""
},
{
"name": "job",
"label": "Job",
"type": "query",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"definition": "label_values(node_uname_info, job)",
"query": {
"query": "label_values(node_uname_info, job)",
"refId": "var-job"
},
"refresh": 2,
"sort": 1,
"multi": true,
"includeAll": true,
"allValue": ".*",
"hide": 0,
"options": [],
"regex": "",
"current": {
"selected": true,
"text": [
"node-exporter"
],
"value": [
"node-exporter"
]
}
},
{
"name": "instance",
"label": "Instance",
"type": "query",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"definition": "label_values(node_uname_info{job=~\"$job\"}, instance)",
"query": {
"query": "label_values(node_uname_info{job=~\"$job\"}, instance)",
"refId": "var-instance"
},
"refresh": 2,
"sort": 1,
"multi": true,
"includeAll": true,
"allValue": ".*",
"hide": 0,
"options": [],
"regex": "",
"current": {
"selected": true,
"text": "All",
"value": "$__all"
}
}
]
},
"panels": [
{
"id": 1,
"type": "row",
"title": "Overview",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"panels": []
},
{
"id": 2,
"type": "stat",
"title": "Hosts Online",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 4,
"w": 3,
"x": 0,
"y": 1
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "count(up{job=~\"$job\",instance=~\"$instance\"} == 1) or vector(0)",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "thresholds"
},
"mappings": []
},
"overrides": []
},
"options": {
"colorMode": "background",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 3,
"type": "stat",
"title": "Hosts Offline",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 4,
"w": 3,
"x": 3,
"y": 1
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "count(up{job=~\"$job\",instance=~\"$instance\"} == 0) or vector(0)",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 1
}
]
},
"color": {
"mode": "thresholds"
},
"mappings": []
},
"overrides": []
},
"options": {
"colorMode": "background",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 4,
"type": "stat",
"title": "Firing Alerts",
"description": "Alerts currently firing in Prometheus (all rules, not filtered by host).",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 4,
"w": 3,
"x": 6,
"y": 1
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "count(ALERTS{alertstate=\"firing\"}) or vector(0)",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 1
}
]
},
"color": {
"mode": "thresholds"
},
"mappings": []
},
"overrides": []
},
"options": {
"colorMode": "background",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 5,
"type": "stat",
"title": "Avg CPU",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 4,
"w": 3,
"x": 9,
"y": 1
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "avg(100 - (avg by (instance) (rate(node_cpu_seconds_total{job=~\"$job\",instance=~\"$instance\",mode=\"idle\"}[$__rate_interval])) * 100))",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
},
"color": {
"mode": "thresholds"
},
"mappings": [],
"decimals": 1,
"min": 0,
"max": 100
},
"overrides": []
},
"options": {
"colorMode": "background",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 6,
"type": "stat",
"title": "Avg RAM",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 4,
"w": 3,
"x": 12,
"y": 1
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "avg(100 * (1 - node_memory_MemAvailable_bytes{job=~\"$job\",instance=~\"$instance\"} / node_memory_MemTotal_bytes{job=~\"$job\",instance=~\"$instance\"}))",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
},
"color": {
"mode": "thresholds"
},
"mappings": [],
"decimals": 1,
"min": 0,
"max": 100
},
"overrides": []
},
"options": {
"colorMode": "background",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 7,
"type": "stat",
"title": "Highest Root Disk",
"description": "Highest usage of the root filesystem (/) across all hosts.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 4,
"w": 3,
"x": 15,
"y": 1
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max(100 - (node_filesystem_avail_bytes{job=~\"$job\",instance=~\"$instance\",mountpoint=\"/\",fstype!~\"rootfs|tmpfs\"} / node_filesystem_size_bytes{job=~\"$job\",instance=~\"$instance\",mountpoint=\"/\",fstype!~\"rootfs|tmpfs\"} * 100))",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
},
"color": {
"mode": "thresholds"
},
"mappings": [],
"decimals": 1,
"min": 0,
"max": 100
},
"overrides": []
},
"options": {
"colorMode": "background",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 8,
"type": "stat",
"title": "Total CPU Cores",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 4,
"w": 3,
"x": 18,
"y": 1
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "sum(count by (instance) (count by (instance, cpu) (node_cpu_seconds_total{job=~\"$job\",instance=~\"$instance\",mode=\"idle\"})))",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "thresholds"
},
"mappings": [],
"decimals": 0
},
"overrides": []
},
"options": {
"colorMode": "background",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 9,
"type": "stat",
"title": "OOM Kills",
"description": "Processes killed by the kernel out-of-memory killer during the selected time range.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 4,
"w": 3,
"x": 21,
"y": 1
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "sum(increase(node_vmstat_oom_kill{job=~\"$job\",instance=~\"$instance\"}[$__range])) or vector(0)",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 1
}
]
},
"color": {
"mode": "thresholds"
},
"mappings": [],
"decimals": 0
},
"overrides": []
},
"options": {
"colorMode": "background",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 10,
"type": "table",
"title": "Server Summary",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"description": "One row per server. Click the instance to open its detailed Node Exporter Full dashboard.",
"gridPos": {
"h": 9,
"w": 24,
"x": 0,
"y": 5
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (up{job=~\"$job\",instance=~\"$instance\"})",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (100 - (avg by (instance) (rate(node_cpu_seconds_total{job=~\"$job\",instance=~\"$instance\",mode=\"idle\"}[$__rate_interval])) * 100))",
"legendFormat": "",
"refId": "B",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (100 * (1 - node_memory_MemAvailable_bytes{job=~\"$job\",instance=~\"$instance\"} / node_memory_MemTotal_bytes{job=~\"$job\",instance=~\"$instance\"}))",
"legendFormat": "",
"refId": "C",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (100 - (node_filesystem_avail_bytes{job=~\"$job\",instance=~\"$instance\",mountpoint=\"/\",fstype!~\"rootfs|tmpfs\"} / node_filesystem_size_bytes{job=~\"$job\",instance=~\"$instance\",mountpoint=\"/\",fstype!~\"rootfs|tmpfs\"} * 100))",
"legendFormat": "",
"refId": "D",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (node_load1{job=~\"$job\",instance=~\"$instance\"})",
"legendFormat": "",
"refId": "E",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (time() - node_boot_time_seconds{job=~\"$job\",instance=~\"$instance\"})",
"legendFormat": "",
"refId": "F",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "count by (instance) (count by (instance, cpu) (node_cpu_seconds_total{job=~\"$job\",instance=~\"$instance\",mode=\"idle\"}))",
"legendFormat": "",
"refId": "G",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance, nodename, release) (node_uname_info{job=~\"$job\",instance=~\"$instance\"})",
"legendFormat": "",
"refId": "H",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (node_memory_MemTotal_bytes{job=~\"$job\",instance=~\"$instance\"})",
"legendFormat": "",
"refId": "I",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (node_memory_MemTotal_bytes{job=~\"$job\",instance=~\"$instance\"} - node_memory_MemAvailable_bytes{job=~\"$job\",instance=~\"$instance\"})",
"legendFormat": "",
"refId": "J",
"range": false,
"instant": true,
"format": "table"
}
],
"transformations": [
{
"id": "joinByField",
"options": {
"byField": "instance",
"mode": "outer"
}
},
{
"id": "filterFieldsByName",
"options": {
"include": {
"pattern": "^(nodename|instance|release|Value #[A-GIJ])$"
}
}
},
{
"id": "organize",
"options": {
"excludeByName": {},
"indexByName": {
"nodename": 0,
"instance": 1,
"Value #A": 2,
"Value #B": 3,
"Value #C": 4,
"Value #J": 5,
"Value #I": 6,
"Value #D": 7,
"Value #E": 8,
"Value #G": 9,
"Value #F": 10,
"release": 11
},
"renameByName": {}
}
}
],
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": false
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"mappings": []
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "nodename"
},
"properties": [
{
"id": "displayName",
"value": "Hostname"
},
{
"id": "custom.width",
"value": 160
}
]
},
{
"matcher": {
"id": "byName",
"options": "instance"
},
"properties": [
{
"id": "displayName",
"value": "Instance"
},
{
"id": "links",
"value": [
{
"title": "Open in Node Exporter Full",
"url": "/d/rYdddlPWk/node-exporter-full?${job:queryparam}&var-nodename=${__data.fields.nodename}&var-node=${__data.fields.instance}",
"targetBlank": true
}
]
},
{
"id": "custom.width",
"value": 170
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #A"
},
"properties": [
{
"id": "displayName",
"value": "Status"
},
{
"id": "mappings",
"value": [
{
"type": "value",
"options": {
"1": {
"text": "UP",
"color": "green",
"index": 0
},
"0": {
"text": "DOWN",
"color": "red",
"index": 1
}
}
}
]
},
{
"id": "custom.cellOptions",
"value": {
"type": "color-background"
}
},
{
"id": "custom.width",
"value": 80
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #B"
},
"properties": [
{
"id": "displayName",
"value": "CPU %"
},
{
"id": "min",
"value": 0
},
{
"id": "max",
"value": 100
},
{
"id": "decimals",
"value": 1
},
{
"id": "thresholds",
"value": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
}
},
{
"id": "custom.cellOptions",
"value": {
"type": "gauge",
"mode": "lcd",
"valueDisplayMode": "text"
}
},
{
"id": "color",
"value": {
"mode": "thresholds"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #C"
},
"properties": [
{
"id": "displayName",
"value": "RAM %"
},
{
"id": "min",
"value": 0
},
{
"id": "max",
"value": 100
},
{
"id": "decimals",
"value": 1
},
{
"id": "thresholds",
"value": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
}
},
{
"id": "custom.cellOptions",
"value": {
"type": "gauge",
"mode": "lcd",
"valueDisplayMode": "text"
}
},
{
"id": "color",
"value": {
"mode": "thresholds"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #J"
},
"properties": [
{
"id": "displayName",
"value": "RAM Used"
},
{
"id": "unit",
"value": "bytes"
},
{
"id": "decimals",
"value": 1
},
{
"id": "custom.width",
"value": 110
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #I"
},
"properties": [
{
"id": "displayName",
"value": "RAM Total"
},
{
"id": "unit",
"value": "bytes"
},
{
"id": "decimals",
"value": 1
},
{
"id": "custom.width",
"value": 100
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #D"
},
"properties": [
{
"id": "displayName",
"value": "Disk /"
},
{
"id": "min",
"value": 0
},
{
"id": "max",
"value": 100
},
{
"id": "decimals",
"value": 1
},
{
"id": "thresholds",
"value": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
}
},
{
"id": "custom.cellOptions",
"value": {
"type": "gauge",
"mode": "lcd",
"valueDisplayMode": "text"
}
},
{
"id": "color",
"value": {
"mode": "thresholds"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #E"
},
"properties": [
{
"id": "displayName",
"value": "Load 1m"
},
{
"id": "decimals",
"value": 2
},
{
"id": "custom.width",
"value": 90
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #F"
},
"properties": [
{
"id": "displayName",
"value": "Uptime"
},
{
"id": "unit",
"value": "s"
},
{
"id": "custom.width",
"value": 130
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #G"
},
"properties": [
{
"id": "displayName",
"value": "Cores"
},
{
"id": "unit",
"value": "short"
},
{
"id": "decimals",
"value": 0
},
{
"id": "custom.width",
"value": 70
}
]
},
{
"matcher": {
"id": "byName",
"options": "release"
},
"properties": [
{
"id": "displayName",
"value": "Kernel"
}
]
}
]
},
"options": {
"showHeader": true,
"cellHeight": "sm",
"footer": {
"show": false
},
"sortBy": [
{
"displayName": "Hostname",
"desc": false
}
]
}
},
{
"id": 11,
"type": "row",
"title": "Alerts & Capacity",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 14
},
"panels": []
},
{
"id": 12,
"type": "table",
"title": "Firing Alerts",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"description": "Alerts currently firing in Prometheus, taken from the built-in ALERTS metric.",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 15
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "ALERTS{alertstate=\"firing\"}",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true,
"format": "table"
}
],
"transformations": [
{
"id": "filterFieldsByName",
"options": {
"include": {
"pattern": "^(alertname|severity|instance|job)$"
}
}
},
{
"id": "organize",
"options": {
"excludeByName": {},
"renameByName": {},
"indexByName": {
"alertname": 0,
"severity": 1,
"instance": 2,
"job": 3
}
}
}
],
"fieldConfig": {
"defaults": {
"noValue": "No firing alerts",
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
}
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"mappings": []
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "alertname"
},
"properties": [
{
"id": "displayName",
"value": "Alert"
}
]
},
{
"matcher": {
"id": "byName",
"options": "severity"
},
"properties": [
{
"id": "displayName",
"value": "Severity"
},
{
"id": "mappings",
"value": [
{
"type": "value",
"options": {
"critical": {
"text": "critical",
"color": "red",
"index": 0
},
"warning": {
"text": "warning",
"color": "orange",
"index": 1
}
}
}
]
},
{
"id": "custom.cellOptions",
"value": {
"type": "color-text"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "instance"
},
"properties": [
{
"id": "displayName",
"value": "Instance"
}
]
},
{
"matcher": {
"id": "byName",
"options": "job"
},
"properties": [
{
"id": "displayName",
"value": "Job"
}
]
}
]
},
"options": {
"showHeader": true,
"cellHeight": "sm",
"footer": {
"show": false
}
}
},
{
"id": 13,
"type": "bargauge",
"title": "Estimated Days Until Root Disk Is Full",
"description": "Linear projection from the last 24 hours of free space on /. Hosts whose free space is stable or growing are not listed.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 15
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) ((node_filesystem_avail_bytes{job=~\"$job\",instance=~\"$instance\",mountpoint=\"/\",fstype!~\"rootfs|tmpfs\"} / (-deriv(node_filesystem_avail_bytes{job=~\"$job\",instance=~\"$instance\",mountpoint=\"/\",fstype!~\"rootfs|tmpfs\"}[24h]) > 0)) / 86400)",
"legendFormat": "{{instance}}",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "d",
"min": 0,
"decimals": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "red",
"value": null
},
{
"color": "orange",
"value": 7
},
{
"color": "green",
"value": 30
}
]
},
"color": {
"mode": "thresholds"
},
"max": 90,
"noValue": "No shrinking trend"
},
"overrides": []
},
"options": {
"displayMode": "gradient",
"orientation": "horizontal",
"showUnfilled": true,
"valueMode": "color",
"namePlacement": "left",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 14,
"type": "row",
"title": "CPU & Memory",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 23
},
"panels": []
},
{
"id": 15,
"type": "timeseries",
"title": "CPU Usage per Server",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 24
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "100 - (avg by (instance) (rate(node_cpu_seconds_total{job=~\"$job\",instance=~\"$instance\",mode=\"idle\"}[$__rate_interval])) * 100)",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0,
"max": 100
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 16,
"type": "timeseries",
"title": "RAM Usage per Server",
"description": "Calculated from MemAvailable, so page cache is not counted as used.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 24
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "100 * (1 - node_memory_MemAvailable_bytes{job=~\"$job\",instance=~\"$instance\"} / node_memory_MemTotal_bytes{job=~\"$job\",instance=~\"$instance\"})",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0,
"max": 100
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 17,
"type": "timeseries",
"title": "Load Average per Core (1m)",
"description": "Values above 1 mean more runnable tasks than CPU cores.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 32
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "node_load1{job=~\"$job\",instance=~\"$instance\"} / on(instance) (count by (instance) (count by (instance, cpu) (node_cpu_seconds_total{job=~\"$job\",instance=~\"$instance\",mode=\"idle\"})))",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 18,
"type": "timeseries",
"title": "Swap Usage",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 32
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "(node_memory_SwapTotal_bytes{job=~\"$job\",instance=~\"$instance\"} - node_memory_SwapFree_bytes{job=~\"$job\",instance=~\"$instance\"}) / (node_memory_SwapTotal_bytes{job=~\"$job\",instance=~\"$instance\"} > 0) * 100",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0,
"max": 100
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 19,
"type": "timeseries",
"title": "RAM Used vs Total (bytes)",
"description": "Dashed lines show the installed RAM capacity of each server.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 40
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "node_memory_MemTotal_bytes{job=~\"$job\",instance=~\"$instance\"} - node_memory_MemAvailable_bytes{job=~\"$job\",instance=~\"$instance\"}",
"legendFormat": "{{instance}} used",
"refId": "A",
"range": true,
"instant": false
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "node_memory_MemTotal_bytes{job=~\"$job\",instance=~\"$instance\"}",
"legendFormat": "{{instance}} total",
"refId": "B",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "bytes",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 10,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0
},
"overrides": [
{
"matcher": {
"id": "byRegexp",
"options": "/ total$/"
},
"properties": [
{
"id": "custom.fillOpacity",
"value": 0
},
{
"id": "custom.lineStyle",
"value": {
"fill": "dash",
"dash": [
10,
10
]
}
},
{
"id": "custom.lineWidth",
"value": 2
}
]
}
]
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 20,
"type": "timeseries",
"title": "Resource Pressure (PSI, worst host)",
"description": "Share of time tasks were stalled waiting for a resource (Linux PSI). Only hosts whose kernel exposes PSI are included.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 40
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max(rate(node_pressure_cpu_waiting_seconds_total{job=~\"$job\",instance=~\"$instance\"}[$__rate_interval]))",
"legendFormat": "CPU",
"refId": "A",
"range": true,
"instant": false
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max(rate(node_pressure_memory_waiting_seconds_total{job=~\"$job\",instance=~\"$instance\"}[$__rate_interval]))",
"legendFormat": "Memory",
"refId": "B",
"range": true,
"instant": false
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max(rate(node_pressure_io_waiting_seconds_total{job=~\"$job\",instance=~\"$instance\"}[$__rate_interval]))",
"legendFormat": "I/O",
"refId": "C",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0,
"max": 1
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 21,
"type": "row",
"title": "Network & Disk",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 48
},
"panels": []
},
{
"id": 22,
"type": "timeseries",
"title": "Network Traffic (in + / out -)",
"description": "Physical interfaces only; loopback, veth, docker, bridge, tap and vmbr interfaces are excluded.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 49
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "sum by (instance) (rate(node_network_receive_bytes_total{job=~\"$job\",instance=~\"$instance\",device!~\"lo|veth.*|docker.*|br-.*|virbr.*|tap.*|fwbr.*|fwpr.*|fwln.*|vmbr.*\"}[$__rate_interval])) * 8",
"legendFormat": "{{instance}} in",
"refId": "A",
"range": true,
"instant": false
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "sum by (instance) (rate(node_network_transmit_bytes_total{job=~\"$job\",instance=~\"$instance\",device!~\"lo|veth.*|docker.*|br-.*|virbr.*|tap.*|fwbr.*|fwpr.*|fwln.*|vmbr.*\"}[$__rate_interval])) * 8",
"legendFormat": "{{instance}} out",
"refId": "B",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "bps",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
}
},
"overrides": [
{
"matcher": {
"id": "byRegexp",
"options": "/ out$/"
},
"properties": [
{
"id": "custom.transform",
"value": "negative-Y"
}
]
}
]
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 23,
"type": "timeseries",
"title": "Disk I/O (read + / write -)",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 49
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "sum by (instance) (rate(node_disk_read_bytes_total{job=~\"$job\",instance=~\"$instance\",device=~\"[a-z]+|nvme[0-9]+n[0-9]+|mmcblk[0-9]+\"}[$__rate_interval]))",
"legendFormat": "{{instance}} read",
"refId": "A",
"range": true,
"instant": false
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "sum by (instance) (rate(node_disk_written_bytes_total{job=~\"$job\",instance=~\"$instance\",device=~\"[a-z]+|nvme[0-9]+n[0-9]+|mmcblk[0-9]+\"}[$__rate_interval]))",
"legendFormat": "{{instance}} write",
"refId": "B",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "Bps",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
}
},
"overrides": [
{
"matcher": {
"id": "byRegexp",
"options": "/ write$/"
},
"properties": [
{
"id": "custom.transform",
"value": "negative-Y"
}
]
}
]
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 24,
"type": "bargauge",
"title": "Root Disk Usage (/)",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 57
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "100 - (node_filesystem_avail_bytes{job=~\"$job\",instance=~\"$instance\",mountpoint=\"/\",fstype!~\"rootfs|tmpfs\"} / node_filesystem_size_bytes{job=~\"$job\",instance=~\"$instance\",mountpoint=\"/\",fstype!~\"rootfs|tmpfs\"} * 100)",
"legendFormat": "{{instance}}",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"decimals": 1,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
},
"color": {
"mode": "thresholds"
},
"max": 100
},
"overrides": []
},
"options": {
"displayMode": "gradient",
"orientation": "horizontal",
"showUnfilled": true,
"valueMode": "color",
"namePlacement": "left",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 25,
"type": "timeseries",
"title": "Network Errors & Dropped Packets",
"description": "Combined receive/transmit errors and drops per second. Anything persistently above zero deserves a look (cabling, duplex, saturation).",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 57
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "sum by (instance) (rate(node_network_receive_errs_total{job=~\"$job\",instance=~\"$instance\",device!~\"lo|veth.*|docker.*|br-.*|virbr.*|tap.*|fwbr.*|fwpr.*|fwln.*|vmbr.*\"}[$__rate_interval]) + rate(node_network_transmit_errs_total{job=~\"$job\",instance=~\"$instance\",device!~\"lo|veth.*|docker.*|br-.*|virbr.*|tap.*|fwbr.*|fwpr.*|fwln.*|vmbr.*\"}[$__rate_interval]) + rate(node_network_receive_drop_total{job=~\"$job\",instance=~\"$instance\",device!~\"lo|veth.*|docker.*|br-.*|virbr.*|tap.*|fwbr.*|fwpr.*|fwln.*|vmbr.*\"}[$__rate_interval]) + rate(node_network_transmit_drop_total{job=~\"$job\",instance=~\"$instance\",device!~\"lo|veth.*|docker.*|br-.*|virbr.*|tap.*|fwbr.*|fwpr.*|fwln.*|vmbr.*\"}[$__rate_interval]))",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "pps",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 26,
"type": "row",
"title": "System Health",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 65
},
"panels": []
},
{
"id": 27,
"type": "timeseries",
"title": "Peak Temperature (CPU / Board)",
"description": "Shown only for hosts that expose hwmon or thermal zone sensors.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 66
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (node_hwmon_temp_celsius{job=~\"$job\",instance=~\"$instance\"}) or max by (instance) (node_thermal_zone_temp{job=~\"$job\",instance=~\"$instance\"})",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "celsius",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 70
},
{
"color": "red",
"value": 85
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 10,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 28,
"type": "timeseries",
"title": "Clock Offset",
"description": "Offset between the system clock and its reference. Large drift breaks TLS, logs and clustering (e.g. Proxmox).",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 66
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "node_timex_offset_seconds{job=~\"$job\",instance=~\"$instance\"}",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 5,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 29,
"type": "timeseries",
"title": "File Descriptor Usage",
"description": "System-wide allocated file descriptors as a percentage of the kernel limit.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 74
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "node_filefd_allocated{job=~\"$job\",instance=~\"$instance\"} / node_filefd_maximum{job=~\"$job\",instance=~\"$instance\"} * 100",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 70
},
{
"color": "red",
"value": 90
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 30,
"type": "timeseries",
"title": "Established TCP Connections",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 74
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "node_netstat_Tcp_CurrEstab{job=~\"$job\",instance=~\"$instance\"}",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
}
]
}Troubleshooting #
| Symptom | Likely cause | First check |
|---|---|---|
Target DOWN in /targets |
Exporter not running, wrong IP/port | docker compose ps, then curl the /metrics URL |
Grafana panel No data |
Data source URL uses localhost |
Change it to http://prometheus:9090 |
| cAdvisor exits immediately | Incomplete mounts | Compare the volumes block with the example |
| pve-exporter restarts endlessly | Missing pve.yml or wrong token |
docker logs pve-exporter |
| YAML error in Prometheus | Typo in a config file | docker logs prometheus, the message names the line |
Wrapping Up #
Done! Every machine, Docker container, and Proxmox node is now tracked from a single Grafana interface, backed by automated alerts that push notifications to your phone via ntfy whenever an issue arises.
If you do not have a home server yet, start with this guide first:
Stuck somewhere, or got a different result than described? Drop it in the comments and we will figure it out together. Cheers 🚀