Pernah nggak sih rasanya buka htop satu per satu cuma buat cari tahu kenapa server lemot? Dulu saya begitu. Tiap ada yang aneh, masuk SSH ke mesin yang salah dulu, keluar, coba mesin satunya lagi. Repot banget.
Di artikel (dan video) ini kita rakit monitoring stack: Prometheus yang ngumpulin metrik dari semua mesin, Grafana buat nampilin grafiknya, plus Alertmanager buat ngasih kabar kalau ada yang nggak normal, misalnya disk mepet. Semua komponennya jalan sebagai container Docker di satu mesin, sedangkan mesin yang dipantau cukup pasang satu exporter ringan.
Bonusnya: kita bisa narik metrik langsung dari API Proxmox VE pakai PVE Exporter, jadi VM dan node Proxmox kebaca semua dari satu dashboard.
Versi yang saya pakai: Prometheus v3.15.1 dan Grafana OSS (latest). Nanti kalau ada versi lebih baru, tinggal ganti angkanya di compose.
Yang Akan Kita Rakit #
Enam container di satu jaringan Docker:
| Container | Tugas | Port |
|---|---|---|
| prometheus | Ngumpulin dan nyimpen metrik | 9090 |
| grafana | Dashboard dan grafik | 3000 |
| node-exporter | Metrik hardware dan OS | 9100 |
| cadvisor | Metrik per-container Docker | 8080 |
| alertmanager | Kirim notifikasi alert | 9093 |
| pve-exporter | Metrik Proxmox VE via API | 9221 |
Cara kerjanya simple. Prometheus modelnya pull: tiap 15 detik dia mampir ke URL /metrics di tiap target, lalu nyimpen angka yang dia temuin. Exporter itu program kecil yang tugasnya nyediain angka tersebut. Grafana nggak nyimpen apa pun, dia cuma nampilin hasil query ke Prometheus.
Persiapan #
- Satu mesin Linux dengan Docker dan Docker Compose plugin v2 terpasang. 2 core CPU dan 4 GB RAM udah cukup enak.
- Akses SSH ke mesin itu, dan ke mesin lain yang mau dipantau.
- Minimal satu node Proxmox VE kalau mau ikutin bagian PVE. Cukup lewat web UI, nggak perlu utak-atik CLI Proxmox.
Langkah 1 📁 Struktur Folder #
Buat folder proyek dan subfolder untuk menyimpan file konfigurasi:
mkdir -p monitoring-stack/prometheus monitoring-stack/alertmanager
cd monitoring-stackDua folder ini bakal jadi rumah file konfigurasi kita. Semua konfig sengaja ditaruh di luar container (volume bind mount) biar gampang diedit pakai editor biasa tanpa perlu build ulang image.
Langkah 2 🐳 Buat File Docker Compose #
Buat file compose.yml di root folder proyek (monitoring-stack/compose.yml):
services:
prometheus:
image: prom/prometheus:v3.15.1
container_name: prometheus
volumes:
- ./prometheus:/etc/prometheus
- prometheus-data:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.retention.time=15d"
- "--web.enable-lifecycle"
ports:
- "9090:9090"
restart: unless-stopped
grafana:
image: grafana/grafana-oss:latest
container_name: grafana
volumes:
- grafana-data:/var/lib/grafana
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=changeme_now
restart: unless-stopped
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
pid: host
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
command:
- "--path.procfs=/host/proc"
- "--path.sysfs=/host/sys"
- "--path.rootfs=/rootfs"
ports:
- "9100:9100"
restart: unless-stopped
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
container_name: cadvisor
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
ports:
- "8080:8080"
restart: unless-stopped
alertmanager:
image: prom/alertmanager:latest
container_name: alertmanager
volumes:
- ./alertmanager:/etc/alertmanager
ports:
- "9093:9093"
restart: unless-stopped
pve-exporter:
image: prompve/prometheus-pve-exporter:latest
container_name: pve-exporter
volumes:
- ./pve.yml:/etc/prometheus/pve.yml
ports:
- "9221:9221"
environment:
- PROXMOX_VE_USERNAME=grafana@pam
- PROXMOX_VE_TOKEN_NAME=grafana
- PROXMOX_VE_TOKEN_VALUE=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
restart: unless-stopped
volumes:
prometheus-data:
grafana-data:Yang wajib diganti: GF_SECURITY_ADMIN_PASSWORD dan PROXMOX_VE_TOKEN_VALUE (sesuaikan dengan token yang kita buat di langkah berikutnya). Opsi --web.enable-lifecycle pada Prometheus penting supaya kita bisa me-reload konfigurasi tanpa restart container.
Jangan jalankan compose dulu—kita siapkan semua file konfigurasinya terlebih dahulu agar container tidak error atau bootloop.
Langkah 3 🔑 Konfigurasi Node Proxmox VE & pve.yml
#
Kalau kamu menggunakan Proxmox VE, ada dua hal yang perlu disiapkan di sisi Proxmox sebelum exporter bisa membaca data cluster.
1. Pasang Node Exporter di Tiap Node Proxmox #
SSH ke tiap node Proxmox, lalu jalankan:
apt update
apt install prometheus-node-exporterPaket ini otomatis jalan sebagai service systemd dan menyajikan metrik hardware di port 9100. Paket apt bawaan Debian ini ringan dan tidak memerlukan Docker di sisi Proxmox.
2. Buat User, API Token, dan File pve.yml
#
PVE Exporter membaca data cluster lewat REST API Proxmox, jadi kita butuh token dengan hak akses baca (read-only):
- Masuk ke Web UI Proxmox, buka Datacenter > Permissions > Users, klik Add. Beri nama user
grafana, realm Linux PAM (sehingga menjadigrafana@pam). - Masuk ke Datacenter > Permissions > API Tokens, klik Add. Pilih user
grafana@pam, Token IDgrafana, dan hilangkan centang pada Privilege separation agar token mewarisi izin user. - Salin token Secret (berupa deretan UUID) yang muncul. Simpan baik-baik karena string ini hanya muncul sekali.
- Masuk ke Datacenter > Permissions > Add > User Permission. Pilih path
/(root level), usergrafana@pam, dan role PVEAuditor.
Izin di level path root
/ otomatis berlaku ke seluruh node jika Proxmox kamu berupa cluster. Jika node berdiri sendiri-sendiri, ulangi langkah user & token ini di masing-masing node.
Sekarang kembali ke mesin Docker, buat file pve.yml di folder root proyek (monitoring-stack/pve.yml):
default:
username: grafana@pam
token_name: grafana
token_value: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
verify_ssl: falseGanti token_value dengan UUID yang tadi disalin. Opsi verify_ssl: false diperlukan di homelab karena sertifikat SSL bawaan Proxmox umumnya bertipe self-signed.
Langkah 4 📝 Konfigurasi Prometheus (prometheus/prometheus.yml)
#
Buat file prometheus/prometheus.yml:
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets: ["alertmanager:9093"]
rule_files:
- "alert_rules.yml"
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "node-exporter"
static_configs:
- targets: ["node-exporter:9100", "192.168.100.10:9100", "192.168.100.20:9100", "192.168.100.60:9100"]
- job_name: "cadvisor"
static_configs:
- targets: ["cadvisor:8080"]
- job_name: "pve"
static_configs:
- targets:
- 192.168.100.10 # Proxmox VE node 1
- 192.168.100.20 # Proxmox VE node 2
metrics_path: /pve
params:
module: [default]
cluster: ['1']
node: ['1']
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: pve-exporter:9221 # PVE exporterCatatan konfigurasi:
- Target
node-exporter:9100adalah container dari compose kita sendiri, otomatis memantau host Docker utama. - Target berupa IP (
192.168.100.x:9100) adalah mesin lain atau node Proxmox yang menjalankan node-exporter. Sesuaikan dengan IP di jaringan kamu. - Pada job
pve, teknik relabeling mengirimkan IP target sebagai parameter ke containerpve-exporter:9221. Satu container PVE Exporter dapat melayani seluruh node Proxmox sekaligus.
Langkah 5 🚨 Konfigurasi Alertmanager, ntfy & Aturan Alert #
Kita siapkan sistem peringatan bahaya sebelum stack dinyalakan.
1. File alertmanager/alertmanager.yml
#
Buat file alertmanager/alertmanager.yml:
route:
group_by: ["alertname"]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: ntfy-notifications
receivers:
# Opsi 1: ntfy Publik / Tanpa Password (ntfy.sh atau self-hosted tanpa auth)
- name: ntfy-notifications
webhook_configs:
- url: "https://ntfy.sh/topik-rahasia-kamu"
send_resolved: true
# Opsi 2: ntfy dengan Password / Token (Self-hosted dengan proteksi auth)
# - name: ntfy-notifications
# webhook_configs:
# - url: "https://ntfy.domainkamu.com/topik-alerts"
# send_resolved: true
# # Metode A: Basic Auth (Username & Password)
# http_config:
# basic_auth:
# username: "admin"
# password: "password_rahasia"
# # Metode B: Bearer Token
# # http_config:
# # authorization:
# # type: Bearer
# # credentials: "tk_xxxxxxxxxxxxxxxxxxxxxx"Blok route mengatur pengelompokan alert (group_by), jeda 30 detik sebelum pengiriman pertama, dan interval pengiriman ulang tiap 4 jam untuk alert yang masih aktif. Notifikasi dikirimkan ke ntfy via webhook. Pilih apakah instance ntfy kamu tanpa autentikasi (gunakan nama topik unik yang tidak mudah ditebak) atau menggunakan password/token.
2. File prometheus/alert_rules.yml
#
Buat file aturan alert di prometheus/alert_rules.yml:
groups:
- name: host-alerts
rules:
- alert: HighCPUUsage
expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU usage on {{ $labels.instance }}"
description: "CPU usage has been above 85% for 5 minutes on {{ $labels.instance }}."
- alert: LowDiskSpace
expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
for: 10m
labels:
severity: critical
annotations:
summary: "Low disk space on {{ $labels.instance }}"
description: "Root filesystem free space is below 10% on {{ $labels.instance }}."- Rule
HighCPUUsage: aktif jika CPU rata-rata di atas 85% selama 5 menit berturut-turut. Parameterfor: 5mmencegah alarm palsu akibat lonjakan CPU sesaat. - Rule
LowDiskSpace: memantau sisa ruang partisi root (/), memicu alert kritis jika sisa ruang di bawah 10% selama 10 menit.
Langkah 6 🚀 Jalankan Stack Docker Compose #
Semua konfigurasi (compose.yml, pve.yml, prometheus.yml, alert_rules.yml, dan alertmanager.yml) sudah lengkap di posisinya masing-masing. Sekarang jalankan stack:
docker compose up -d
docker compose psPastikan keenam container berstatus Up. Jika ada container yang berstatus Restarting, biasanya karena ada volume bind mount yang filenya tidak ditemukan (paling sering: lupa membuat file pve.yml).
Langkah 7 ✅ Cek Target dan Rules di Prometheus #
Buka browser dan akses Web UI Prometheus di: http://IP-mesin:9090/targets
Pastikan seluruh target berstatus hijau (UP):
prometheus(localhost:9090)node-exporter(container lokal dan seluruh IP server remote)cadvisor(container lokal)pve(seluruh node Proxmox yang didaftarkan)
Jika ada target berstatus DOWN, periksa apakah IP sudah sesuai, port 9100 tidak terblokir firewall, atau service exporter di mesin target sudah aktif.
Selanjutnya, cek tab Alerts di http://IP-mesin:9090/alerts. Kedua aturan (HighCPUUsage dan LowDiskSpace) harus terdaftar dengan status Inactive (kondisi normal).
Langkah 8 📊 Sambungkan Grafana ke Prometheus #
- Buka browser dan akses Grafana di:
http://IP-mesin:3000 - Login menggunakan user
admindan password yang tadi kamu tentukan dicompose.yml. - Buka menu Connections > Data sources > Add data source, lalu pilih Prometheus.
- Pada kolom Prometheus server URL, masukkan:
http://prometheus:9090 - Klik Save & test.
URL data source harus
http://prometheus:9090, BUKAN localhost. Grafana berjalan di dalam container Docker tersendiri, sehingga localhost di dalam container akan merujuk ke dirinya sendiri. Gunakan nama service Docker-nya.
Langkah 9 📈 Tambahkan Dashboard #
Komunitas Grafana telah menyediakan dashboard siap pakai yang sangat lengkap. Buka menu Dashboards > New > Import, lalu masukkan ID berikut:
- 1860, Node Exporter Full: visualisasi menyeluruh satu mesin host (CPU, memori, disk I/O, network). Pilih mesin melalui dropdown Host.
- 25823, cAdvisor - Containers, Host & Runtime: visualisasi performa per-container Docker secara real-time.
- 10347, Proxmox via Prometheus: ringkasan cluster Proxmox, status node, storage ZFS/LVM, dan penggunaan resource VM/LXC.
🎁 Bonus: Custom Dashboard “Multi Server Overview” #
Kalau kamu memantau beberapa server sekaligus (misal: Docker host utama, beberapa VM, dan node Proxmox), bolak-balik ganti dropdown filter di dashboard standar itu agak merepotkan.
Di sini saya buatkan custom dashboard khusus (Multi Server Overview) untuk melihat seluruh fleet server kamu dalam satu dashboard ringkas:
- Fleet Status: jumlah host online vs offline, alert yang sedang firing, rata-rata CPU/RAM, dan deteksi OOM kills.
- Server Summary Table: tabel satu baris per server yang menampilkan status UP/DOWN, persentase CPU & RAM, penggunaan root disk, Load 1m, Uptime, Cores, sampai versi Kernel. Setiap baris instance bisa langsung diklik untuk loncat ke detail Node Exporter Full.
- Estimasi Disk Penuh: bar gauge proyeksi linear sisa hari sebelum disk root (
/) penuh berdasarkan tren 24 jam terakhir (capacity planning). - Multi-Host Time Series: grafik perbandingan real-time untuk CPU, RAM, Load average per core, Network traffic (in/out), Disk I/O (read/write), suhu hardware, clock drift, dan status koneksi TCP di seluruh mesin.
Cara Pasang:
- Di Grafana, buka menu Dashboards > New > Import.
- Copy seluruh isi kode JSON di bawah, lalu paste ke kolom Import via panel json.
- Klik Load, pilih data source Prometheus kamu, lalu klik Import.
Klik di sini untuk melihat & menyalin kode JSON Dashboard
{
"id": null,
"uid": "multi-server-overview",
"title": "Multi Server Overview",
"description": "Fleet-wide overview of all servers monitored with node_exporter. Click an instance for detailed metrics.",
"tags": [
"node-exporter",
"prometheus",
"multi-server",
"overview"
],
"timezone": "browser",
"editable": true,
"graphTooltip": 1,
"schemaVersion": 39,
"version": 2,
"refresh": "30s",
"time": {
"from": "now-6h",
"to": "now"
},
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"links": [
{
"title": "Node Exporter Full",
"type": "link",
"icon": "dashboard",
"url": "/d/rYdddlPWk/node-exporter-full",
"targetBlank": true,
"includeVars": false,
"keepTime": true,
"tags": [],
"asDropdown": false,
"tooltip": ""
}
],
"templating": {
"list": [
{
"name": "ds_prometheus",
"label": "Datasource",
"type": "datasource",
"query": "prometheus",
"refresh": 1,
"hide": 0,
"includeAll": false,
"multi": false,
"options": [],
"current": {},
"regex": ""
},
{
"name": "job",
"label": "Job",
"type": "query",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"definition": "label_values(node_uname_info, job)",
"query": {
"query": "label_values(node_uname_info, job)",
"refId": "var-job"
},
"refresh": 2,
"sort": 1,
"multi": true,
"includeAll": true,
"allValue": ".*",
"hide": 0,
"options": [],
"regex": "",
"current": {
"selected": true,
"text": [
"node-exporter"
],
"value": [
"node-exporter"
]
}
},
{
"name": "instance",
"label": "Instance",
"type": "query",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"definition": "label_values(node_uname_info{job=~\"$job\"}, instance)",
"query": {
"query": "label_values(node_uname_info{job=~\"$job\"}, instance)",
"refId": "var-instance"
},
"refresh": 2,
"sort": 1,
"multi": true,
"includeAll": true,
"allValue": ".*",
"hide": 0,
"options": [],
"regex": "",
"current": {
"selected": true,
"text": "All",
"value": "$__all"
}
}
]
},
"panels": [
{
"id": 1,
"type": "row",
"title": "Overview",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"panels": []
},
{
"id": 2,
"type": "stat",
"title": "Hosts Online",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 4,
"w": 3,
"x": 0,
"y": 1
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "count(up{job=~\"$job\",instance=~\"$instance\"} == 1) or vector(0)",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "thresholds"
},
"mappings": []
},
"overrides": []
},
"options": {
"colorMode": "background",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 3,
"type": "stat",
"title": "Hosts Offline",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 4,
"w": 3,
"x": 3,
"y": 1
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "count(up{job=~\"$job\",instance=~\"$instance\"} == 0) or vector(0)",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 1
}
]
},
"color": {
"mode": "thresholds"
},
"mappings": []
},
"overrides": []
},
"options": {
"colorMode": "background",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 4,
"type": "stat",
"title": "Firing Alerts",
"description": "Alerts currently firing in Prometheus (all rules, not filtered by host).",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 4,
"w": 3,
"x": 6,
"y": 1
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "count(ALERTS{alertstate=\"firing\"}) or vector(0)",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 1
}
]
},
"color": {
"mode": "thresholds"
},
"mappings": []
},
"overrides": []
},
"options": {
"colorMode": "background",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 5,
"type": "stat",
"title": "Avg CPU",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 4,
"w": 3,
"x": 9,
"y": 1
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "avg(100 - (avg by (instance) (rate(node_cpu_seconds_total{job=~\"$job\",instance=~\"$instance\",mode=\"idle\"}[$__rate_interval])) * 100))",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
},
"color": {
"mode": "thresholds"
},
"mappings": [],
"decimals": 1,
"min": 0,
"max": 100
},
"overrides": []
},
"options": {
"colorMode": "background",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 6,
"type": "stat",
"title": "Avg RAM",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 4,
"w": 3,
"x": 12,
"y": 1
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "avg(100 * (1 - node_memory_MemAvailable_bytes{job=~\"$job\",instance=~\"$instance\"} / node_memory_MemTotal_bytes{job=~\"$job\",instance=~\"$instance\"}))",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
},
"color": {
"mode": "thresholds"
},
"mappings": [],
"decimals": 1,
"min": 0,
"max": 100
},
"overrides": []
},
"options": {
"colorMode": "background",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 7,
"type": "stat",
"title": "Highest Root Disk",
"description": "Highest usage of the root filesystem (/) across all hosts.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 4,
"w": 3,
"x": 15,
"y": 1
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max(100 - (node_filesystem_avail_bytes{job=~\"$job\",instance=~\"$instance\",mountpoint=\"/\",fstype!~\"rootfs|tmpfs\"} / node_filesystem_size_bytes{job=~\"$job\",instance=~\"$instance\",mountpoint=\"/\",fstype!~\"rootfs|tmpfs\"} * 100))",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
},
"color": {
"mode": "thresholds"
},
"mappings": [],
"decimals": 1,
"min": 0,
"max": 100
},
"overrides": []
},
"options": {
"colorMode": "background",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 8,
"type": "stat",
"title": "Total CPU Cores",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 4,
"w": 3,
"x": 18,
"y": 1
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "sum(count by (instance) (count by (instance, cpu) (node_cpu_seconds_total{job=~\"$job\",instance=~\"$instance\",mode=\"idle\"})))",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "thresholds"
},
"mappings": [],
"decimals": 0
},
"overrides": []
},
"options": {
"colorMode": "background",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 9,
"type": "stat",
"title": "OOM Kills",
"description": "Processes killed by the kernel out-of-memory killer during the selected time range.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 4,
"w": 3,
"x": 21,
"y": 1
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "sum(increase(node_vmstat_oom_kill{job=~\"$job\",instance=~\"$instance\"}[$__range])) or vector(0)",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 1
}
]
},
"color": {
"mode": "thresholds"
},
"mappings": [],
"decimals": 0
},
"overrides": []
},
"options": {
"colorMode": "background",
"graphMode": "none",
"justifyMode": "center",
"orientation": "auto",
"textMode": "value",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 10,
"type": "table",
"title": "Server Summary",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"description": "One row per server. Click the instance to open its detailed Node Exporter Full dashboard.",
"gridPos": {
"h": 9,
"w": 24,
"x": 0,
"y": 5
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (up{job=~\"$job\",instance=~\"$instance\"})",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (100 - (avg by (instance) (rate(node_cpu_seconds_total{job=~\"$job\",instance=~\"$instance\",mode=\"idle\"}[$__rate_interval])) * 100))",
"legendFormat": "",
"refId": "B",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (100 * (1 - node_memory_MemAvailable_bytes{job=~\"$job\",instance=~\"$instance\"} / node_memory_MemTotal_bytes{job=~\"$job\",instance=~\"$instance\"}))",
"legendFormat": "",
"refId": "C",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (100 - (node_filesystem_avail_bytes{job=~\"$job\",instance=~\"$instance\",mountpoint=\"/\",fstype!~\"rootfs|tmpfs\"} / node_filesystem_size_bytes{job=~\"$job\",instance=~\"$instance\",mountpoint=\"/\",fstype!~\"rootfs|tmpfs\"} * 100))",
"legendFormat": "",
"refId": "D",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (node_load1{job=~\"$job\",instance=~\"$instance\"})",
"legendFormat": "",
"refId": "E",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (time() - node_boot_time_seconds{job=~\"$job\",instance=~\"$instance\"})",
"legendFormat": "",
"refId": "F",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "count by (instance) (count by (instance, cpu) (node_cpu_seconds_total{job=~\"$job\",instance=~\"$instance\",mode=\"idle\"}))",
"legendFormat": "",
"refId": "G",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance, nodename, release) (node_uname_info{job=~\"$job\",instance=~\"$instance\"})",
"legendFormat": "",
"refId": "H",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (node_memory_MemTotal_bytes{job=~\"$job\",instance=~\"$instance\"})",
"legendFormat": "",
"refId": "I",
"range": false,
"instant": true,
"format": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (node_memory_MemTotal_bytes{job=~\"$job\",instance=~\"$instance\"} - node_memory_MemAvailable_bytes{job=~\"$job\",instance=~\"$instance\"})",
"legendFormat": "",
"refId": "J",
"range": false,
"instant": true,
"format": "table"
}
],
"transformations": [
{
"id": "joinByField",
"options": {
"byField": "instance",
"mode": "outer"
}
},
{
"id": "filterFieldsByName",
"options": {
"include": {
"pattern": "^(nodename|instance|release|Value #[A-GIJ])$"
}
}
},
{
"id": "organize",
"options": {
"excludeByName": {},
"indexByName": {
"nodename": 0,
"instance": 1,
"Value #A": 2,
"Value #B": 3,
"Value #C": 4,
"Value #J": 5,
"Value #I": 6,
"Value #D": 7,
"Value #E": 8,
"Value #G": 9,
"Value #F": 10,
"release": 11
},
"renameByName": {}
}
}
],
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": false
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"mappings": []
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "nodename"
},
"properties": [
{
"id": "displayName",
"value": "Hostname"
},
{
"id": "custom.width",
"value": 160
}
]
},
{
"matcher": {
"id": "byName",
"options": "instance"
},
"properties": [
{
"id": "displayName",
"value": "Instance"
},
{
"id": "links",
"value": [
{
"title": "Open in Node Exporter Full",
"url": "/d/rYdddlPWk/node-exporter-full?${job:queryparam}&var-nodename=${__data.fields.nodename}&var-node=${__data.fields.instance}",
"targetBlank": true
}
]
},
{
"id": "custom.width",
"value": 170
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #A"
},
"properties": [
{
"id": "displayName",
"value": "Status"
},
{
"id": "mappings",
"value": [
{
"type": "value",
"options": {
"1": {
"text": "UP",
"color": "green",
"index": 0
},
"0": {
"text": "DOWN",
"color": "red",
"index": 1
}
}
}
]
},
{
"id": "custom.cellOptions",
"value": {
"type": "color-background"
}
},
{
"id": "custom.width",
"value": 80
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #B"
},
"properties": [
{
"id": "displayName",
"value": "CPU %"
},
{
"id": "min",
"value": 0
},
{
"id": "max",
"value": 100
},
{
"id": "decimals",
"value": 1
},
{
"id": "thresholds",
"value": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
}
},
{
"id": "custom.cellOptions",
"value": {
"type": "gauge",
"mode": "lcd",
"valueDisplayMode": "text"
}
},
{
"id": "color",
"value": {
"mode": "thresholds"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #C"
},
"properties": [
{
"id": "displayName",
"value": "RAM %"
},
{
"id": "min",
"value": 0
},
{
"id": "max",
"value": 100
},
{
"id": "decimals",
"value": 1
},
{
"id": "thresholds",
"value": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
}
},
{
"id": "custom.cellOptions",
"value": {
"type": "gauge",
"mode": "lcd",
"valueDisplayMode": "text"
}
},
{
"id": "color",
"value": {
"mode": "thresholds"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #J"
},
"properties": [
{
"id": "displayName",
"value": "RAM Used"
},
{
"id": "unit",
"value": "bytes"
},
{
"id": "decimals",
"value": 1
},
{
"id": "custom.width",
"value": 110
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #I"
},
"properties": [
{
"id": "displayName",
"value": "RAM Total"
},
{
"id": "unit",
"value": "bytes"
},
{
"id": "decimals",
"value": 1
},
{
"id": "custom.width",
"value": 100
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #D"
},
"properties": [
{
"id": "displayName",
"value": "Disk /"
},
{
"id": "min",
"value": 0
},
{
"id": "max",
"value": 100
},
{
"id": "decimals",
"value": 1
},
{
"id": "thresholds",
"value": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
}
},
{
"id": "custom.cellOptions",
"value": {
"type": "gauge",
"mode": "lcd",
"valueDisplayMode": "text"
}
},
{
"id": "color",
"value": {
"mode": "thresholds"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #E"
},
"properties": [
{
"id": "displayName",
"value": "Load 1m"
},
{
"id": "decimals",
"value": 2
},
{
"id": "custom.width",
"value": 90
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #F"
},
"properties": [
{
"id": "displayName",
"value": "Uptime"
},
{
"id": "unit",
"value": "s"
},
{
"id": "custom.width",
"value": 130
}
]
},
{
"matcher": {
"id": "byName",
"options": "Value #G"
},
"properties": [
{
"id": "displayName",
"value": "Cores"
},
{
"id": "unit",
"value": "short"
},
{
"id": "decimals",
"value": 0
},
{
"id": "custom.width",
"value": 70
}
]
},
{
"matcher": {
"id": "byName",
"options": "release"
},
"properties": [
{
"id": "displayName",
"value": "Kernel"
}
]
}
]
},
"options": {
"showHeader": true,
"cellHeight": "sm",
"footer": {
"show": false
},
"sortBy": [
{
"displayName": "Hostname",
"desc": false
}
]
}
},
{
"id": 11,
"type": "row",
"title": "Alerts & Capacity",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 14
},
"panels": []
},
{
"id": 12,
"type": "table",
"title": "Firing Alerts",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"description": "Alerts currently firing in Prometheus, taken from the built-in ALERTS metric.",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 15
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "ALERTS{alertstate=\"firing\"}",
"legendFormat": "",
"refId": "A",
"range": false,
"instant": true,
"format": "table"
}
],
"transformations": [
{
"id": "filterFieldsByName",
"options": {
"include": {
"pattern": "^(alertname|severity|instance|job)$"
}
}
},
{
"id": "organize",
"options": {
"excludeByName": {},
"renameByName": {},
"indexByName": {
"alertname": 0,
"severity": 1,
"instance": 2,
"job": 3
}
}
}
],
"fieldConfig": {
"defaults": {
"noValue": "No firing alerts",
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
}
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"mappings": []
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "alertname"
},
"properties": [
{
"id": "displayName",
"value": "Alert"
}
]
},
{
"matcher": {
"id": "byName",
"options": "severity"
},
"properties": [
{
"id": "displayName",
"value": "Severity"
},
{
"id": "mappings",
"value": [
{
"type": "value",
"options": {
"critical": {
"text": "critical",
"color": "red",
"index": 0
},
"warning": {
"text": "warning",
"color": "orange",
"index": 1
}
}
}
]
},
{
"id": "custom.cellOptions",
"value": {
"type": "color-text"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "instance"
},
"properties": [
{
"id": "displayName",
"value": "Instance"
}
]
},
{
"matcher": {
"id": "byName",
"options": "job"
},
"properties": [
{
"id": "displayName",
"value": "Job"
}
]
}
]
},
"options": {
"showHeader": true,
"cellHeight": "sm",
"footer": {
"show": false
}
}
},
{
"id": 13,
"type": "bargauge",
"title": "Estimated Days Until Root Disk Is Full",
"description": "Linear projection from the last 24 hours of free space on /. Hosts whose free space is stable or growing are not listed.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 15
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) ((node_filesystem_avail_bytes{job=~\"$job\",instance=~\"$instance\",mountpoint=\"/\",fstype!~\"rootfs|tmpfs\"} / (-deriv(node_filesystem_avail_bytes{job=~\"$job\",instance=~\"$instance\",mountpoint=\"/\",fstype!~\"rootfs|tmpfs\"}[24h]) > 0)) / 86400)",
"legendFormat": "{{instance}}",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "d",
"min": 0,
"decimals": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "red",
"value": null
},
{
"color": "orange",
"value": 7
},
{
"color": "green",
"value": 30
}
]
},
"color": {
"mode": "thresholds"
},
"max": 90,
"noValue": "No shrinking trend"
},
"overrides": []
},
"options": {
"displayMode": "gradient",
"orientation": "horizontal",
"showUnfilled": true,
"valueMode": "color",
"namePlacement": "left",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 14,
"type": "row",
"title": "CPU & Memory",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 23
},
"panels": []
},
{
"id": 15,
"type": "timeseries",
"title": "CPU Usage per Server",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 24
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "100 - (avg by (instance) (rate(node_cpu_seconds_total{job=~\"$job\",instance=~\"$instance\",mode=\"idle\"}[$__rate_interval])) * 100)",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0,
"max": 100
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 16,
"type": "timeseries",
"title": "RAM Usage per Server",
"description": "Calculated from MemAvailable, so page cache is not counted as used.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 24
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "100 * (1 - node_memory_MemAvailable_bytes{job=~\"$job\",instance=~\"$instance\"} / node_memory_MemTotal_bytes{job=~\"$job\",instance=~\"$instance\"})",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0,
"max": 100
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 17,
"type": "timeseries",
"title": "Load Average per Core (1m)",
"description": "Values above 1 mean more runnable tasks than CPU cores.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 32
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "node_load1{job=~\"$job\",instance=~\"$instance\"} / on(instance) (count by (instance) (count by (instance, cpu) (node_cpu_seconds_total{job=~\"$job\",instance=~\"$instance\",mode=\"idle\"})))",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 18,
"type": "timeseries",
"title": "Swap Usage",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 32
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "(node_memory_SwapTotal_bytes{job=~\"$job\",instance=~\"$instance\"} - node_memory_SwapFree_bytes{job=~\"$job\",instance=~\"$instance\"}) / (node_memory_SwapTotal_bytes{job=~\"$job\",instance=~\"$instance\"} > 0) * 100",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0,
"max": 100
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 19,
"type": "timeseries",
"title": "RAM Used vs Total (bytes)",
"description": "Dashed lines show the installed RAM capacity of each server.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 40
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "node_memory_MemTotal_bytes{job=~\"$job\",instance=~\"$instance\"} - node_memory_MemAvailable_bytes{job=~\"$job\",instance=~\"$instance\"}",
"legendFormat": "{{instance}} used",
"refId": "A",
"range": true,
"instant": false
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "node_memory_MemTotal_bytes{job=~\"$job\",instance=~\"$instance\"}",
"legendFormat": "{{instance}} total",
"refId": "B",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "bytes",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 10,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0
},
"overrides": [
{
"matcher": {
"id": "byRegexp",
"options": "/ total$/"
},
"properties": [
{
"id": "custom.fillOpacity",
"value": 0
},
{
"id": "custom.lineStyle",
"value": {
"fill": "dash",
"dash": [
10,
10
]
}
},
{
"id": "custom.lineWidth",
"value": 2
}
]
}
]
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 20,
"type": "timeseries",
"title": "Resource Pressure (PSI, worst host)",
"description": "Share of time tasks were stalled waiting for a resource (Linux PSI). Only hosts whose kernel exposes PSI are included.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 40
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max(rate(node_pressure_cpu_waiting_seconds_total{job=~\"$job\",instance=~\"$instance\"}[$__rate_interval]))",
"legendFormat": "CPU",
"refId": "A",
"range": true,
"instant": false
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max(rate(node_pressure_memory_waiting_seconds_total{job=~\"$job\",instance=~\"$instance\"}[$__rate_interval]))",
"legendFormat": "Memory",
"refId": "B",
"range": true,
"instant": false
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max(rate(node_pressure_io_waiting_seconds_total{job=~\"$job\",instance=~\"$instance\"}[$__rate_interval]))",
"legendFormat": "I/O",
"refId": "C",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0,
"max": 1
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 21,
"type": "row",
"title": "Network & Disk",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 48
},
"panels": []
},
{
"id": 22,
"type": "timeseries",
"title": "Network Traffic (in + / out -)",
"description": "Physical interfaces only; loopback, veth, docker, bridge, tap and vmbr interfaces are excluded.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 49
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "sum by (instance) (rate(node_network_receive_bytes_total{job=~\"$job\",instance=~\"$instance\",device!~\"lo|veth.*|docker.*|br-.*|virbr.*|tap.*|fwbr.*|fwpr.*|fwln.*|vmbr.*\"}[$__rate_interval])) * 8",
"legendFormat": "{{instance}} in",
"refId": "A",
"range": true,
"instant": false
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "sum by (instance) (rate(node_network_transmit_bytes_total{job=~\"$job\",instance=~\"$instance\",device!~\"lo|veth.*|docker.*|br-.*|virbr.*|tap.*|fwbr.*|fwpr.*|fwln.*|vmbr.*\"}[$__rate_interval])) * 8",
"legendFormat": "{{instance}} out",
"refId": "B",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "bps",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
}
},
"overrides": [
{
"matcher": {
"id": "byRegexp",
"options": "/ out$/"
},
"properties": [
{
"id": "custom.transform",
"value": "negative-Y"
}
]
}
]
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 23,
"type": "timeseries",
"title": "Disk I/O (read + / write -)",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 49
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "sum by (instance) (rate(node_disk_read_bytes_total{job=~\"$job\",instance=~\"$instance\",device=~\"[a-z]+|nvme[0-9]+n[0-9]+|mmcblk[0-9]+\"}[$__rate_interval]))",
"legendFormat": "{{instance}} read",
"refId": "A",
"range": true,
"instant": false
},
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "sum by (instance) (rate(node_disk_written_bytes_total{job=~\"$job\",instance=~\"$instance\",device=~\"[a-z]+|nvme[0-9]+n[0-9]+|mmcblk[0-9]+\"}[$__rate_interval]))",
"legendFormat": "{{instance}} write",
"refId": "B",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "Bps",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
}
},
"overrides": [
{
"matcher": {
"id": "byRegexp",
"options": "/ write$/"
},
"properties": [
{
"id": "custom.transform",
"value": "negative-Y"
}
]
}
]
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 24,
"type": "bargauge",
"title": "Root Disk Usage (/)",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 57
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "100 - (node_filesystem_avail_bytes{job=~\"$job\",instance=~\"$instance\",mountpoint=\"/\",fstype!~\"rootfs|tmpfs\"} / node_filesystem_size_bytes{job=~\"$job\",instance=~\"$instance\",mountpoint=\"/\",fstype!~\"rootfs|tmpfs\"} * 100)",
"legendFormat": "{{instance}}",
"refId": "A",
"range": false,
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"decimals": 1,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 80
},
{
"color": "red",
"value": 90
}
]
},
"color": {
"mode": "thresholds"
},
"max": 100
},
"overrides": []
},
"options": {
"displayMode": "gradient",
"orientation": "horizontal",
"showUnfilled": true,
"valueMode": "color",
"namePlacement": "left",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
}
},
{
"id": 25,
"type": "timeseries",
"title": "Network Errors & Dropped Packets",
"description": "Combined receive/transmit errors and drops per second. Anything persistently above zero deserves a look (cabling, duplex, saturation).",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 57
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "sum by (instance) (rate(node_network_receive_errs_total{job=~\"$job\",instance=~\"$instance\",device!~\"lo|veth.*|docker.*|br-.*|virbr.*|tap.*|fwbr.*|fwpr.*|fwln.*|vmbr.*\"}[$__rate_interval]) + rate(node_network_transmit_errs_total{job=~\"$job\",instance=~\"$instance\",device!~\"lo|veth.*|docker.*|br-.*|virbr.*|tap.*|fwbr.*|fwpr.*|fwln.*|vmbr.*\"}[$__rate_interval]) + rate(node_network_receive_drop_total{job=~\"$job\",instance=~\"$instance\",device!~\"lo|veth.*|docker.*|br-.*|virbr.*|tap.*|fwbr.*|fwpr.*|fwln.*|vmbr.*\"}[$__rate_interval]) + rate(node_network_transmit_drop_total{job=~\"$job\",instance=~\"$instance\",device!~\"lo|veth.*|docker.*|br-.*|virbr.*|tap.*|fwbr.*|fwpr.*|fwln.*|vmbr.*\"}[$__rate_interval]))",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "pps",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 26,
"type": "row",
"title": "System Health",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 65
},
"panels": []
},
{
"id": 27,
"type": "timeseries",
"title": "Peak Temperature (CPU / Board)",
"description": "Shown only for hosts that expose hwmon or thermal zone sensors.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 66
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "max by (instance) (node_hwmon_temp_celsius{job=~\"$job\",instance=~\"$instance\"}) or max by (instance) (node_thermal_zone_temp{job=~\"$job\",instance=~\"$instance\"})",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "celsius",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 70
},
{
"color": "red",
"value": 85
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 10,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 28,
"type": "timeseries",
"title": "Clock Offset",
"description": "Offset between the system clock and its reference. Large drift breaks TLS, logs and clustering (e.g. Proxmox).",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 66
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "node_timex_offset_seconds{job=~\"$job\",instance=~\"$instance\"}",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 5,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 29,
"type": "timeseries",
"title": "File Descriptor Usage",
"description": "System-wide allocated file descriptors as a percentage of the kernel limit.",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 74
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "node_filefd_allocated{job=~\"$job\",instance=~\"$instance\"} / node_filefd_maximum{job=~\"$job\",instance=~\"$instance\"} * 100",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 70
},
{
"color": "red",
"value": 90
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 30,
"type": "timeseries",
"title": "Established TCP Connections",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 74
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${ds_prometheus}"
},
"editorMode": "code",
"expr": "node_netstat_Tcp_CurrEstab{job=~\"$job\",instance=~\"$instance\"}",
"legendFormat": "{{instance}}",
"refId": "A",
"range": true,
"instant": false
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"lineInterpolation": "smooth",
"lineWidth": 1,
"fillOpacity": 15,
"showPoints": "never",
"spanNulls": false,
"axisPlacement": "auto",
"stacking": {
"mode": "none",
"group": "A"
},
"thresholdsStyle": {
"mode": "off"
}
},
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "bottom",
"showLegend": true,
"calcs": [
"mean",
"max",
"lastNotNull"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
}
]
}Kalau Ada Masalah #
| Gejala | Kemungkinan penyebab | Cek pertama |
|---|---|---|
Target DOWN di /targets |
Exporter nggak jalan atau IP/port salah | docker compose ps, lalu curl ke /metrics |
Panel Grafana No data |
URL data source pakai localhost |
Ganti ke http://prometheus:9090 |
| cAdvisor langsung exit | Mount nggak lengkap | Bandingkan blok volumes sama contoh |
| pve-exporter restart terus | pve.yml nggak ada atau token salah |
docker logs pve-exporter |
| Error YAML di Prometheus | Typo di file konfigurasi | docker logs prometheus, pesannya nyebutin barisnya |
Penutup #
Beres! Sekarang seluruh mesin, container Docker, sampai cluster Proxmox kepantau dari satu dashboard Grafana, lengkap dengan alert otomatis yang siap mengirim notifikasi ke HP via ntfy saat ada masalah.
Kalau belum punya home server sama sekali, mulai dari artikel ini dulu:
Ada yang macet atau hasilnya beda dari tulisan ini? Drop di komen, kita bahas bareng. Salam 🚀