init
This commit is contained in:
227
docker-compose/METRICS.md
Normal file
227
docker-compose/METRICS.md
Normal file
@@ -0,0 +1,227 @@
|
||||
# Bottom OpenTelemetry Metrics Reference
|
||||
|
||||
This document lists all metrics exported by Bottom when running with the `opentelemetry` feature enabled.
|
||||
|
||||
## System Metrics
|
||||
|
||||
### CPU
|
||||
|
||||
| Metric Name | Type | Labels | Description |
|
||||
|------------|------|--------|-------------|
|
||||
| `system_cpu_usage_percent` | Gauge | `cpu_id` | CPU usage percentage per core |
|
||||
|
||||
**Example:**
|
||||
```promql
|
||||
# Average CPU across all cores
|
||||
avg(system_cpu_usage_percent)
|
||||
|
||||
# CPU usage for core 0
|
||||
system_cpu_usage_percent{cpu_id="0"}
|
||||
```
|
||||
|
||||
### Memory
|
||||
|
||||
| Metric Name | Type | Labels | Description |
|
||||
|------------|------|--------|-------------|
|
||||
| `system_memory_usage_bytes` | Gauge | - | RAM memory currently in use |
|
||||
| `system_memory_total_bytes` | Gauge | - | Total RAM memory available |
|
||||
| `system_swap_usage_bytes` | Gauge | - | Swap memory currently in use |
|
||||
| `system_swap_total_bytes` | Gauge | - | Total swap memory available |
|
||||
|
||||
**Example:**
|
||||
```promql
|
||||
# Memory usage percentage
|
||||
(system_memory_usage_bytes / system_memory_total_bytes) * 100
|
||||
|
||||
# Available memory
|
||||
system_memory_total_bytes - system_memory_usage_bytes
|
||||
```
|
||||
|
||||
### Network
|
||||
|
||||
| Metric Name | Type | Labels | Description |
|
||||
|------------|------|--------|-------------|
|
||||
| `system_network_rx_bytes_rate` | Gauge | `interface` | Network receive rate in bytes/sec |
|
||||
| `system_network_tx_bytes_rate` | Gauge | `interface` | Network transmit rate in bytes/sec |
|
||||
|
||||
**Example:**
|
||||
```promql
|
||||
# Total network throughput
|
||||
sum(system_network_rx_bytes_rate) + sum(system_network_tx_bytes_rate)
|
||||
|
||||
# RX rate for specific interface
|
||||
system_network_rx_bytes_rate{interface="eth0"}
|
||||
```
|
||||
|
||||
### Disk
|
||||
|
||||
| Metric Name | Type | Labels | Description |
|
||||
|------------|------|--------|-------------|
|
||||
| `system_disk_usage_bytes` | Gauge | `device`, `mount` | Disk space currently in use |
|
||||
| `system_disk_total_bytes` | Gauge | `device`, `mount` | Total disk space available |
|
||||
|
||||
**Example:**
|
||||
```promql
|
||||
# Disk usage percentage
|
||||
(system_disk_usage_bytes / system_disk_total_bytes) * 100
|
||||
|
||||
# Free disk space
|
||||
system_disk_total_bytes - system_disk_usage_bytes
|
||||
```
|
||||
|
||||
### Temperature
|
||||
|
||||
| Metric Name | Type | Labels | Description |
|
||||
|------------|------|--------|-------------|
|
||||
| `system_temperature_celsius` | Gauge | `sensor` | Temperature readings in Celsius |
|
||||
|
||||
**Example:**
|
||||
```promql
|
||||
# Average temperature across all sensors
|
||||
avg(system_temperature_celsius)
|
||||
|
||||
# Maximum temperature
|
||||
max(system_temperature_celsius)
|
||||
```
|
||||
|
||||
## Process Metrics
|
||||
|
||||
| Metric Name | Type | Labels | Description |
|
||||
|------------|------|--------|-------------|
|
||||
| `system_process_cpu_usage_percent` | Gauge | `name`, `pid` | CPU usage percentage per process |
|
||||
| `system_process_memory_usage_bytes` | Gauge | `name`, `pid` | Memory usage in bytes per process |
|
||||
| `system_process_count` | Gauge | - | Total number of processes |
|
||||
|
||||
**Example:**
|
||||
```promql
|
||||
# Top 10 processes by CPU
|
||||
topk(10, system_process_cpu_usage_percent)
|
||||
|
||||
# Top 10 processes by memory
|
||||
topk(10, system_process_memory_usage_bytes)
|
||||
|
||||
# Total memory used by all Chrome processes
|
||||
sum(system_process_memory_usage_bytes{name=~".*chrome.*"})
|
||||
```
|
||||
|
||||
## Recording Rules
|
||||
|
||||
The following recording rules are pre-configured in Prometheus (see `rules/bottom_rules.yml`):
|
||||
|
||||
| Rule Name | Expression | Description |
|
||||
|-----------|------------|-------------|
|
||||
| `system_process_cpu_usage_percent:recent` | Recent process CPU metrics | Filters out stale process data (>2 min old) |
|
||||
| `system_process_memory_usage_bytes:recent` | Recent process memory metrics | Filters out stale process data (>2 min old) |
|
||||
|
||||
**Example:**
|
||||
```promql
|
||||
# Query only recent process data
|
||||
topk(10, system_process_cpu_usage_percent:recent)
|
||||
```
|
||||
|
||||
## Common Queries
|
||||
|
||||
### System Health
|
||||
|
||||
```promql
|
||||
# Overall system CPU usage
|
||||
avg(system_cpu_usage_percent)
|
||||
|
||||
# Memory pressure (>80% is high)
|
||||
(system_memory_usage_bytes / system_memory_total_bytes) * 100
|
||||
|
||||
# Disk pressure (>90% is critical)
|
||||
(system_disk_usage_bytes / system_disk_total_bytes) * 100
|
||||
```
|
||||
|
||||
### Resource Hogs
|
||||
|
||||
```promql
|
||||
# Top CPU consumers
|
||||
topk(5, system_process_cpu_usage_percent)
|
||||
|
||||
# Top memory consumers
|
||||
topk(5, system_process_memory_usage_bytes)
|
||||
|
||||
# Processes using >1GB memory
|
||||
system_process_memory_usage_bytes > 1073741824
|
||||
```
|
||||
|
||||
### Network Analysis
|
||||
|
||||
```promql
|
||||
# Total network traffic (RX + TX)
|
||||
sum(system_network_rx_bytes_rate) + sum(system_network_tx_bytes_rate)
|
||||
|
||||
# Network traffic by interface
|
||||
sum by (interface) (system_network_rx_bytes_rate + system_network_tx_bytes_rate)
|
||||
|
||||
# Interfaces with high RX rate (>10MB/s)
|
||||
system_network_rx_bytes_rate > 10485760
|
||||
```
|
||||
|
||||
## Alerting Examples
|
||||
|
||||
### Sample Prometheus Alert Rules
|
||||
|
||||
```yaml
|
||||
groups:
|
||||
- name: bottom_alerts
|
||||
interval: 30s
|
||||
rules:
|
||||
- alert: HighCPUUsage
|
||||
expr: avg(system_cpu_usage_percent) > 80
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "High CPU usage detected"
|
||||
description: "Average CPU usage is {{ $value }}%"
|
||||
|
||||
- alert: HighMemoryUsage
|
||||
expr: (system_memory_usage_bytes / system_memory_total_bytes) * 100 > 90
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "High memory usage detected"
|
||||
description: "Memory usage is {{ $value }}%"
|
||||
|
||||
- alert: DiskAlmostFull
|
||||
expr: (system_disk_usage_bytes / system_disk_total_bytes) * 100 > 90
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Disk {{ $labels.mount }} almost full"
|
||||
description: "Disk usage is {{ $value }}% on {{ $labels.mount }}"
|
||||
```
|
||||
|
||||
## Label Reference
|
||||
|
||||
| Label | Used In | Description |
|
||||
|-------|---------|-------------|
|
||||
| `cpu_id` | CPU metrics | CPU core identifier (0, 1, 2, ...) |
|
||||
| `interface` | Network metrics | Network interface name (eth0, wlan0, ...) |
|
||||
| `device` | Disk metrics | Device name (/dev/sda1, ...) |
|
||||
| `mount` | Disk metrics | Mount point (/, /home, ...) |
|
||||
| `sensor` | Temperature | Temperature sensor name |
|
||||
| `name` | Process metrics | Process name |
|
||||
| `pid` | Process metrics | Process ID |
|
||||
| `exported_job` | All | Always "bottom-system-monitor" |
|
||||
| `otel_scope_name` | All | Always "bottom-system-monitor" |
|
||||
|
||||
## Data Retention
|
||||
|
||||
By default, Prometheus stores metrics for 15 days. You can adjust this in the Prometheus configuration:
|
||||
|
||||
```yaml
|
||||
# In prometheus.yml
|
||||
global:
|
||||
retention_time: 30d # Keep data for 30 days
|
||||
```
|
||||
|
||||
For long-term storage, consider using:
|
||||
- **TimescaleDB** (see `docker-compose-timescale.yml.ko`)
|
||||
- **Thanos** for multi-cluster metrics
|
||||
- **Cortex** for horizontally scalable storage
|
||||
Reference in New Issue
Block a user