196 lines
4.2 KiB
Markdown
196 lines
4.2 KiB
Markdown
# Bottom OpenTelemetry Docker Compose Setup
|
|
|
|
This directory contains a Docker Compose setup for running an observability stack to monitor Bottom with OpenTelemetry.
|
|
|
|
## Architecture
|
|
|
|
The stack includes:
|
|
|
|
1. **OpenTelemetry Collector** - Receives metrics from Bottom via OTLP protocol
|
|
2. **Prometheus** - Scrapes and stores metrics from the OTEL Collector
|
|
3. **Grafana** - Visualizes metrics from Prometheus
|
|
|
|
```
|
|
Bottom (with --headless flag)
|
|
↓ (OTLP/gRPC on port 4317)
|
|
OpenTelemetry Collector
|
|
↓ (Prometheus scrape on port 8889)
|
|
Prometheus
|
|
↓ (Query on port 9090)
|
|
Grafana (accessible on port 3000)
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
### 1. Start the observability stack
|
|
|
|
```bash
|
|
cd docker-compose
|
|
docker-compose up -d
|
|
```
|
|
|
|
This will start:
|
|
- OpenTelemetry Collector on ports 4317 (gRPC), 4318 (HTTP), 8889 (metrics)
|
|
- Prometheus on port 9090
|
|
- Grafana on port 3000
|
|
|
|
### 2. Build Bottom with OpenTelemetry support
|
|
|
|
```bash
|
|
cd ..
|
|
cargo build --release --features opentelemetry
|
|
```
|
|
|
|
### 3. Create a configuration file
|
|
|
|
Create a `bottom-config.toml` file:
|
|
|
|
```toml
|
|
[opentelemetry]
|
|
enabled = true
|
|
endpoint = "http://localhost:4317"
|
|
service_name = "bottom-system-monitor"
|
|
export_interval_ms = 5000
|
|
|
|
[opentelemetry.metrics]
|
|
cpu = true
|
|
memory = true
|
|
network = true
|
|
disk = true
|
|
processes = true
|
|
temperature = true
|
|
gpu = true
|
|
```
|
|
|
|
### 4. Run Bottom in headless mode
|
|
|
|
```bash
|
|
./target/release/btm --config bottom-config.toml --headless
|
|
```
|
|
|
|
Or without config file:
|
|
|
|
```bash
|
|
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
|
|
./target/release/btm --headless
|
|
```
|
|
|
|
### 5. Access the dashboards
|
|
|
|
- **Prometheus**: http://localhost:9090
|
|
- **Grafana**: http://localhost:3000 (username: `admin`, password: `admin`)
|
|
|
|
## Configuration Files
|
|
|
|
### otel-collector-config.yml
|
|
|
|
Configures the OpenTelemetry Collector to:
|
|
- Receive OTLP data on ports 4317 (gRPC) and 4318 (HTTP)
|
|
- Export metrics in Prometheus format on port 9090
|
|
- Debug log all received data
|
|
|
|
### prometheus.yml
|
|
|
|
Configures Prometheus to:
|
|
- Scrape metrics from the OTEL Collector every 10 seconds
|
|
- Load alerting rules from `rules/bottom_rules.yml`
|
|
|
|
### rules/bottom_rules.yml
|
|
|
|
Contains Prometheus recording rules for Bottom metrics, including:
|
|
- Recent process CPU usage metrics
|
|
- Recent process memory usage metrics
|
|
|
|
## Viewing Metrics in Prometheus
|
|
|
|
1. Go to http://localhost:9090
|
|
2. Click on "Graph"
|
|
3. Try these example queries:
|
|
|
|
```promql
|
|
# CPU usage by core
|
|
system_cpu_usage_percent
|
|
|
|
# Memory usage
|
|
system_memory_usage_bytes
|
|
|
|
# Network RX/TX
|
|
system_network_rx_bytes
|
|
system_network_tx_bytes
|
|
|
|
# Disk usage
|
|
system_disk_usage_bytes
|
|
|
|
# Top processes by CPU
|
|
topk(10, system_process_cpu_usage_percent)
|
|
|
|
# Top processes by memory
|
|
topk(10, system_process_memory_usage_bytes)
|
|
```
|
|
|
|
## Grafana Configuration
|
|
|
|
Grafana is automatically configured with:
|
|
- **Prometheus data source** (http://prometheus:9090) - pre-configured
|
|
- **Bottom System Overview dashboard** - pre-loaded
|
|
|
|
To access:
|
|
1. Go to http://localhost:3000 (username: `admin`, password: `admin`)
|
|
2. Navigate to Dashboards → Browse → "Bottom System Overview"
|
|
|
|
The dashboard includes:
|
|
- CPU usage by core
|
|
- Memory usage (RAM/Swap)
|
|
- Network traffic
|
|
- Disk usage
|
|
- Top 10 processes by CPU
|
|
- Top 10 processes by Memory
|
|
|
|
## Stopping the Stack
|
|
|
|
```bash
|
|
docker-compose down
|
|
```
|
|
|
|
To also remove volumes:
|
|
|
|
```bash
|
|
docker-compose down -v
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Bottom not sending metrics
|
|
|
|
Check the OTEL Collector logs:
|
|
```bash
|
|
docker-compose logs -f otel-collector
|
|
```
|
|
|
|
You should see messages about receiving metrics.
|
|
|
|
### Prometheus not scraping
|
|
|
|
1. Check Prometheus targets at http://localhost:9090/targets
|
|
2. The `otel-collector` target should be UP
|
|
|
|
### No data in Grafana
|
|
|
|
1. Verify Prometheus data source is configured correctly
|
|
2. Check that Prometheus has data by querying directly
|
|
3. Ensure your time range in Grafana includes when Bottom was running
|
|
|
|
## Advanced Configuration
|
|
|
|
### Using with TimescaleDB (optional)
|
|
|
|
A TimescaleDB configuration file is available as `docker-compose-timescale.yml.ko` for long-term storage of metrics. Rename it to include it in your stack.
|
|
|
|
### Custom Prometheus Rules
|
|
|
|
Edit `rules/bottom_rules.yml` to add custom recording or alerting rules.
|
|
|
|
### OTEL Collector Sampling
|
|
|
|
Edit `otel-collector-config.yml` to adjust the batch processor settings for different performance characteristics.
|