symon/README.md

# Symon

**Lightweight system metrics exporter for OpenTelemetry**

Symon is a dedicated daemon for collecting and exporting system metrics to OpenTelemetry collectors. Unlike traditional monitoring tools with TUI interfaces, Symon is purpose-built for headless operation, making it ideal for server environments.

## Features

- 🚀 **Lightweight & Fast** - Minimal resource footprint, optimized for server environments
- 📊 **Comprehensive Metrics** - CPU, memory, network, disk, processes, and temperature
- 🔍 **Smart Process Filtering** - Whitelist/blacklist with regex pattern support
- 📁 **External Config Files** - Server-specific process lists via include files
- 🔄 **OTLP Native** - Direct export to OpenTelemetry Collector via gRPC
- ⚙️ **Zero UI Overhead** - No terminal rendering, just pure metrics collection

## Metrics Exported

- **CPU**: Usage per core
- **Memory**: RAM and swap usage/total
- **Network**: RX/TX bytes per interface
- **Disk**: Usage and total space per mount
- **Processes**: Top 10 by CPU/Memory (with optional filtering)
- **Temperature**: System sensors (when available)

All metrics follow OpenTelemetry semantic conventions with proper labels and descriptions.

## Installation

### From Source

```bash
git clone https://github.com/battilo/symon.git
cd symon
cargo build --release
sudo cp target/release/symon /usr/local/bin/
```

### Binary Release

Download the latest release from [GitHub Releases](https://github.com/battilo/symon/releases).

## Quick Start

1. **Create a configuration file** (`symon.toml`):

```toml
collection_interval_secs = 5

[otlp]
endpoint = "http://localhost:4317"
export_interval_secs = 10
service_name = "symon"

[metrics]
cpu = true
memory = true
network = true
disk = true
processes = true
temperature = true
```

2. **Run Symon**:

```bash
symon --config symon.toml
```

3. **View metrics** in your observability stack (Prometheus + Grafana)

## Configuration

### Basic Configuration

```toml
# How often to collect metrics (seconds)
collection_interval_secs = 5

[otlp]
# OpenTelemetry Collector endpoint
endpoint = "http://localhost:4317"

# How often to export metrics (seconds)
export_interval_secs = 10

# Service identification
service_name = "symon"
service_version = "0.1.0"

# Custom resource attributes
[otlp.resource_attributes]
environment = "production"
datacenter = "us-east-1"
host = "server-01"
```

### Metrics Selection

```toml
[metrics]
cpu = true          # CPU usage per core
memory = true       # RAM and swap
network = true      # Network interfaces
disk = true         # Disk usage
processes = true    # Top 10 processes
temperature = true  # System temperatures
```

### Process Filtering

Monitor only specific processes to reduce cardinality:

**Option 1: Inline Configuration**

```toml
[metrics.process_filter]
filter_mode = "whitelist"  # or "blacklist"

# Substring match (case-insensitive)
names = ["nginx", "postgres", "redis"]

# Regex patterns (case-sensitive)
patterns = [
    "^node-v[0-9]+",           # Node.js with version
    "java.*MyApplication",      # Java with specific class
    "^gunicorn: worker \\[.*\\]" # Gunicorn workers
]

# Specific PIDs
pids = [1234, 5678]
```

**Option 2: External File (recommended for multiple servers)**

```toml
[metrics.process_filter]
include = "processes.toml"  # or "/etc/symon/processes.toml"
```

`processes.toml`:
```toml
filter_mode = "whitelist"
names = ["nginx", "postgres"]
patterns = ["^docker-.*"]
```

This allows different process lists per server while keeping the main config identical.

## Docker Compose Testing Stack

A complete observability stack is provided for testing:

```bash
cd docker-compose
docker-compose up -d
```

This starts:
- **OTEL Collector** (ports 4317/4318)
- **Prometheus** (port 9090)
- **Grafana** (port 3000, admin/admin)

Then run Symon:

```bash
symon -c docker-compose/symon-config-example.toml
```

Access Grafana at http://localhost:3000 to view the pre-configured "System Overview" dashboard.

## Systemd Service

Create `/etc/systemd/system/symon.service`:

```ini
[Unit]
Description=Symon - System Metrics Exporter
After=network.target

[Service]
Type=simple
User=symon
ExecStart=/usr/local/bin/symon --config /etc/symon/symon.toml
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
```

Enable and start:

```bash
sudo systemctl enable symon
sudo systemctl start symon
sudo systemctl status symon
```

## Comparison with Bottom

Symon was originally developed as an OpenTelemetry export feature for [bottom](https://github.com/ClementTsang/bottom), but was split into a separate project as it better serves as a dedicated metrics daemon rather than a TUI tool extension.

| Feature | Bottom | Symon |
|---------|--------|-------|
| **Purpose** | Interactive TUI monitor | Headless metrics exporter |
| **UI** | Rich terminal interface | None (daemon only) |
| **Resource Usage** | Higher (rendering) | Minimal (no UI) |
| **OTEL Export** | Optional feature | Core purpose |
| **Process Filtering** | UI-based | Config-based with regex |
| **Use Case** | Developer workstations | Production servers |

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## License

Licensed under either of:

- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE))
- MIT license ([LICENSE-MIT](LICENSE-MIT))

at your option.

## Acknowledgments

- Inspired by [bottom](https://github.com/ClementTsang/bottom) by Clement Tsang
- Built with [OpenTelemetry](https://opentelemetry.io/) and [sysinfo](https://github.com/GuillaumeGomez/sysinfo)