# Validation Module

System for validating Python sensor processing implementation against original MATLAB outputs.

## Overview

This module provides comprehensive tools to compare the outputs of the Python implementation with the original MATLAB code, ensuring numerical equivalence within acceptable tolerance levels.

## Architecture

```
validation/
├── __init__.py          # Module initialization
├── comparator.py        # Core comparison logic and metrics
├── db_extractor.py      # Database query functions
├── validator.py         # High-level validation orchestration
├── cli.py              # Command-line interface
└── README.md           # This file
```

## Components

### 1. DataComparator (`comparator.py`)

Performs statistical and numerical comparisons:

- **Array comparison**: Element-wise differences, RMSE, correlation
- **Scalar comparison**: Single value comparison
- **Record comparison**: Database record matching and comparison
- **Tolerance checking**: Absolute and relative tolerance validation

Key metrics:
- Maximum absolute difference
- Maximum relative difference (as percentage)
- Mean absolute difference
- Root mean square error (RMSE)
- Pearson correlation coefficient

### 2. DataExtractor (`db_extractor.py`)

Extracts processed data from database tables:

- `extract_rsn_data()` - RSN sensor data
- `extract_tilt_data()` - Tilt sensor data (all types)
- `extract_atd_radial_link_data()` - ATD RL data
- `extract_atd_load_link_data()` - ATD LL data
- `extract_atd_pressure_link_data()` - ATD PL data
- `extract_atd_extensometer_3d_data()` - ATD 3DEL data
- `extract_atd_crackmeter_data()` - ATD CrL/2DCrL/3DCrL data
- `extract_atd_pcl_data()` - ATD PCL/PCLHR data
- `extract_atd_tube_link_data()` - ATD TuL data

Each function supports optional date filtering for comparing specific processing runs.

### 3. OutputValidator (`validator.py`)

High-level validation orchestration:

- `validate_rsn()` - Validate RSN sensors
- `validate_tilt()` - Validate Tilt sensors
- `validate_atd_radial_link()` - Validate ATD RL
- `validate_atd_load_link()` - Validate ATD LL
- `validate_atd_pressure_link()` - Validate ATD PL
- `validate_all()` - Validate all available sensors

Returns `ValidationReport` with all comparison results.

### 4. CLI Tool (`cli.py`)

Command-line interface for running validations:

```bash
python -m src.validation.cli <control_unit_id> <chain> [options]
```

See main README.md for usage examples.

## Usage Examples

### Command Line

Basic validation:
```bash
python -m src.validation.cli CU001 A
```

Specific sensor type:
```bash
python -m src.validation.cli CU001 A --type rsn
```

With custom tolerances:
```bash
python -m src.validation.cli CU001 A --abs-tol 1e-8 --rel-tol 1e-6
```

Save report to file:
```bash
python -m src.validation.cli CU001 A --output report.txt
```

### Programmatic Usage

```python
from src.common.database import DatabaseConfig, DatabaseConnection
from src.validation.validator import OutputValidator

# Connect to database
db_config = DatabaseConfig()
with DatabaseConnection(db_config) as conn:
    # Create validator
    validator = OutputValidator(conn)

    # Run validation
    report = validator.validate_rsn('CU001', 'A')

    # Check results
    if report.is_valid():
        print("✓ Validation passed")
    else:
        print("✗ Validation failed")

    # Generate report
    print(report.generate_report())

    # Save to file
    report.save_report('validation_report.txt')
```

## Comparison Workflow

### Standard Workflow

1. **MATLAB processes data** → writes to database
2. **Python processes same data** → writes to database
3. **Validation compares** both outputs from database

```
Raw Data → MATLAB → DB Table ─┐
                              ├→ Validation → Report
Raw Data → Python → DB Table ─┘
```

### With Timestamps

If MATLAB and Python run at different times:

```bash
# MATLAB ran on 2025-10-12
# Python ran on 2025-10-13
python -m src.validation.cli CU001 A \
    --matlab-date 2025-10-12 \
    --python-date 2025-10-13
```

## Tolerance Levels

### Default Tolerances

```python
abs_tol = 1e-6       # Absolute tolerance (0.000001)
rel_tol = 1e-4       # Relative tolerance (0.01%)
max_rel_tol = 0.01   # Max acceptable (1%)
```

### Classification

- **IDENTICAL**: Exact match (all bits equal)
- **EQUIVALENT**: Within tolerance (passes validation)
- **DIFFERENT**: Exceeds tolerance (fails validation)

### Adjusting Tolerances

For stricter validation:
```bash
python -m src.validation.cli CU001 A --abs-tol 1e-10 --rel-tol 1e-8
```

For more lenient validation:
```bash
python -m src.validation.cli CU001 A --abs-tol 1e-4 --rel-tol 1e-2 --max-rel-tol 0.05
```

## Report Format

### Summary Section

```
SUMMARY:
  ✓ Identical:  2    # Exact matches
  ✓ Equivalent: 8    # Within tolerance
  ✗ Different:  0    # Exceeds tolerance
  ? Missing (MATLAB): 0
  ? Missing (Python): 0
  ! Errors: 0
```

### Detailed Results

For each field:
```
✓ X: EQUIVALENT (within tolerance)
  Max abs diff: 3.45e-07    # Largest absolute error
  Max rel diff: 0.0023%     # Largest relative error
  RMSE: 1.12e-07            # Root mean square error
  Correlation: 0.999998     # Pearson correlation
```

## Troubleshooting

### No data found

**Problem**: "No MATLAB data found" or "No Python data found"

**Solutions**:
1. Check that both MATLAB and Python have processed the data
2. Verify control unit ID and chain identifier
3. Use `--matlab-date` and `--python-date` if needed
4. Check database connection

### Record count mismatch

**Problem**: Different number of records in MATLAB vs Python

**Causes**:
- Different time ranges processed
- One implementation filtered more invalid data
- Database write errors in one implementation

**Solution**: Review logs from both implementations

### High differences

**Problem**: Validation fails with large differences

**Causes**:
- Algorithm implementation differences
- Calibration data mismatch
- Floating-point precision issues
- Bug in Python implementation

**Solution**:
1. Check calibration files are identical
2. Review Python implementation against MATLAB code
3. Add debug logging to compare intermediate values
4. Test with simpler/smaller datasets first

## Extending Validation

To add validation for new sensor types:

1. **Add extractor function** in `db_extractor.py`:
```python
def extract_new_sensor_data(self, control_unit_id, chain, ...):
    query = "SELECT ... FROM NEW_TABLE WHERE ..."
    return self.conn.execute_query(query, params)
```

2. **Add validator function** in `validator.py`:
```python
def validate_new_sensor(self, control_unit_id, chain, ...):
    matlab_data = self.extractor.extract_new_sensor_data(...)
    python_data = self.extractor.extract_new_sensor_data(...)
    results = self.comparator.compare_records(...)
    self.report.add_results(results)
    return self.report
```

3. **Add CLI option** in `cli.py`:
```python
parser.add_argument('--type', choices=[..., 'new-sensor'])
# Add corresponding elif branch
```

## Best Practices

1. **Always validate after migration**: Run validation on representative datasets
2. **Use version control**: Track validation reports over time
3. **Document differences**: If intentional differences exist, document why
4. **Automate validation**: Include in CI/CD pipeline
5. **Test edge cases**: Validate with extreme values, missing data, errors
6. **Compare intermediate values**: If final results differ, compare each pipeline stage

## Performance

- **Single sensor validation**: ~1-5 seconds
- **All sensors validation**: ~10-30 seconds
- **Memory usage**: O(n) where n = number of records

For large datasets, use date filtering to validate in chunks.
