# Changelog

## [Current] - 2025-12-30

### Added
- **Consolidation-based incremental migration**: Uses consolidation keys `(UnitName, ToolNameID, EventDate, EventTime)` instead of timestamps
- **MySQL ID optimization**: Uses `MAX(mysql_max_id)` from PostgreSQL to filter MySQL queries, avoiding full table scans
- **State management in PostgreSQL**: Replaced JSON file with `migration_state` table for more reliable tracking
- **Sync utility**: Added `scripts/sync_migration_state.py` to sync state with actual data
- **Performance optimization**: MySQL queries now instant using PRIMARY KEY filter
- **Better documentation**: Consolidated and updated all documentation files

### Changed
- **Incremental migration**: Now uses consolidation keys instead of timestamp-based approach
- **Full migration**: Improved to save global `last_key` after completing all partitions
- **State tracking**: Moved from `migration_state.json` to PostgreSQL table `migration_state`
- **Query performance**: Added `min_mysql_id` parameter to `fetch_consolidation_keys_after()` for optimization
- **Configuration**: Renamed `BATCH_SIZE` to `CONSOLIDATION_GROUP_LIMIT` to better reflect what it controls
- **Configuration**: Added `PROGRESS_LOG_INTERVAL` to control logging frequency
- **Configuration**: Added `BENCHMARK_OUTPUT_DIR` to specify benchmark results directory
- **Documentation**: Updated README.md, MIGRATION_WORKFLOW.md, QUICKSTART.md, EXAMPLE_WORKFLOW.md with current implementation

### Removed
- **migration_state.json**: Replaced by PostgreSQL table
- **Timestamp-based migration**: Replaced by consolidation key-based approach
- **ID-based resumable migration**: Consolidated into single consolidation-based approach
- **Temporary debug scripts**: Cleaned up all `/tmp/` debug files

### Fixed
- **Incremental migration performance**: MySQL queries now ~1000x faster with ID filter
- **State synchronization**: Can now sync `migration_state` with actual data using utility script
- **Duplicate handling**: Uses `ON CONFLICT DO NOTHING` to prevent duplicates
- **Last key tracking**: Properly updates global state after full migration

### Migration Guide (from old to new)

If you have an existing installation with `migration_state.json`:

1. **Backup your data** (optional but recommended):
   ```bash
   cp migration_state.json migration_state.json.backup
   ```

2. **Run full migration** to populate `migration_state` table:
   ```bash
   python main.py migrate full
   ```

3. **Sync state** (if you have existing data):
   ```bash
   python scripts/sync_migration_state.py
   ```

4. **Remove old state file**:
   ```bash
   rm migration_state.json
   ```

5. **Run incremental migration**:
   ```bash
   python main.py migrate incremental --dry-run
   python main.py migrate incremental
   ```

### Performance Improvements

- **MySQL query time**: From 60+ seconds to <0.1 seconds (600x faster)
- **Consolidation efficiency**: Multiple MySQL rows → single PostgreSQL record
- **State reliability**: PostgreSQL table instead of JSON file

### Breaking Changes

- `--state-file` parameter removed from incremental migration (no longer uses JSON)
- `--use-id` flag removed (consolidation-based approach is now default)
- Incremental migration requires full migration to be run first
- `BATCH_SIZE` environment variable renamed to `CONSOLIDATION_GROUP_LIMIT` (update your .env file)

## [Previous] - Before 2025-12-30

### Features
- Full migration support
- Incremental migration with timestamp tracking
- JSONB transformation
- Partitioning by year
- GIN indexes for JSONB queries
- Benchmark system
- Progress tracking
- Rich logging