Commit Graph

6 Commits

Author SHA1 Message Date
3a53564bb5 con web ui 2026-01-11 21:56:50 +01:00
931fec0959 fix logger x partition 2026-01-05 14:00:29 +01:00
5c9df3d06f fix incremental 2025-12-30 15:16:54 +01:00
58624c0866 feat: Add --partition flag to migrate only specific partition
Allows testing/debugging by migrating only a single partition instead of
the entire table. Useful for:
- Testing consolidation on specific partitions
- Quick verification of fixes without full migration
- Targeted debugging

Usage:
    python3 main.py migrate full --table ELABDATADISP --partition d10
    python3 main.py migrate full --table RAWDATACOR --partition d11 --resume

Changes:
- Add partition parameter to FullMigrator.migrate()
- Filter partitions list to only specified partition if provided
- Validate partition exists in available partitions
- Add --partition CLI option to migrate full command
- Update message to show partition in progress

🤖 Generated with Claude Code

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2025-12-27 09:05:11 +01:00
0f217379ea fix: Use actual PostgreSQL row count for total_rows_migrated tracking
Replace session-level counting with direct table COUNT queries to ensure
total_rows_migrated always reflects actual reality in PostgreSQL. This fixes
the discrepancy where the counter was only tracking rows from the current session
and didn't account for earlier insertions or duplicates from failed resume attempts.

Key improvements:
- Use get_row_count() after each batch to get authoritative total
- Preserve previous count on resume and accumulate across sessions
- Remove dependency on error-prone session-level counters
- Ensures migration_state.total_rows_migrated matches actual table row count

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2025-12-23 15:33:27 +01:00
62577d3200 feat: Add MySQL to PostgreSQL migration tool with JSONB transformation
Implement comprehensive migration solution with:
- Full and incremental migration modes
- JSONB schema transformation for RAWDATACOR and ELABDATADISP tables
- Native PostgreSQL partitioning (2014-2031)
- Optimized GIN indexes for JSONB queries
- Rich logging with progress tracking
- Complete benchmark system for MySQL vs PostgreSQL comparison
- CLI interface with multiple commands (setup, migrate, benchmark)
- Configuration management via .env file
- Error handling and retry logic
- Batch processing for performance (configurable batch size)

Database transformations:
- RAWDATACOR: 16 Val columns + units → single JSONB measurements
- ELABDATADISP: 25+ measurement fields → structured JSONB with categories

🤖 Generated with Claude Code

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2025-12-10 19:57:11 +01:00