alex/mysql2postgres
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Add error logging and fix incremental migration state tracking

Browse Source 
Implement comprehensive error handling and fix state management bug in incremental migration:

Error Logging System:
- Add validation for consolidation keys (NULL dates, empty IDs, corrupted Java strings)
- Log invalid keys to dedicated error files with detailed reasons
- Full migration: migration_errors_<table>_<partition>.log
- Incremental migration: migration_errors_<table>_incremental_<timestamp>.log (timestamped to preserve history)
- Report total count of skipped invalid keys at migration completion
- Auto-delete empty error log files

State Tracking Fix:
- Fix critical bug where last_key wasn't updated after final buffer flush
- Track last_processed_key throughout migration loop
- Update state both during periodic flushes and after final flush
- Ensures incremental migration correctly resumes from last migrated key

Validation Checks:
- EventDate IS NULL or EventDate = '0000-00-00'
- EventTime IS NULL
- ToolNameID IS NULL or empty string
- UnitName IS NULL or empty string
- UnitName starting with '[L' (corrupted Java strings)

Documentation:
- Update README.md with error logging behavior
- Update MIGRATION_WORKFLOW.md with validation details
- Update CHANGELOG.md with new features and fixes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
alex
2026-01-01 19:49:44 +01:00
parent 03e39eb925
commit 23e9fc9d82
5 changed files with 142 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
README.md
View File

@@ -138,6 +138,30 @@ Il tool non migra le righe MySQL 1:1 in PostgreSQL. Invece, **consolida** multip
(UnitName, ToolNameID, EventDate, EventTime)
```
### Validazione e Gestione Dati Corrotti
La migrazione valida automaticamente le chiavi di consolidamento e gestisce dati corrotti:
**Validazioni applicate:**
- `EventDate IS NULL` o `EventDate = '0000-00-00'`
- `ToolNameID IS NULL` o `ToolNameID = ''` (stringa vuota)
- `UnitName IS NULL` o `UnitName = ''` (stringa vuota)
- `UnitName` che inizia con `[L` (stringhe Java corrotte come `[Ljava.lang.String;@...`)
- `EventTime IS NULL`
**Comportamento:**
- Le chiavi non valide vengono **saltate automaticamente** per evitare interruzioni
- Ogni chiave scartata viene **loggata in file dedicati** per tracciabilità
- Il numero totale di chiavi scartate viene **riportato alla fine** della migrazione
**File di log degli errori:**
- Full migration: `migration_errors_<table>_<partition>.log` (es. `migration_errors_rawdatacor_p2024.log`)
- Incremental migration: `migration_errors_<table>_incremental_<timestamp>.log` (es. `migration_errors_rawdatacor_incremental_20260101_194500.log`)
Ogni esecuzione incrementale crea un nuovo file con timestamp per mantenere lo storico.
Questo approccio garantisce che la migrazione non si interrompa per dati corrotti, permettendo comunque di tracciare e analizzare le anomalie.
### Perché Consolidare?
**MySQL** ha molte righe per lo stesso momento: