alex/mysql2postgres
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Add error logging and fix incremental migration state tracking

Browse Source 
Implement comprehensive error handling and fix state management bug in incremental migration:

Error Logging System:
- Add validation for consolidation keys (NULL dates, empty IDs, corrupted Java strings)
- Log invalid keys to dedicated error files with detailed reasons
- Full migration: migration_errors_<table>_<partition>.log
- Incremental migration: migration_errors_<table>_incremental_<timestamp>.log (timestamped to preserve history)
- Report total count of skipped invalid keys at migration completion
- Auto-delete empty error log files

State Tracking Fix:
- Fix critical bug where last_key wasn't updated after final buffer flush
- Track last_processed_key throughout migration loop
- Update state both during periodic flushes and after final flush
- Ensures incremental migration correctly resumes from last migrated key

Validation Checks:
- EventDate IS NULL or EventDate = '0000-00-00'
- EventTime IS NULL
- ToolNameID IS NULL or empty string
- UnitName IS NULL or empty string
- UnitName starting with '[L' (corrupted Java strings)

Documentation:
- Update README.md with error logging behavior
- Update MIGRATION_WORKFLOW.md with validation details
- Update CHANGELOG.md with new features and fixes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
alex
2026-01-01 19:49:44 +01:00
parent 03e39eb925
commit 23e9fc9d82
5 changed files with 142 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
MIGRATION_WORKFLOW.md
View File

@@ -81,6 +81,26 @@ Both tables use consolidation to group multiple measurements into a single JSONB
(UnitName, ToolNameID, EventDate, EventTime)
```
### Data Quality Validation
The migration automatically validates and logs invalid consolidation keys:
- `EventDate IS NULL` or `EventDate = '0000-00-00'`
- `ToolNameID IS NULL` or `ToolNameID = ''` (empty string)
- `UnitName IS NULL` or `UnitName = ''` (empty string)
- `UnitName` starting with `[L` (corrupted Java strings like `[Ljava.lang.String;@...`)
- `EventTime IS NULL`
Invalid keys are:
- **Logged to error files** for tracking and analysis
- **Skipped automatically** to prevent migration failures
- **Counted and reported** at the end of migration
Error log files:
- Full migration: `migration_errors_<table>_<partition>.log` (e.g., `migration_errors_rawdatacor_p2024.log`)
- Incremental migration: `migration_errors_<table>_incremental_<timestamp>.log` (e.g., `migration_errors_rawdatacor_incremental_20260101_194500.log`)
Each incremental migration creates a new timestamped file to preserve history.
### Why Consolidation?
Instead of migrating individual sensor readings, we: