The consolidation grouping logic now properly handles rows with the same
consolidation key (UnitName, ToolNameID, EventDate, EventTime) that span
across multiple fetch batches.
Key improvements:
- Added buffering of incomplete groups at batch boundaries
- When a batch is full (has exactly limit rows), the final group is buffered
to be prepended to the next batch, ensuring complete group consolidation
- When the final batch is reached (fewer than limit rows), all buffered and
current groups are yielded
This ensures that all nodes with the same consolidation key are grouped
together in a single consolidated row, eliminating node fragmentation.
Added comprehensive unit tests verifying:
- Multi-node consolidation with batch boundaries
- RAWDATACOR consolidation with multiple nodes
- Groups that span batch boundaries are kept complete
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Replace session-level counting with direct table COUNT queries to ensure
total_rows_migrated always reflects actual reality in PostgreSQL. This fixes
the discrepancy where the counter was only tracking rows from the current session
and didn't account for earlier insertions or duplicates from failed resume attempts.
Key improvements:
- Use get_row_count() after each batch to get authoritative total
- Preserve previous count on resume and accumulate across sessions
- Remove dependency on error-prone session-level counters
- Ensures migration_state.total_rows_migrated matches actual table row count
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Configuration improvements:
- Set read_timeout=300 (5 minutes) to handle long queries
- Set write_timeout=300 (5 minutes) for writes
- Set max_allowed_packet=64MB to handle larger data transfers
Retry logic:
- Added retry mechanism with max 3 retries on fetch failure
- Auto-reconnect on connection loss before retry
- Better error messages showing retry attempts
This fixes the 'connection is lost' error that occurs during
long-running migrations by:
1. Giving MySQL queries more time to complete
2. Allowing larger packet sizes for bulk data
3. Automatically recovering from connection drops
Fixes: 'Connection is lost' error during full migration
