Problems identified:
1. Buffer size of batch_size * 10 (100k rows) was too large, causing
migration_state to not update for several minutes on low-consolidation partitions
2. State updates only happened every 10 batches, not reflecting actual progress
Changes:
- Reduce insert_buffer_size from 10x to 5x batch_size (50k rows)
- Update migration_state after EVERY batch flush, not every 10 batches
- Add debug logging showing flush operations and total migrated count
- This provides better visibility into migration progress and checkpointing
For partitions with low consolidation ratio (like d0 with 1.1x), this ensures
migration_state is updated more frequently, supporting better resume capability
and providing visibility into actual progress.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
When processing partitions with many small consolidation groups (low consolidation
ratio), the previous approach of inserting each group individually caused excessive
database round-trips.
Example from partition d0:
- 572k MySQL rows
- 514k unique consolidation keys (1.1x consolidation ratio)
- 514k separate INSERT statements = severe performance bottleneck
Changes:
- Accumulate consolidated rows in a buffer (size = batch_size * 10)
- Flush buffer to PostgreSQL when full or when partition is complete
- Reduces 514k INSERT statements to ~50 batches for d0
- Significant performance improvement expected (8-10x faster for low-consolidation partitions)
The progress tracker still counts MySQL source rows (before consolidation), so
the progress bar remains accurate.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
CRITICAL FIX: Previous implementation was doing GROUP BY to get unique
keys, then a separate WHERE query for EACH group. With millions of groups,
this meant millions of separate MySQL queries = 12 bytes/sec = unusable.
New approach (single query):
- Fetch all rows from partition ordered by consolidation key
- Group them in Python as we iterate
- One query per LIMIT batch, not one per group
- ~100,000x faster than N+1 approach
Query uses index efficiently: ORDER BY (UnitName, ToolNameID, EventDate, EventTime, NodeNum)
matches index prefix and keeps groups together for consolidation.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Log shows:
- Current partition index and total ([X/Y])
- Partition name being processed
- Number of groups consolidated per partition after completion
This helps track migration progress when processing 18 partitions,
making it easier to identify slow partitions or issues.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
With partition-based consolidation, resume is now simpler:
- No longer track last_migrated_id (not useful for partition iteration)
- Resume capability: if rows exist in target table, migration was interrupted
- Use total_rows_migrated count to calculate remaining work
- Update state every 10 consolidations instead of maintaining per-batch state
This aligns resume mechanism with the new partition-based architecture
where we process complete consolidation groups, not sequential ID ranges.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Changed consolidation strategy to leverage MySQL partitioning:
- Added get_table_partitions() to list all partitions
- Added fetch_consolidation_groups_from_partition() to read groups by consolidation key
- Each group (UnitName, ToolNameID, EventDate, EventTime) is fetched completely
- All nodes of same group are consolidated into single row with JSONB measurements
- Process partitions sequentially for predictable memory usage
Key benefits:
- Guaranteed complete consolidation (no fragmentation across batches)
- Deterministic behavior - same group always consolidated together
- Better memory efficiency with partition limits (100k groups per query)
- Clear audit trail of which partition each row came from
Tested with partition d3: 6960 input rows → 100 consolidated rows (69.6:1 ratio)
with groups containing 24-72 nodes each.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Previously, consolidation happened per-batch, which meant if the same
(unit, tool, date, time) group spanned multiple batches, nodes would be
split into separate rows. For example, nodes 1-32 would be split into 4
separate rows instead of 1 consolidated row.
Now, we buffer rows with the same consolidation key and only consolidate
when we see a NEW consolidation key. This ensures all nodes of the same
group are consolidated together, regardless of batch boundaries.
Results: Proper 25:1 consolidation ratio with all nodes grouped correctly.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The _update_migration_state() method was using logic:
status = "in_progress" if last_id is not None else "completed"
This was incorrect because:
1. last_id is always set during periodic updates (to track resume point)
2. So status would always be "in_progress" even when migration finished
3. migration_completed_at would never be set
Solution: Add is_final parameter to explicitly mark when migration is
complete. During periodic updates, is_final=False (status="in_progress").
Only when called at the end, is_final=True (status="completed").
This ensures:
- migration_state.status = "completed" when done
- migration_state.migration_completed_at is set
- Proper tracking for knowing if migration is finished
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Added logging to trace the final migration state update process:
- Log final count from PostgreSQL
- Log final last ID from table
- Log before and after _update_migration_state() call
This helps debug why migration_state might not be getting updated
when migration completes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The _update_migration_state() method was using pg_conn.execute() which has
its own connection management. This could cause issues with transaction
handling when called at end of migration.
Changed to use explicit cursor with guaranteed commit:
- Use pg_conn.connection.cursor() to get a direct cursor
- Execute the INSERT ... ON CONFLICT query
- Explicitly call pg_conn.connection.commit()
- This matches the pattern used in other parts of the code
This ensures that final migration state (completed status, final counts)
are properly persisted to the database.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
When migration finishes, we need to update migration_state with:
1. The final actual row count from PostgreSQL
2. The final last_migrated_id (MAX(id) from the table)
3. Mark status as 'completed' (handled by _update_migration_state)
Previously, the final state update was missing, so migration_state
was left with stale data from the periodic updates.
Now _update_migration_state is called at the end to record the
authoritative final state.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The previous fix was too aggressive - calling get_row_count() on every batch
meant executing COUNT(*) on a 14M row table for each batch. With a typical
batch size of ~10k rows and consolidation ratio of ~10:1, this meant:
- ~500-1000 batches total
- ~500k COUNT(*) queries on a huge table = completely destroyed performance
New approach:
- Keep local accumulator for migrated count (fast)
- Update total_rows_migrated to DB only every 10 batches (reduces COUNT(*) 50x)
- Update last_migrated_id on every batch via UPDATE (fast, no COUNT)
- Do final COUNT(*) at end of migration for accurate total
This maintains accuracy while being performant. The local count is reliable
because we're tracking inserts in a single sequential migration.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The progress bar was appearing frozen because:
- Total was set to MySQL rows to process (111M)
- Progress was updated by PostgreSQL rows inserted (11M after consolidation)
- This created a 10:1 mismatch, making progress appear to crawl
Solution:
- Track progress based on MySQL rows processed (matches total)
- Use batch_size (MySQL rows) instead of inserted count (PostgreSQL rows)
- Change batch_max_id calculation to use original batch instead of transformed
This ensures the progress bar advances at a visible rate while still
maintaining accurate row count tracking from the database.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Replace session-level counting with direct table COUNT queries to ensure
total_rows_migrated always reflects actual reality in PostgreSQL. This fixes
the discrepancy where the counter was only tracking rows from the current session
and didn't account for earlier insertions or duplicates from failed resume attempts.
Key improvements:
- Use get_row_count() after each batch to get authoritative total
- Preserve previous count on resume and accumulate across sessions
- Remove dependency on error-prone session-level counters
- Ensures migration_state.total_rows_migrated matches actual table row count
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
