alex/mysql2postgres
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

debug: Add detailed logging to consolidation group buffering

Browse Source 
Added logging to track:
- When groups are buffered at batch boundaries
- Group consolidation keys and row counts
- When buffered groups are resumed in next batch
- Final batch group yields

This will help diagnose why some nodes are being lost during consolidation
(observed: nodes 1-11 missing from consolidated group, only nodes 12-22 present).

🤖 Generated with Claude Code

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This commit is contained in:
alex
2025-12-26 00:03:17 +01:00
parent 9ef65995d4
commit 3687f77911
1 changed files with 12 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
src/connectors/mysql_connector.py
View File

@@ -412,6 +412,10 @@ class MySQLConnector:
# If we have a buffered group, prepend it to continue # If we have a buffered group, prepend it to continue
if buffered_group: if buffered_group:
logger.debug(
f"Resuming buffered group: key={last_buffered_key}, "
f"prev_buffered_rows={len(buffered_group)}, new_rows={len(rows)}"
)
sorted_rows = buffered_group + sorted_rows sorted_rows = buffered_group + sorted_rows
buffered_group = [] buffered_group = []
@@ -442,9 +446,17 @@ class MySQLConnector:
# Buffer incomplete group at boundary for next batch # Buffer incomplete group at boundary for next batch
buffered_group = current_group buffered_group = current_group
last_buffered_key = last_key last_buffered_key = last_key
logger.debug(
f"Buffering group at boundary: key={last_key}, "
f"rows_in_group={len(current_group)}, total_rows_fetched={len(rows)}"
)
else: else:
# This is the last batch, yield final group # This is the last batch, yield final group
if current_group: if current_group:
logger.debug(
f"Final batch: yielding group key={last_key}, "
f"rows_in_group={len(current_group)}, total_rows_fetched={len(rows)}"
)
yield current_group yield current_group
last_id = rows[-1][id_column] last_id = rows[-1][id_column]