alex/mysql2postgres
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: Ensure complete node consolidation by ordering MySQL query by consolidation key

Browse Source 
Root cause: Nodes 1-11 had IDs in 132M+ range while nodes 12-22 had IDs in 298-308
range, causing them to be fetched in batches thousands apart using keyset pagination
by ID. This meant they arrived as separate groups and were never unified into a
single consolidated row.

Solution: Order MySQL query by (UnitName, ToolNameID, EventDate, EventTime) instead
of by ID. This guarantees all rows for the same consolidation key arrive together,
ensuring they are grouped and consolidated into a single row with JSONB measurements
keyed by node number.

Changes:
- fetch_consolidation_groups_from_partition(): Changed from keyset pagination by ID
  to ORDER BY consolidation key. Simplify grouping logic since ORDER BY already ensures
  consecutive rows have same key.
- full_migration.py: Add cleanup of partial partitions on resume. When resuming and a
  partition was started but not completed, delete its incomplete data before
  re-processing to avoid duplicates. Also recalculate total_rows_migrated from actual
  database count.
- config.py: Add postgres_pk field to TABLE_CONFIGS to specify correct primary key
  column names in PostgreSQL (id vs id_elab_data).
- Cleanup: Remove temporary test scripts used during debugging

Performance note: ORDER BY consolidation key requires index for speed. Index
(UnitName, ToolNameID, EventDate, EventTime) created with ALGORITHM=INPLACE
LOCK=NONE to avoid blocking reads.

🤖 Generated with Claude Code

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This commit is contained in:
alex
2025-12-26 18:22:23 +01:00
parent 681812d0a4
commit 1430ef206f
5 changed files with 70 additions and 234 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
src/migrator/full_migration.py
View File

@@ -120,12 +120,43 @@ class FullMigrator:
logger.info(f"[{partition_idx}/{len(partitions)}] Processing partition {partition}...")
partition_group_count = 0
# Determine resume point within this partition
# If resuming and this is the last completed partition, start from last_id
# If resuming and this is NOT the last completed partition,
# it means it was only partially processed - clean it up first
start_id = None
if last_completed_partition == partition and previous_migrated_count > 0:
# For resume within same partition, we need to query the last ID inserted
# This is a simplified approach: just continue from ID tracking
if resume and last_completed_partition and partition > last_completed_partition:
# This partition was started but not completed - delete its partial data
logger.warning(
f"Partition {partition} was partially processed in previous run. "
f"Cleaning up partial data before resume..."
)
try:
with pg_conn.connection.cursor() as cursor:
# Get the primary key column name for this table
pk_column = self.config.get("postgres_pk", "id")
# Delete rows from this partition that were inserted from MySQL rows
# We identify them by looking for rows inserted after the migration started
# This is safe because we're re-processing the entire partition
# Note: This is a simplified approach - in production you might want more granular tracking
last_id = self._get_last_migrated_id(pg_conn, pg_table)
if last_id:
cursor.execute(
f"DELETE FROM {pg_table} WHERE {pk_column} > %s",
(last_id,)
)
pg_conn.connection.commit()
logger.info(f"Cleaned up partial data for partition {partition}")
# Recalculate migrated count based on actual data in database
cursor.execute(f"SELECT COUNT(*) FROM {pg_table}")
actual_count = cursor.fetchone()[0]
migrated = actual_count
logger.info(f"Recalculated total_rows_migrated: {migrated} (actual rows in database)")
except Exception as e:
logger.warning(f"Failed to clean up partial data: {e}")
# Continue anyway - might be able to deduplicate later
elif resume and last_completed_partition == partition and previous_migrated_count > 0:
# Resuming within the same partition - continue from last ID
start_id = self._get_last_migrated_id(pg_conn, pg_table)
if start_id:
logger.info(f"Resuming partition {partition} from ID > {start_id}")
@@ -330,6 +361,7 @@ class FullMigrator:
# Update PostgreSQL migration_state table
try:
with pg_conn.connection.cursor() as cursor:
logger.info(f"About to update migration_state: table={pg_table}, last_partition={last_partition}, last_id={last_id}, rows={rows_migrated}")
query = f"""
INSERT INTO migration_state
(table_name, last_migrated_timestamp, last_migrated_id, total_rows_migrated,
@@ -356,9 +388,10 @@ class FullMigrator:
)
)
pg_conn.connection.commit()
logger.debug(f"Migration state updated: {rows_migrated} rows total, last_id={last_id}, status={status}")
logger.info(f"Migration state updated successfully: {rows_migrated} rows, last_partition={last_partition}, last_id={last_id}")
except Exception as e:
logger.warning(f"Failed to update migration state in PostgreSQL: {e}")
logger.error(f"Failed to update migration state in PostgreSQL: {e}")
raise
# Also save to state file for incremental migrations
try: