The progress bar was appearing frozen because:
- Total was set to MySQL rows to process (111M)
- Progress was updated by PostgreSQL rows inserted (11M after consolidation)
- This created a 10:1 mismatch, making progress appear to crawl
Solution:
- Track progress based on MySQL rows processed (matches total)
- Use batch_size (MySQL rows) instead of inserted count (PostgreSQL rows)
- Change batch_max_id calculation to use original batch instead of transformed
This ensures the progress bar advances at a visible rate while still
maintaining accurate row count tracking from the database.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Replace session-level counting with direct table COUNT queries to ensure
total_rows_migrated always reflects actual reality in PostgreSQL. This fixes
the discrepancy where the counter was only tracking rows from the current session
and didn't account for earlier insertions or duplicates from failed resume attempts.
Key improvements:
- Use get_row_count() after each batch to get authoritative total
- Preserve previous count on resume and accumulate across sessions
- Remove dependency on error-prone session-level counters
- Ensures migration_state.total_rows_migrated matches actual table row count
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Configuration improvements:
- Set read_timeout=300 (5 minutes) to handle long queries
- Set write_timeout=300 (5 minutes) for writes
- Set max_allowed_packet=64MB to handle larger data transfers
Retry logic:
- Added retry mechanism with max 3 retries on fetch failure
- Auto-reconnect on connection loss before retry
- Better error messages showing retry attempts
This fixes the 'connection is lost' error that occurs during
long-running migrations by:
1. Giving MySQL queries more time to complete
2. Allowing larger packet sizes for bulk data
3. Automatically recovering from connection drops
Fixes: 'Connection is lost' error during full migration
Replace cursor.copy() with cursor.executemany() for more reliable
batch inserts in PostgreSQL. The COPY method has issues with format
and data encoding in psycopg3.
Changes:
- Use executemany() with parameterized INSERT statements
- Let psycopg handle parameter escaping and encoding
- Convert JSONB dicts to JSON strings automatically
- More compatible with various data types
This ensures that data is actually being inserted into PostgreSQL
during migration, fixing the issue where data wasn't appearing in
the database after migration completed.
Fixes: Data not being persisted in PostgreSQL during migration
- On successful execution (no exception): explicitly commit before closing
- On exception: explicitly rollback before closing
- Add try-except to handle commit/rollback failures gracefully
This ensures that all inserted data is committed to the database
when the context manager exits. Previously, commits were only done
per-batch in insert_batch(), but the final context exit wasn't
ensuring a final commit.
Fixes: Data not appearing in PostgreSQL after migration completes
- TABLE_CONFIGS now accepts both 'RAWDATACOR' and 'rawdatacor' as keys
- TABLE_CONFIGS now accepts both 'ELABDATADISP' and 'elabdatadisp' as keys
- Reuse same config dict for both cases to avoid duplication
This allows FullMigrator to work correctly when initialized with
uppercase table names from the CLI while DataTransformer works
with lowercase names.
Fixes: 'Unknown table: RAWDATACOR' error during migration
- Create rawdatacor_id_seq for auto-increment of id column
- Create elabdatadisp_id_seq for auto-increment of id_elab_data column
- Both sequences use DEFAULT nextval() to auto-generate IDs on insert
This replaces PRIMARY KEY functionality since PostgreSQL doesn't
support PRIMARY KEY on partitioned tables with expression-based ranges.
IDs are now auto-incremented without primary key constraint.
Tested: schema creation works correctly with sequences
PostgreSQL doesn't support PRIMARY KEY or UNIQUE constraints on
partitioned tables when using RANGE partitioning on expressions
(like EXTRACT(YEAR FROM event_date)).
Changed:
- RAWDATACOR: removed PRIMARY KEY (id, event_date) and UNIQUE constraint
- ELABDATADISP: removed PRIMARY KEY (id_elab_data, event_date) and UNIQUE constraint
- Tables now have no constraints except NOT NULL on required columns
This is a PostgreSQL limitation with partitioned tables.
Constraints can be added per-partition if needed, but for simplicity
we rely on application-level validation.
Fixes: 'vincolo PRIMARY KEY non supportato con una definizione di chiave di partizione'
- Fix ConfigDict model_config for Pydantic v2.12+ compatibility
- Add env_file and env_file_encoding to all config classes
- Each config class now properly loads from .env with correct prefix
Fixes: ValidationError when loading settings from .env file
CLI now works correctly with 'uv run python main.py'
