fix: Add timeout settings and retry logic to MySQL connector

Configuration improvements:
- Set read_timeout=300 (5 minutes) to handle long queries
- Set write_timeout=300 (5 minutes) for writes
- Set max_allowed_packet=64MB to handle larger data transfers

Retry logic:
- Added retry mechanism with max 3 retries on fetch failure
- Auto-reconnect on connection loss before retry
- Better error messages showing retry attempts

This fixes the 'connection is lost' error that occurs during
long-running migrations by:
1. Giving MySQL queries more time to complete
2. Allowing larger packet sizes for bulk data
3. Automatically recovering from connection drops

Fixes: 'Connection is lost' error during full migration
This commit is contained in:
2025-12-21 09:53:34 +01:00
parent 821cda850e
commit b09cfcf9df
8 changed files with 761 additions and 119 deletions

View File

@@ -156,13 +156,13 @@ _rawdatacor_config = {
"mysql_table": "RAWDATACOR",
"postgres_table": "rawdatacor",
"primary_key": "id",
"partition_key": "event_date",
"partition_key": "event_timestamp",
}
_elabdatadisp_config = {
"mysql_table": "ELABDATADISP",
"postgres_table": "elabdatadisp",
"primary_key": "idElabData",
"partition_key": "event_date",
"partition_key": "event_timestamp",
}
TABLE_CONFIGS = {