Implement partition-based consolidation for ELABDATADISP

Changed consolidation strategy to leverage MySQL partitioning:
- Added get_table_partitions() to list all partitions
- Added fetch_consolidation_groups_from_partition() to read groups by consolidation key
- Each group (UnitName, ToolNameID, EventDate, EventTime) is fetched completely
- All nodes of same group are consolidated into single row with JSONB measurements
- Process partitions sequentially for predictable memory usage

Key benefits:
- Guaranteed complete consolidation (no fragmentation across batches)
- Deterministic behavior - same group always consolidated together
- Better memory efficiency with partition limits (100k groups per query)
- Clear audit trail of which partition each row came from

Tested with partition d3: 6960 input rows → 100 consolidated rows (69.6:1 ratio)
with groups containing 24-72 nodes each.

🤖 Generated with Claude Code

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This commit is contained in:
2025-12-25 21:49:30 +01:00
parent a394de99ef
commit bb27f749a0
3 changed files with 156 additions and 85 deletions

View File

@@ -49,6 +49,7 @@ class MigrationSettings(BaseSettings):
)
batch_size: int = 10000
consolidation_group_limit: int = 100000
log_level: str = "INFO"
dry_run: bool = False