- Fix year_to_partition_name() RAWDATACOR logic: properly clamp year between 2014-2024
before calculating partition index with formula (year - 2014)
- Previously: incorrectly tried to return "d" partition type with wrong formula
- Now: correctly returns "part{year-2014}" for RAWDATACOR table
- Update docstring: clarify d17 = 2030 (not 2031) as maximum ELABDATADISP partition
- Ensure partition mapping is consistent between year_to_partition_name() and
get_partitions_from_year() functions
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Fix timezone inconsistency between migration_started_at and migration_completed_at:
Schema Changes:
- Change TIMESTAMP to TIMESTAMPTZ for migration_started_at and migration_completed_at
- PostgreSQL stores timestamps in UTC and converts to local timezone on display
- Ensures consistent timezone handling across all timestamp columns
Code Changes:
- Replace datetime.utcnow() with datetime.now(timezone.utc)
- Use timezone-aware datetime objects for proper TIMESTAMPTZ handling
- Import timezone module for UTC timezone support
Impact:
- Previous issue: migration_completed_at was 1 hour behind migration_started_at
- Root cause: CURRENT_TIMESTAMP (local time) vs datetime.utcnow() (UTC naive)
- Solution: Both columns now use TIMESTAMPTZ with timezone-aware datetimes
Note: Existing migration_state records will have old TIMESTAMP format until table is altered:
ALTER TABLE migration_state
ALTER COLUMN migration_started_at TYPE TIMESTAMPTZ,
ALTER COLUMN migration_completed_at TYPE TIMESTAMPTZ;
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Implement comprehensive error handling and fix state management bug in incremental migration:
Error Logging System:
- Add validation for consolidation keys (NULL dates, empty IDs, corrupted Java strings)
- Log invalid keys to dedicated error files with detailed reasons
- Full migration: migration_errors_<table>_<partition>.log
- Incremental migration: migration_errors_<table>_incremental_<timestamp>.log (timestamped to preserve history)
- Report total count of skipped invalid keys at migration completion
- Auto-delete empty error log files
State Tracking Fix:
- Fix critical bug where last_key wasn't updated after final buffer flush
- Track last_processed_key throughout migration loop
- Update state both during periodic flushes and after final flush
- Ensures incremental migration correctly resumes from last migrated key
Validation Checks:
- EventDate IS NULL or EventDate = '0000-00-00'
- EventTime IS NULL
- ToolNameID IS NULL or empty string
- UnitName IS NULL or empty string
- UnitName starting with '[L' (corrupted Java strings)
Documentation:
- Update README.md with error logging behavior
- Update MIGRATION_WORKFLOW.md with validation details
- Update CHANGELOG.md with new features and fixes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Critical bug: current_group and current_key were inside the while loop,
causing them to be reset on each batch iteration. When an incomplete group
spanned a batch boundary, it would be:
1. Buffered at end of batch N (in local current_group)
2. LOST when loop continued (new local variables created)
3. Re-fetched and yielded again in batch N+1
This caused the same consolidated record to be inserted many times.
Solution: Move current_group and current_key OUTSIDE while loop to persist
across batch iterations. Incomplete groups now properly merge across batch
boundaries without duplication.
Algorithm:
- Only yield groups when we're 100% certain they're complete
- A group is complete when the next key differs from current key
- At batch boundaries, incomplete groups stay buffered for next batch
- Resume always uses last_completed_key to avoid re-processing
This fixes the user's observation of 27 identical rows for the same
consolidated record.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Problem: When a batch ended with an incomplete group (same consolidation key
as last row), the code was not yielding it (correct), but was also updating
last_key to that incomplete key. Next iteration would query:
WHERE (key) > (incomplete_key)
This would SKIP all remaining rows of that key that were on next page!
Result: Groups that span batch boundaries got split - e.g., nodes 1-11 in
first batch yield as incomplete (not yielded), nodes 12-22 never fetched
because next query starts AFTER the incomplete key.
Fix: Track whether current_group is incomplete (pending) at batch boundary.
If incomplete (last_row_key == current_key), keep it in memory and DON'T
update last_key. This ensures next batch continues from where the incomplete
group started, fetching remaining rows of that key.
Logic:
- If last_row_key != current_key: group is complete, yield it
- If last_row_key == current_key: group is incomplete, keep buffering
- If current_key is None: all groups complete, update last_key normally
- If current_key is not None: group pending, DON'T update last_key
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Problem: When using --partition to test a specific partition, the code would
still read last_completed_partition from migration_state and skip partitions
based on that, potentially skipping the requested partition.
Solution: When partition parameter is specified, set last_completed_partition
to None to force processing the requested partition regardless of what's in
migration_state.
This ensures --partition works as expected:
- python3 main.py migrate full --table ELABDATADISP --partition d10 --resume
Will process ONLY d10, not resume from migration_state partition tracking
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Problem: dry-run was blocked with 'Migration already in progress' error,
even though dry-run should not require --resume (it doesn't modify data).
Fix: Skip the 'already in progress' check if dry_run is True. This allows:
- Testing partitions without interrupting active migration
- Verifying what would be migrated without needing --resume
- Checking consolidation logic on specific partitions
Also improved dry-run message to show partition info if specified.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Allows testing/debugging by migrating only a single partition instead of
the entire table. Useful for:
- Testing consolidation on specific partitions
- Quick verification of fixes without full migration
- Targeted debugging
Usage:
python3 main.py migrate full --table ELABDATADISP --partition d10
python3 main.py migrate full --table RAWDATACOR --partition d11 --resume
Changes:
- Add partition parameter to FullMigrator.migrate()
- Filter partitions list to only specified partition if provided
- Validate partition exists in available partitions
- Add --partition CLI option to migrate full command
- Update message to show partition in progress
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Previously incremental migration (both timestamp-based and ID-based) did not
consolidate rows, resulting in one row per node instead of consolidated
measurements with JSONB nodes.
Solution: Add _consolidate_batch() method to group rows by consolidation key
and consolidate them before transformation. Apply consolidation in both:
1. _migrate_by_timestamp() - timestamp-based incremental migration
2. _migrate_by_id() - ID-based incremental migration
Changes:
- For RAWDATACOR and ELABDATADISP tables: consolidate batch by grouping rows
with same consolidation key before transforming
- Pass consolidate=False to transform_batch since rows are already consolidated
- Handle cases where batch has single rows (no consolidation needed)
This ensures incremental migration produces the same consolidated output as
full migration, with multiple nodes properly merged into single row with JSONB
measurements.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Bug: When batch limit was reached (len(rows) >= limit), code was yielding the
current_group immediately, even if it was incomplete. This caused groups that
spanned multiple batches to be split.
Example:
- First batch contains UnitA nodes 1-11 with same consolidation key
- Code yields them as complete group before seeing nodes 12-22 in next batch
- Next batch starts with different key, so incomplete group is never merged
- Result: 11 separate rows instead of 1 consolidated row
Root cause: Not checking if the group might continue in the next batch
Fix: Before yielding at batch boundary, check if the LAST row in current batch
has the SAME consolidation key as the current_group:
- If YES (last_row_key == current_key): DON'T yield yet, keep buffering
- If NO (last_row_key != current_key): Yield, group is definitely complete
This ensures groups that span batch boundaries are kept together and fully
consolidated.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Problem: During batch flushes, last_id was passed as None to migration_state
updates. This meant the migration_state table never had the last_migrated_id
populated, making resume from specific ID impossible.
Solution: Call _get_last_migrated_id() after each batch flush and partition
completion to get the actual last inserted ID, and pass it to migration_state
updates. This ensures resume can pick up from the exact row that was last
migrated.
Changes:
- After each batch flush: get current_last_id and pass to _update_migration_state
- After partition completion: get final_last_id and pass to _update_migration_state
- This enables proper resume from specific row, not just partition boundaries
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Problem: During partition processing, frequent batch flushes were updating
migration_state but NOT passing last_partition parameter. This meant that even
though last_processed_partition was being tracked, it was being overwritten with
NULL every time the buffer was flushed.
Result: Migration state would show last_partition=None despite partitions being
completed, making resume tracking useless.
Solution: Pass last_processed_partition to ALL _update_migration_state() calls,
not just the final one after partition completion. This ensures the last
completed partition is always preserved in the database.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Bug: After fetching rows ordered by consolidation key (UnitName, ToolNameID,
EventDate, EventTime) from MySQL, code was re-sorting by NodeNum. This breaks
the grouping because rows with different consolidation keys get intermixed.
Example of what was happening:
- MySQL returns: (Unit1, Tool1, Date1, Time1, Node1),
(Unit1, Tool1, Date1, Time1, Node12),
(Unit2, Tool2, Date2, Time2, Node1)
- Re-sorting by NodeNum gives: (Unit1, Tool1, Date1, Time1, Node1),
(Unit2, Tool2, Date2, Time2, Node1),
(Unit1, Tool1, Date1, Time1, Node12)
- Result: Different consolidation keys are now mixed, each node becomes separate group!
Fix: Remove the re-sort. Trust MySQL's ORDER BY to keep rows of same key together.
The clustering nature of InnoDB ensures rows with same consolidation key are
physically adjacent.
This was causing 1 row per node instead of consolidating all nodes of same
measurement into 1 row.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Problem: The final migration_state update (when marking migration as complete)
was not passing last_partition parameter, so the last completed partition was
being lost in migration_state table. If migration was interrupted at any point,
resume would lose the partition tracking.
Solution:
1. Track last_processed_partition throughout the migration loop
2. Update it when each partition completes
3. Pass it to final _update_migration_state() call when marking migration as complete
Additional fix:
- Use correct postgres_pk column when querying MAX() ID for final state update
- This ensures we get the correct last ID even for tables with non-standard PK names
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Root cause: Nodes 1-11 had IDs in 132M+ range while nodes 12-22 had IDs in 298-308
range, causing them to be fetched in batches thousands apart using keyset pagination
by ID. This meant they arrived as separate groups and were never unified into a
single consolidated row.
Solution: Order MySQL query by (UnitName, ToolNameID, EventDate, EventTime) instead
of by ID. This guarantees all rows for the same consolidation key arrive together,
ensuring they are grouped and consolidated into a single row with JSONB measurements
keyed by node number.
Changes:
- fetch_consolidation_groups_from_partition(): Changed from keyset pagination by ID
to ORDER BY consolidation key. Simplify grouping logic since ORDER BY already ensures
consecutive rows have same key.
- full_migration.py: Add cleanup of partial partitions on resume. When resuming and a
partition was started but not completed, delete its incomplete data before
re-processing to avoid duplicates. Also recalculate total_rows_migrated from actual
database count.
- config.py: Add postgres_pk field to TABLE_CONFIGS to specify correct primary key
column names in PostgreSQL (id vs id_elab_data).
- Cleanup: Remove temporary test scripts used during debugging
Performance note: ORDER BY consolidation key requires index for speed. Index
(UnitName, ToolNameID, EventDate, EventTime) created with ALGORITHM=INPLACE
LOCK=NONE to avoid blocking reads.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Removed extensive debug logging that was added while troubleshooting the
consolidation grouping issue. The new simplified logic using NodeNum
sequence detection is clear enough without the additional logging.
This keeps the code cleaner and reduces log verbosity during migration.
Replace complex buffering logic with simpler approach: detect consolidation
group boundaries by NodeNum sequence. When NodeNum decreases (e.g., from 18
back to 1), we know a new measurement has started.
Changes:
- Sort rows by (consolidation_key, NodeNum) instead of just consolidation_key
- Detect group boundary when NodeNum decreases
- Still buffer incomplete groups at batch boundaries
- Merge buffered groups with same consolidation key in next batch
This approach is more intuitive and handles the case where nodes of the same
measurement are split across batches with non-contiguous IDs.
Example: Nodes 1-11 with ID 132657553-132657655, then nodes 12-22 with ID
298-308 - now correctly consolidated into single group instead of 15 separate rows.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The migration_state table was missing the last_completed_partition column
that was referenced in the migration update queries. This column tracks
which partition was last completed to enable accurate resume capability.
To apply this change to existing databases:
ALTER TABLE migration_state ADD COLUMN last_completed_partition VARCHAR(255);
For new databases, the table will be created with the column automatically.
Fix critical issue where consolidation groups with the same consolidation key
(UnitName, ToolNameID, EventDate, EventTime) but arriving in different batches
were being yielded separately instead of being merged.
Now when a buffered group has the same key as the start of the next batch,
they are prepended and consolidated together. If the key changes, the buffered
group is yielded before processing the new key's rows.
This fixes the issue where nodes 1-11 and 12-22 (with the same consolidation key)
were being inserted as two separate rows instead of one consolidated row with all 22 nodes.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Previously, last_completed_partition was updated during batch flushes while
the partition was still being processed. This caused resume to skip partitions
that were only partially completed.
Now, last_completed_partition is only updated AFTER all consolidation groups
in a partition have been processed and the final buffer flush is complete.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Remove unused fetch_all_rows() and fetch_rows_ordered_for_consolidation() methods.
These were part of the old migration strategy before partition-based consolidation.
The current implementation uses fetch_consolidation_groups_from_partition() which
handles keyset pagination and consolidation group buffering more efficiently.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Enhanced debug logging to show:
- Max ID for each yielded group (important for resume tracking)
- Group size and consolidation key for each operation
- Clear distinction between buffered and final groups
The max ID is tracked because:
- PostgreSQL stores MAX(id) per consolidated group for resume
- This logging helps verify correct ID tracking
- Assists debugging consolidation completeness
No functional changes, improved observability.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Critical bug fix for missing nodes in consolidated groups.
Problem: When a partition batch contained multiple consolidation groups,
only the LAST group was being buffered/yielded, causing earlier groups to
be lost. This happened when:
1. Batch < limit rows (final batch)
2. Multiple different consolidation keys present
3. First groups were yielded correctly
4. But FINAL group was only yielded if batch == limit
5. If batch < limit, final group was discarded
Example from partition d10:
- Fetch returns 22 rows with 2 groups: (nodes 1-11) and (nodes 12-22)
- Old code: yield nodes 1-11 on key change, then didn't yield nodes 12-22
- Result: inserted row had only nodes 12-22
Fix: Detect final batch with len(rows) < limit, then yield ALL groups
including the final one instead of buffering it.
Changes:
- Detect final batch early: is_final_batch = len(rows) < limit
- If final batch: yield current_group even if no key change follows
- If NOT final batch: buffer last group for continuity (original logic)
Now all nodes from all groups are properly consolidated.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Added logging to track:
- When groups are buffered at batch boundaries
- Group consolidation keys and row counts
- When buffered groups are resumed in next batch
- Final batch group yields
This will help diagnose why some nodes are being lost during consolidation
(observed: nodes 1-11 missing from consolidated group, only nodes 12-22 present).
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Problem: If migration was interrupted in the middle of processing a partition
(e.g., at row 100k of 500k), resume would re-process all 100k rows, causing
duplicate insertions and wasted time.
Solution:
1. Modified fetch_consolidation_groups_from_partition() to accept start_id parameter
2. When resuming within the same partition, query the last inserted ID from
migration_state.last_migrated_id
3. Use keyset pagination starting from (id > last_id) to skip already-processed rows
4. Added logic to detect when we're resuming within the same partition vs resuming
from a new partition
Flow:
- If last_completed_partition < current_partition: start from beginning of partition
- If last_completed_partition == current_partition: start from last_migrated_id
- If last_completed_partition > current_partition: skip to next uncompleted partition
This ensures resume is granular:
- Won't re-insert already inserted rows within a partition
- Continues exactly from where it stopped
- Combines with existing partition tracking for complete accuracy
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Problem: Resume was re-processing all partitions from the beginning because
migration_state didn't track which partition was the last one completed.
This caused duplicate data insertion and wasted time.
Solution:
1. Added 'last_completed_partition' column to migration_state table
2. Created _get_last_completed_partition() method to retrieve saved state
3. Updated _update_migration_state() to accept and save last_partition parameter
4. Modified migration loop to:
- Retrieve last_completed_partition on resume
- Skip partitions that were already completed (partition <= last_completed_partition)
- Update last_completed_partition after each partition finishes
- Log which partitions are being skipped during resume
Now when resuming:
- Only processes partitions after the last completed one
- Avoids re-migrating already completed partitions
- Provides clear logging showing which partitions are skipped
For example, if migration was at partition d5 when interrupted, resume will:
- Skip d0 through d5 (logging each skip)
- Continue with d6 onwards
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Problems identified:
1. Buffer size of batch_size * 10 (100k rows) was too large, causing
migration_state to not update for several minutes on low-consolidation partitions
2. State updates only happened every 10 batches, not reflecting actual progress
Changes:
- Reduce insert_buffer_size from 10x to 5x batch_size (50k rows)
- Update migration_state after EVERY batch flush, not every 10 batches
- Add debug logging showing flush operations and total migrated count
- This provides better visibility into migration progress and checkpointing
For partitions with low consolidation ratio (like d0 with 1.1x), this ensures
migration_state is updated more frequently, supporting better resume capability
and providing visibility into actual progress.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
When processing partitions with many small consolidation groups (low consolidation
ratio), the previous approach of inserting each group individually caused excessive
database round-trips.
Example from partition d0:
- 572k MySQL rows
- 514k unique consolidation keys (1.1x consolidation ratio)
- 514k separate INSERT statements = severe performance bottleneck
Changes:
- Accumulate consolidated rows in a buffer (size = batch_size * 10)
- Flush buffer to PostgreSQL when full or when partition is complete
- Reduces 514k INSERT statements to ~50 batches for d0
- Significant performance improvement expected (8-10x faster for low-consolidation partitions)
The progress tracker still counts MySQL source rows (before consolidation), so
the progress bar remains accurate.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The consolidation grouping logic now properly handles rows with the same
consolidation key (UnitName, ToolNameID, EventDate, EventTime) that span
across multiple fetch batches.
Key improvements:
- Added buffering of incomplete groups at batch boundaries
- When a batch is full (has exactly limit rows), the final group is buffered
to be prepended to the next batch, ensuring complete group consolidation
- When the final batch is reached (fewer than limit rows), all buffered and
current groups are yielded
This ensures that all nodes with the same consolidation key are grouped
together in a single consolidated row, eliminating node fragmentation.
Added comprehensive unit tests verifying:
- Multi-node consolidation with batch boundaries
- RAWDATACOR consolidation with multiple nodes
- Groups that span batch boundaries are kept complete
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
CRITICAL FIX: Previous implementation was doing GROUP BY to get unique
keys, then a separate WHERE query for EACH group. With millions of groups,
this meant millions of separate MySQL queries = 12 bytes/sec = unusable.
New approach (single query):
- Fetch all rows from partition ordered by consolidation key
- Group them in Python as we iterate
- One query per LIMIT batch, not one per group
- ~100,000x faster than N+1 approach
Query uses index efficiently: ORDER BY (UnitName, ToolNameID, EventDate, EventTime, NodeNum)
matches index prefix and keeps groups together for consolidation.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Changed consolidation_group_limit from 100k to 10k for faster queries.
Reverted to GROUP BY approach for getting consolidation keys:
- Uses MySQL index efficiently: (UnitName, ToolNameID, NodeNum, EventDate, EventTime)
- GROUP BY with NodeNum ensures we don't lose any combinations
- Faster GROUP BY queries than large ORDER BY queries
- Smaller LIMIT = faster pagination
This matches the original optimization suggestion and should be faster.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Log shows:
- Current partition index and total ([X/Y])
- Partition name being processed
- Number of groups consolidated per partition after completion
This helps track migration progress when processing 18 partitions,
making it easier to identify slow partitions or issues.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
With partition-based consolidation, resume is now simpler:
- No longer track last_migrated_id (not useful for partition iteration)
- Resume capability: if rows exist in target table, migration was interrupted
- Use total_rows_migrated count to calculate remaining work
- Update state every 10 consolidations instead of maintaining per-batch state
This aligns resume mechanism with the new partition-based architecture
where we process complete consolidation groups, not sequential ID ranges.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Changed consolidation strategy to leverage MySQL partitioning:
- Added get_table_partitions() to list all partitions
- Added fetch_consolidation_groups_from_partition() to read groups by consolidation key
- Each group (UnitName, ToolNameID, EventDate, EventTime) is fetched completely
- All nodes of same group are consolidated into single row with JSONB measurements
- Process partitions sequentially for predictable memory usage
Key benefits:
- Guaranteed complete consolidation (no fragmentation across batches)
- Deterministic behavior - same group always consolidated together
- Better memory efficiency with partition limits (100k groups per query)
- Clear audit trail of which partition each row came from
Tested with partition d3: 6960 input rows → 100 consolidated rows (69.6:1 ratio)
with groups containing 24-72 nodes each.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Previously, consolidation happened per-batch, which meant if the same
(unit, tool, date, time) group spanned multiple batches, nodes would be
split into separate rows. For example, nodes 1-32 would be split into 4
separate rows instead of 1 consolidated row.
Now, we buffer rows with the same consolidation key and only consolidate
when we see a NEW consolidation key. This ensures all nodes of the same
group are consolidated together, regardless of batch boundaries.
Results: Proper 25:1 consolidation ratio with all nodes grouped correctly.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
When fetching rows for consolidation, the original keyset pagination only
ordered by id, which caused nodes from the same (unit, tool, timestamp) to
be split across multiple batches. This resulted in incomplete consolidation,
with some nodes being missed.
Solution: Order by consolidation columns in addition to id:
- Primary: id (for keyset pagination)
- Secondary: UnitName, ToolNameID, EventDate, EventTime, NodeNum
This ensures all nodes with the same (unit, tool, timestamp) are grouped
together in the same batch, allowing proper consolidation within the batch.
Fixes: Nodes being lost during ELABDATADISP consolidation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Added logging to track which nodes are being consolidated and how many
measurement categories each node has. This helps debug cases where data
appears to be lost during consolidation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
MySQL returns numeric values as Decimal objects, which are not JSON serializable.
PostgreSQL JSONB requires proper JSON types.
Added convert_value() helper in _build_measurement_for_elabdatadisp_node() to:
- Convert Decimal → float
- Convert str → float
- Pass through other types unchanged
This ensures all numeric values are JSON-serializable before insertion into
the measurements JSONB column.
Fixes: "Object of type Decimal is not JSON serializable" error
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
ELABDATADISP uses 'idElabData' as the primary key, while RAWDATACOR uses 'id'.
Updated the fetch method to detect the correct column based on the table name:
- RAWDATACOR: use 'id' column
- ELABDATADISP: use 'idElabData' column
This allows keyset pagination to work correctly for both tables.
Fixes: "Unknown column 'id' in 'order clause'" error when fetching ELABDATADISP
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The method was restricted to only RAWDATACOR, but the consolidation logic
works for both tables. Updated the check to allow both:
- RAWDATACOR
- ELABDATADISP
The keyset pagination (id-based WHERE clause) works identically for both
tables, and consolidation happens in Python for both.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The updated_at column was removed from the schema but should be kept for
consistency with the original table structure and to track when rows are
modified.
Changes:
- Added updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP to table schema
- Added updated_at to get_column_order() for elabdatadisp
- Added updated_at to transform_elabdatadisp_row() output
This maintains backward compatibility while still consolidating node_num,
state, and calc_err into the measurements JSONB.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Updated get_column_order() for elabdatadisp table to return only the
columns that are now stored separately:
- id_elab_data
- unit_name
- tool_name_id
- event_timestamp
- measurements (includes node_num, state, calc_err keyed by node)
- created_at
Removed: node_num, state, calc_err, updated_at (not used after consolidation)
This matches the schema defined in schema_transformer.py where these fields
are noted as being stored in the JSONB measurements column.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Add consolidation logic to ELABDATADISP similar to RAWDATACOR:
- Group rows by (unit_name, tool_name_id, event_timestamp)
- Consolidate multiple nodes with same timestamp into single row
- Store node_num, state, calc_err in JSONB measurements keyed by node
Changes:
1. Add _build_measurement_for_elabdatadisp_node() helper
- Builds measurement object with state, calc_err, and measurement categories
- Filters out empty categories to save space
2. Update transform_elabdatadisp_row() signature
- Accept optional measurements parameter for consolidated rows
- Build from single row if measurements not provided
- Remove node_num, state, calc_err from returned columns (now in JSONB)
- Keep only: id_elab_data, unit_name, tool_name_id, event_timestamp, measurements, created_at
3. Add consolidate_elabdatadisp_batch() method
- Group rows by consolidation key
- Build consolidated measurements with node numbers as keys
- Use MAX(idElabData) for checkpoint tracking (resume capability)
- Use MIN(idElabData) as template for other fields
4. Update transform_batch() to support ELABDATADISP consolidation
- Check consolidate flag for both tables
- Call consolidate_elabdatadisp_batch() when needed
Result: ELABDATADISP now consolidates ~5-10:1 like RAWDATACOR,
with all node data (node_num, state, calc_err, measurements) keyed
by node number in JSONB.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
