Commit Graph

9 Commits

Author SHA1 Message Date
e381618255 fix: Support both uppercase and lowercase table names in TABLE_CONFIGS
- TABLE_CONFIGS now accepts both 'RAWDATACOR' and 'rawdatacor' as keys
- TABLE_CONFIGS now accepts both 'ELABDATADISP' and 'elabdatadisp' as keys
- Reuse same config dict for both cases to avoid duplication

This allows FullMigrator to work correctly when initialized with
uppercase table names from the CLI while DataTransformer works
with lowercase names.

Fixes: 'Unknown table: RAWDATACOR' error during migration
2025-12-10 20:28:19 +01:00
de6bde17c9 feat: Add sequences for auto-incrementing IDs
- Create rawdatacor_id_seq for auto-increment of id column
- Create elabdatadisp_id_seq for auto-increment of id_elab_data column
- Both sequences use DEFAULT nextval() to auto-generate IDs on insert

This replaces PRIMARY KEY functionality since PostgreSQL doesn't
support PRIMARY KEY on partitioned tables with expression-based ranges.
IDs are now auto-incremented without primary key constraint.

Tested: schema creation works correctly with sequences
2025-12-10 20:20:52 +01:00
2834f8b578 fix: Remove unsupported constraints from partitioned tables
PostgreSQL doesn't support PRIMARY KEY or UNIQUE constraints on
partitioned tables when using RANGE partitioning on expressions
(like EXTRACT(YEAR FROM event_date)).

Changed:
- RAWDATACOR: removed PRIMARY KEY (id, event_date) and UNIQUE constraint
- ELABDATADISP: removed PRIMARY KEY (id_elab_data, event_date) and UNIQUE constraint
- Tables now have no constraints except NOT NULL on required columns

This is a PostgreSQL limitation with partitioned tables.
Constraints can be added per-partition if needed, but for simplicity
we rely on application-level validation.

Fixes: 'vincolo PRIMARY KEY non supportato con una definizione di chiave di partizione'
2025-12-10 20:18:20 +01:00
410b253808 fix: Update Pydantic v2 configuration for .env loading
- Fix ConfigDict model_config for Pydantic v2.12+ compatibility
- Add env_file and env_file_encoding to all config classes
- Each config class now properly loads from .env with correct prefix

Fixes: ValidationError when loading settings from .env file
CLI now works correctly with 'uv run python main.py'
2025-12-10 20:11:12 +01:00
9b18db029b docs: Add quick navigation guide (START_HERE.md) 2025-12-10 20:00:50 +01:00
8e705e33da docs: Add detailed example workflow 2025-12-10 19:59:22 +01:00
38c6b4c6d8 docs: Add implementation summary 2025-12-10 19:58:49 +01:00
fccc83eb74 docs: Add comprehensive documentation and helper scripts
Add:
- QUICKSTART.md: 5-minute quick start guide with examples
- scripts/incus_setup.sh: Automated PostgreSQL container setup
- scripts/validate_migration.sql: SQL validation queries
- scripts/setup_cron.sh: Cron job setup for incremental migrations
- tests/test_setup.py: Unit tests for configuration and transformation
- install.sh: Quick installation script

Documentation includes:
- Step-by-step setup instructions
- Example queries for RAWDATACOR and ELABDATADISP
- Troubleshooting guide
- Performance optimization tips

🤖 Generated with Claude Code

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2025-12-10 19:58:20 +01:00
62577d3200 feat: Add MySQL to PostgreSQL migration tool with JSONB transformation
Implement comprehensive migration solution with:
- Full and incremental migration modes
- JSONB schema transformation for RAWDATACOR and ELABDATADISP tables
- Native PostgreSQL partitioning (2014-2031)
- Optimized GIN indexes for JSONB queries
- Rich logging with progress tracking
- Complete benchmark system for MySQL vs PostgreSQL comparison
- CLI interface with multiple commands (setup, migrate, benchmark)
- Configuration management via .env file
- Error handling and retry logic
- Batch processing for performance (configurable batch size)

Database transformations:
- RAWDATACOR: 16 Val columns + units → single JSONB measurements
- ELABDATADISP: 25+ measurement fields → structured JSONB with categories

🤖 Generated with Claude Code

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2025-12-10 19:57:11 +01:00