diff --git a/.env.example b/.env.example index 72e7f80..e075983 100644 --- a/.env.example +++ b/.env.example @@ -13,10 +13,10 @@ POSTGRES_PASSWORD=your_postgres_password POSTGRES_DATABASE=migrated_db # Migration Settings -BATCH_SIZE=10000 LOG_LEVEL=INFO DRY_RUN=false -CONSOLIDATION_GROUP_LIMIT=50000 +CONSOLIDATION_GROUP_LIMIT=40000 +PROGRESS_LOG_INTERVAL=10000 # Performance Testing BENCHMARK_OUTPUT_DIR=benchmark_results diff --git a/CHANGELOG.md b/CHANGELOG.md index d8c075e..3972926 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,10 @@ - **Full migration**: Improved to save global `last_key` after completing all partitions - **State tracking**: Moved from `migration_state.json` to PostgreSQL table `migration_state` - **Query performance**: Added `min_mysql_id` parameter to `fetch_consolidation_keys_after()` for optimization -- **Documentation**: Updated README.md, MIGRATION_WORKFLOW.md, QUICKSTART.md with current implementation +- **Configuration**: Renamed `BATCH_SIZE` to `CONSOLIDATION_GROUP_LIMIT` to better reflect what it controls +- **Configuration**: Added `PROGRESS_LOG_INTERVAL` to control logging frequency +- **Configuration**: Added `BENCHMARK_OUTPUT_DIR` to specify benchmark results directory +- **Documentation**: Updated README.md, MIGRATION_WORKFLOW.md, QUICKSTART.md, EXAMPLE_WORKFLOW.md with current implementation ### Removed - **migration_state.json**: Replaced by PostgreSQL table @@ -70,6 +73,7 @@ If you have an existing installation with `migration_state.json`: - `--state-file` parameter removed from incremental migration (no longer uses JSON) - `--use-id` flag removed (consolidation-based approach is now default) - Incremental migration requires full migration to be run first +- `BATCH_SIZE` environment variable renamed to `CONSOLIDATION_GROUP_LIMIT` (update your .env file) ## [Previous] - Before 2025-12-30 diff --git a/EXAMPLE_WORKFLOW.md b/EXAMPLE_WORKFLOW.md index 682e217..6299749 100644 --- a/EXAMPLE_WORKFLOW.md +++ b/EXAMPLE_WORKFLOW.md @@ -74,8 +74,9 @@ nano .env # POSTGRES_PASSWORD=password123 # POSTGRES_DATABASE=production_migrated # -# BATCH_SIZE=50000 # Large batches for speed # LOG_LEVEL=INFO +# CONSOLIDATION_GROUP_LIMIT=80000 # Large batches for speed +# PROGRESS_LOG_INTERVAL=20000 ``` ### 4. Verifica Configurazione (5 min) @@ -335,8 +336,11 @@ python main.py setup --create-schema ### Migrazione molto lenta ```bash -# Aumentare batch size temporaneamente -# Editare .env: BATCH_SIZE=100000 +# Aumentare consolidation group limit temporaneamente +# Editare .env: CONSOLIDATION_GROUP_LIMIT=100000 + +# Ridurre logging +# Editare .env: PROGRESS_LOG_INTERVAL=50000 # Oppure verificare: # - Latency rete MySQL↔PostgreSQL diff --git a/QUICKSTART.md b/QUICKSTART.md index d8165ee..fa019ec 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -42,8 +42,9 @@ POSTGRES_USER=postgres POSTGRES_PASSWORD=pgpassword POSTGRES_DATABASE=migrated_db -BATCH_SIZE=10000 LOG_LEVEL=INFO +CONSOLIDATION_GROUP_LIMIT=40000 +PROGRESS_LOG_INTERVAL=10000 ``` ### 3. Creare PostgreSQL in Incus @@ -224,8 +225,11 @@ python main.py setup --create-schema ### "Migration is slow" ```bash -# Aumentare batch size in .env -BATCH_SIZE=50000 +# Aumentare consolidation group limit in .env +CONSOLIDATION_GROUP_LIMIT=80000 + +# Oppure ridurre logging +PROGRESS_LOG_INTERVAL=20000 # Oppure ottimizzare MySQL mysql> FLUSH PRIVILEGES; diff --git a/README.md b/README.md index 6a84481..3af958d 100644 --- a/README.md +++ b/README.md @@ -54,22 +54,24 @@ Modificare `.env` con i tuoi dettagli: MYSQL_HOST=localhost MYSQL_PORT=3306 MYSQL_USER=root -MYSQL_PASSWORD=your_password -MYSQL_DATABASE=your_database +MYSQL_PASSWORD=your_mysql_password +MYSQL_DATABASE=your_database_name # PostgreSQL Target Database (container Incus) POSTGRES_HOST=localhost POSTGRES_PORT=5432 POSTGRES_USER=postgres -POSTGRES_PASSWORD=your_password +POSTGRES_PASSWORD=your_postgres_password POSTGRES_DATABASE=migrated_db # Migration Settings -BATCH_SIZE=10000 LOG_LEVEL=INFO DRY_RUN=false +CONSOLIDATION_GROUP_LIMIT=40000 +PROGRESS_LOG_INTERVAL=10000 -# Benchmark Settings +# Performance Testing +BENCHMARK_OUTPUT_DIR=benchmark_results BENCHMARK_ITERATIONS=5 ``` @@ -401,7 +403,7 @@ mysql2postgres/ - Verificare credenziali: `psql -h localhost -U postgres` ### Timeout durante migrazione -- Aumentare `BATCH_SIZE` in `.env` (default: 10000) +- Aumentare `CONSOLIDATION_GROUP_LIMIT` in `.env` (default: 40000) - Verificare performance di rete tra MySQL e PostgreSQL ### "No previous migration found" (incremental) @@ -427,7 +429,8 @@ mysql2postgres/ ## Performance Tips 1. **Migration** - - Aumentare `BATCH_SIZE` per meno transazioni (es. 50000) + - Aumentare `CONSOLIDATION_GROUP_LIMIT` per processare più chiavi per batch (default: 40000) + - Aumentare `PROGRESS_LOG_INTERVAL` per ridurre logging (default: 10000) - Disabilitare indici durante migrazione se possibile (non implementato) 2. **Queries on JSONB**