fix .env.example var and docs

This commit is contained in:
2025-12-30 15:29:26 +01:00
parent 5f6e3215a5
commit bcedae40fc
5 changed files with 31 additions and 16 deletions

View File

@@ -13,10 +13,10 @@ POSTGRES_PASSWORD=your_postgres_password
POSTGRES_DATABASE=migrated_db POSTGRES_DATABASE=migrated_db
# Migration Settings # Migration Settings
BATCH_SIZE=10000
LOG_LEVEL=INFO LOG_LEVEL=INFO
DRY_RUN=false DRY_RUN=false
CONSOLIDATION_GROUP_LIMIT=50000 CONSOLIDATION_GROUP_LIMIT=40000
PROGRESS_LOG_INTERVAL=10000
# Performance Testing # Performance Testing
BENCHMARK_OUTPUT_DIR=benchmark_results BENCHMARK_OUTPUT_DIR=benchmark_results

View File

@@ -15,7 +15,10 @@
- **Full migration**: Improved to save global `last_key` after completing all partitions - **Full migration**: Improved to save global `last_key` after completing all partitions
- **State tracking**: Moved from `migration_state.json` to PostgreSQL table `migration_state` - **State tracking**: Moved from `migration_state.json` to PostgreSQL table `migration_state`
- **Query performance**: Added `min_mysql_id` parameter to `fetch_consolidation_keys_after()` for optimization - **Query performance**: Added `min_mysql_id` parameter to `fetch_consolidation_keys_after()` for optimization
- **Documentation**: Updated README.md, MIGRATION_WORKFLOW.md, QUICKSTART.md with current implementation - **Configuration**: Renamed `BATCH_SIZE` to `CONSOLIDATION_GROUP_LIMIT` to better reflect what it controls
- **Configuration**: Added `PROGRESS_LOG_INTERVAL` to control logging frequency
- **Configuration**: Added `BENCHMARK_OUTPUT_DIR` to specify benchmark results directory
- **Documentation**: Updated README.md, MIGRATION_WORKFLOW.md, QUICKSTART.md, EXAMPLE_WORKFLOW.md with current implementation
### Removed ### Removed
- **migration_state.json**: Replaced by PostgreSQL table - **migration_state.json**: Replaced by PostgreSQL table
@@ -70,6 +73,7 @@ If you have an existing installation with `migration_state.json`:
- `--state-file` parameter removed from incremental migration (no longer uses JSON) - `--state-file` parameter removed from incremental migration (no longer uses JSON)
- `--use-id` flag removed (consolidation-based approach is now default) - `--use-id` flag removed (consolidation-based approach is now default)
- Incremental migration requires full migration to be run first - Incremental migration requires full migration to be run first
- `BATCH_SIZE` environment variable renamed to `CONSOLIDATION_GROUP_LIMIT` (update your .env file)
## [Previous] - Before 2025-12-30 ## [Previous] - Before 2025-12-30

View File

@@ -74,8 +74,9 @@ nano .env
# POSTGRES_PASSWORD=password123 # POSTGRES_PASSWORD=password123
# POSTGRES_DATABASE=production_migrated # POSTGRES_DATABASE=production_migrated
# #
# BATCH_SIZE=50000 # Large batches for speed
# LOG_LEVEL=INFO # LOG_LEVEL=INFO
# CONSOLIDATION_GROUP_LIMIT=80000 # Large batches for speed
# PROGRESS_LOG_INTERVAL=20000
``` ```
### 4. Verifica Configurazione (5 min) ### 4. Verifica Configurazione (5 min)
@@ -335,8 +336,11 @@ python main.py setup --create-schema
### Migrazione molto lenta ### Migrazione molto lenta
```bash ```bash
# Aumentare batch size temporaneamente # Aumentare consolidation group limit temporaneamente
# Editare .env: BATCH_SIZE=100000 # Editare .env: CONSOLIDATION_GROUP_LIMIT=100000
# Ridurre logging
# Editare .env: PROGRESS_LOG_INTERVAL=50000
# Oppure verificare: # Oppure verificare:
# - Latency rete MySQL↔PostgreSQL # - Latency rete MySQL↔PostgreSQL

View File

@@ -42,8 +42,9 @@ POSTGRES_USER=postgres
POSTGRES_PASSWORD=pgpassword POSTGRES_PASSWORD=pgpassword
POSTGRES_DATABASE=migrated_db POSTGRES_DATABASE=migrated_db
BATCH_SIZE=10000
LOG_LEVEL=INFO LOG_LEVEL=INFO
CONSOLIDATION_GROUP_LIMIT=40000
PROGRESS_LOG_INTERVAL=10000
``` ```
### 3. Creare PostgreSQL in Incus ### 3. Creare PostgreSQL in Incus
@@ -224,8 +225,11 @@ python main.py setup --create-schema
### "Migration is slow" ### "Migration is slow"
```bash ```bash
# Aumentare batch size in .env # Aumentare consolidation group limit in .env
BATCH_SIZE=50000 CONSOLIDATION_GROUP_LIMIT=80000
# Oppure ridurre logging
PROGRESS_LOG_INTERVAL=20000
# Oppure ottimizzare MySQL # Oppure ottimizzare MySQL
mysql> FLUSH PRIVILEGES; mysql> FLUSH PRIVILEGES;

View File

@@ -54,22 +54,24 @@ Modificare `.env` con i tuoi dettagli:
MYSQL_HOST=localhost MYSQL_HOST=localhost
MYSQL_PORT=3306 MYSQL_PORT=3306
MYSQL_USER=root MYSQL_USER=root
MYSQL_PASSWORD=your_password MYSQL_PASSWORD=your_mysql_password
MYSQL_DATABASE=your_database MYSQL_DATABASE=your_database_name
# PostgreSQL Target Database (container Incus) # PostgreSQL Target Database (container Incus)
POSTGRES_HOST=localhost POSTGRES_HOST=localhost
POSTGRES_PORT=5432 POSTGRES_PORT=5432
POSTGRES_USER=postgres POSTGRES_USER=postgres
POSTGRES_PASSWORD=your_password POSTGRES_PASSWORD=your_postgres_password
POSTGRES_DATABASE=migrated_db POSTGRES_DATABASE=migrated_db
# Migration Settings # Migration Settings
BATCH_SIZE=10000
LOG_LEVEL=INFO LOG_LEVEL=INFO
DRY_RUN=false DRY_RUN=false
CONSOLIDATION_GROUP_LIMIT=40000
PROGRESS_LOG_INTERVAL=10000
# Benchmark Settings # Performance Testing
BENCHMARK_OUTPUT_DIR=benchmark_results
BENCHMARK_ITERATIONS=5 BENCHMARK_ITERATIONS=5
``` ```
@@ -401,7 +403,7 @@ mysql2postgres/
- Verificare credenziali: `psql -h localhost -U postgres` - Verificare credenziali: `psql -h localhost -U postgres`
### Timeout durante migrazione ### Timeout durante migrazione
- Aumentare `BATCH_SIZE` in `.env` (default: 10000) - Aumentare `CONSOLIDATION_GROUP_LIMIT` in `.env` (default: 40000)
- Verificare performance di rete tra MySQL e PostgreSQL - Verificare performance di rete tra MySQL e PostgreSQL
### "No previous migration found" (incremental) ### "No previous migration found" (incremental)
@@ -427,7 +429,8 @@ mysql2postgres/
## Performance Tips ## Performance Tips
1. **Migration** 1. **Migration**
- Aumentare `BATCH_SIZE` per meno transazioni (es. 50000) - Aumentare `CONSOLIDATION_GROUP_LIMIT` per processare più chiavi per batch (default: 40000)
- Aumentare `PROGRESS_LOG_INTERVAL` per ridurre logging (default: 10000)
- Disabilitare indici durante migrazione se possibile (non implementato) - Disabilitare indici durante migrazione se possibile (non implementato)
2. **Queries on JSONB** 2. **Queries on JSONB**