fix .env.example var and docs
This commit is contained in:
@@ -13,10 +13,10 @@ POSTGRES_PASSWORD=your_postgres_password
|
||||
POSTGRES_DATABASE=migrated_db
|
||||
|
||||
# Migration Settings
|
||||
BATCH_SIZE=10000
|
||||
LOG_LEVEL=INFO
|
||||
DRY_RUN=false
|
||||
CONSOLIDATION_GROUP_LIMIT=50000
|
||||
CONSOLIDATION_GROUP_LIMIT=40000
|
||||
PROGRESS_LOG_INTERVAL=10000
|
||||
|
||||
# Performance Testing
|
||||
BENCHMARK_OUTPUT_DIR=benchmark_results
|
||||
|
||||
@@ -15,7 +15,10 @@
|
||||
- **Full migration**: Improved to save global `last_key` after completing all partitions
|
||||
- **State tracking**: Moved from `migration_state.json` to PostgreSQL table `migration_state`
|
||||
- **Query performance**: Added `min_mysql_id` parameter to `fetch_consolidation_keys_after()` for optimization
|
||||
- **Documentation**: Updated README.md, MIGRATION_WORKFLOW.md, QUICKSTART.md with current implementation
|
||||
- **Configuration**: Renamed `BATCH_SIZE` to `CONSOLIDATION_GROUP_LIMIT` to better reflect what it controls
|
||||
- **Configuration**: Added `PROGRESS_LOG_INTERVAL` to control logging frequency
|
||||
- **Configuration**: Added `BENCHMARK_OUTPUT_DIR` to specify benchmark results directory
|
||||
- **Documentation**: Updated README.md, MIGRATION_WORKFLOW.md, QUICKSTART.md, EXAMPLE_WORKFLOW.md with current implementation
|
||||
|
||||
### Removed
|
||||
- **migration_state.json**: Replaced by PostgreSQL table
|
||||
@@ -70,6 +73,7 @@ If you have an existing installation with `migration_state.json`:
|
||||
- `--state-file` parameter removed from incremental migration (no longer uses JSON)
|
||||
- `--use-id` flag removed (consolidation-based approach is now default)
|
||||
- Incremental migration requires full migration to be run first
|
||||
- `BATCH_SIZE` environment variable renamed to `CONSOLIDATION_GROUP_LIMIT` (update your .env file)
|
||||
|
||||
## [Previous] - Before 2025-12-30
|
||||
|
||||
|
||||
@@ -74,8 +74,9 @@ nano .env
|
||||
# POSTGRES_PASSWORD=password123
|
||||
# POSTGRES_DATABASE=production_migrated
|
||||
#
|
||||
# BATCH_SIZE=50000 # Large batches for speed
|
||||
# LOG_LEVEL=INFO
|
||||
# CONSOLIDATION_GROUP_LIMIT=80000 # Large batches for speed
|
||||
# PROGRESS_LOG_INTERVAL=20000
|
||||
```
|
||||
|
||||
### 4. Verifica Configurazione (5 min)
|
||||
@@ -335,8 +336,11 @@ python main.py setup --create-schema
|
||||
|
||||
### Migrazione molto lenta
|
||||
```bash
|
||||
# Aumentare batch size temporaneamente
|
||||
# Editare .env: BATCH_SIZE=100000
|
||||
# Aumentare consolidation group limit temporaneamente
|
||||
# Editare .env: CONSOLIDATION_GROUP_LIMIT=100000
|
||||
|
||||
# Ridurre logging
|
||||
# Editare .env: PROGRESS_LOG_INTERVAL=50000
|
||||
|
||||
# Oppure verificare:
|
||||
# - Latency rete MySQL↔PostgreSQL
|
||||
|
||||
@@ -42,8 +42,9 @@ POSTGRES_USER=postgres
|
||||
POSTGRES_PASSWORD=pgpassword
|
||||
POSTGRES_DATABASE=migrated_db
|
||||
|
||||
BATCH_SIZE=10000
|
||||
LOG_LEVEL=INFO
|
||||
CONSOLIDATION_GROUP_LIMIT=40000
|
||||
PROGRESS_LOG_INTERVAL=10000
|
||||
```
|
||||
|
||||
### 3. Creare PostgreSQL in Incus
|
||||
@@ -224,8 +225,11 @@ python main.py setup --create-schema
|
||||
|
||||
### "Migration is slow"
|
||||
```bash
|
||||
# Aumentare batch size in .env
|
||||
BATCH_SIZE=50000
|
||||
# Aumentare consolidation group limit in .env
|
||||
CONSOLIDATION_GROUP_LIMIT=80000
|
||||
|
||||
# Oppure ridurre logging
|
||||
PROGRESS_LOG_INTERVAL=20000
|
||||
|
||||
# Oppure ottimizzare MySQL
|
||||
mysql> FLUSH PRIVILEGES;
|
||||
|
||||
17
README.md
17
README.md
@@ -54,22 +54,24 @@ Modificare `.env` con i tuoi dettagli:
|
||||
MYSQL_HOST=localhost
|
||||
MYSQL_PORT=3306
|
||||
MYSQL_USER=root
|
||||
MYSQL_PASSWORD=your_password
|
||||
MYSQL_DATABASE=your_database
|
||||
MYSQL_PASSWORD=your_mysql_password
|
||||
MYSQL_DATABASE=your_database_name
|
||||
|
||||
# PostgreSQL Target Database (container Incus)
|
||||
POSTGRES_HOST=localhost
|
||||
POSTGRES_PORT=5432
|
||||
POSTGRES_USER=postgres
|
||||
POSTGRES_PASSWORD=your_password
|
||||
POSTGRES_PASSWORD=your_postgres_password
|
||||
POSTGRES_DATABASE=migrated_db
|
||||
|
||||
# Migration Settings
|
||||
BATCH_SIZE=10000
|
||||
LOG_LEVEL=INFO
|
||||
DRY_RUN=false
|
||||
CONSOLIDATION_GROUP_LIMIT=40000
|
||||
PROGRESS_LOG_INTERVAL=10000
|
||||
|
||||
# Benchmark Settings
|
||||
# Performance Testing
|
||||
BENCHMARK_OUTPUT_DIR=benchmark_results
|
||||
BENCHMARK_ITERATIONS=5
|
||||
```
|
||||
|
||||
@@ -401,7 +403,7 @@ mysql2postgres/
|
||||
- Verificare credenziali: `psql -h localhost -U postgres`
|
||||
|
||||
### Timeout durante migrazione
|
||||
- Aumentare `BATCH_SIZE` in `.env` (default: 10000)
|
||||
- Aumentare `CONSOLIDATION_GROUP_LIMIT` in `.env` (default: 40000)
|
||||
- Verificare performance di rete tra MySQL e PostgreSQL
|
||||
|
||||
### "No previous migration found" (incremental)
|
||||
@@ -427,7 +429,8 @@ mysql2postgres/
|
||||
## Performance Tips
|
||||
|
||||
1. **Migration**
|
||||
- Aumentare `BATCH_SIZE` per meno transazioni (es. 50000)
|
||||
- Aumentare `CONSOLIDATION_GROUP_LIMIT` per processare più chiavi per batch (default: 40000)
|
||||
- Aumentare `PROGRESS_LOG_INTERVAL` per ridurre logging (default: 10000)
|
||||
- Disabilitare indici durante migrazione se possibile (non implementato)
|
||||
|
||||
2. **Queries on JSONB**
|
||||
|
||||
Reference in New Issue
Block a user