fix .env.example var and docs
This commit is contained in:
@@ -13,10 +13,10 @@ POSTGRES_PASSWORD=your_postgres_password
|
|||||||
POSTGRES_DATABASE=migrated_db
|
POSTGRES_DATABASE=migrated_db
|
||||||
|
|
||||||
# Migration Settings
|
# Migration Settings
|
||||||
BATCH_SIZE=10000
|
|
||||||
LOG_LEVEL=INFO
|
LOG_LEVEL=INFO
|
||||||
DRY_RUN=false
|
DRY_RUN=false
|
||||||
CONSOLIDATION_GROUP_LIMIT=50000
|
CONSOLIDATION_GROUP_LIMIT=40000
|
||||||
|
PROGRESS_LOG_INTERVAL=10000
|
||||||
|
|
||||||
# Performance Testing
|
# Performance Testing
|
||||||
BENCHMARK_OUTPUT_DIR=benchmark_results
|
BENCHMARK_OUTPUT_DIR=benchmark_results
|
||||||
|
|||||||
@@ -15,7 +15,10 @@
|
|||||||
- **Full migration**: Improved to save global `last_key` after completing all partitions
|
- **Full migration**: Improved to save global `last_key` after completing all partitions
|
||||||
- **State tracking**: Moved from `migration_state.json` to PostgreSQL table `migration_state`
|
- **State tracking**: Moved from `migration_state.json` to PostgreSQL table `migration_state`
|
||||||
- **Query performance**: Added `min_mysql_id` parameter to `fetch_consolidation_keys_after()` for optimization
|
- **Query performance**: Added `min_mysql_id` parameter to `fetch_consolidation_keys_after()` for optimization
|
||||||
- **Documentation**: Updated README.md, MIGRATION_WORKFLOW.md, QUICKSTART.md with current implementation
|
- **Configuration**: Renamed `BATCH_SIZE` to `CONSOLIDATION_GROUP_LIMIT` to better reflect what it controls
|
||||||
|
- **Configuration**: Added `PROGRESS_LOG_INTERVAL` to control logging frequency
|
||||||
|
- **Configuration**: Added `BENCHMARK_OUTPUT_DIR` to specify benchmark results directory
|
||||||
|
- **Documentation**: Updated README.md, MIGRATION_WORKFLOW.md, QUICKSTART.md, EXAMPLE_WORKFLOW.md with current implementation
|
||||||
|
|
||||||
### Removed
|
### Removed
|
||||||
- **migration_state.json**: Replaced by PostgreSQL table
|
- **migration_state.json**: Replaced by PostgreSQL table
|
||||||
@@ -70,6 +73,7 @@ If you have an existing installation with `migration_state.json`:
|
|||||||
- `--state-file` parameter removed from incremental migration (no longer uses JSON)
|
- `--state-file` parameter removed from incremental migration (no longer uses JSON)
|
||||||
- `--use-id` flag removed (consolidation-based approach is now default)
|
- `--use-id` flag removed (consolidation-based approach is now default)
|
||||||
- Incremental migration requires full migration to be run first
|
- Incremental migration requires full migration to be run first
|
||||||
|
- `BATCH_SIZE` environment variable renamed to `CONSOLIDATION_GROUP_LIMIT` (update your .env file)
|
||||||
|
|
||||||
## [Previous] - Before 2025-12-30
|
## [Previous] - Before 2025-12-30
|
||||||
|
|
||||||
|
|||||||
@@ -74,8 +74,9 @@ nano .env
|
|||||||
# POSTGRES_PASSWORD=password123
|
# POSTGRES_PASSWORD=password123
|
||||||
# POSTGRES_DATABASE=production_migrated
|
# POSTGRES_DATABASE=production_migrated
|
||||||
#
|
#
|
||||||
# BATCH_SIZE=50000 # Large batches for speed
|
|
||||||
# LOG_LEVEL=INFO
|
# LOG_LEVEL=INFO
|
||||||
|
# CONSOLIDATION_GROUP_LIMIT=80000 # Large batches for speed
|
||||||
|
# PROGRESS_LOG_INTERVAL=20000
|
||||||
```
|
```
|
||||||
|
|
||||||
### 4. Verifica Configurazione (5 min)
|
### 4. Verifica Configurazione (5 min)
|
||||||
@@ -335,8 +336,11 @@ python main.py setup --create-schema
|
|||||||
|
|
||||||
### Migrazione molto lenta
|
### Migrazione molto lenta
|
||||||
```bash
|
```bash
|
||||||
# Aumentare batch size temporaneamente
|
# Aumentare consolidation group limit temporaneamente
|
||||||
# Editare .env: BATCH_SIZE=100000
|
# Editare .env: CONSOLIDATION_GROUP_LIMIT=100000
|
||||||
|
|
||||||
|
# Ridurre logging
|
||||||
|
# Editare .env: PROGRESS_LOG_INTERVAL=50000
|
||||||
|
|
||||||
# Oppure verificare:
|
# Oppure verificare:
|
||||||
# - Latency rete MySQL↔PostgreSQL
|
# - Latency rete MySQL↔PostgreSQL
|
||||||
|
|||||||
@@ -42,8 +42,9 @@ POSTGRES_USER=postgres
|
|||||||
POSTGRES_PASSWORD=pgpassword
|
POSTGRES_PASSWORD=pgpassword
|
||||||
POSTGRES_DATABASE=migrated_db
|
POSTGRES_DATABASE=migrated_db
|
||||||
|
|
||||||
BATCH_SIZE=10000
|
|
||||||
LOG_LEVEL=INFO
|
LOG_LEVEL=INFO
|
||||||
|
CONSOLIDATION_GROUP_LIMIT=40000
|
||||||
|
PROGRESS_LOG_INTERVAL=10000
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3. Creare PostgreSQL in Incus
|
### 3. Creare PostgreSQL in Incus
|
||||||
@@ -224,8 +225,11 @@ python main.py setup --create-schema
|
|||||||
|
|
||||||
### "Migration is slow"
|
### "Migration is slow"
|
||||||
```bash
|
```bash
|
||||||
# Aumentare batch size in .env
|
# Aumentare consolidation group limit in .env
|
||||||
BATCH_SIZE=50000
|
CONSOLIDATION_GROUP_LIMIT=80000
|
||||||
|
|
||||||
|
# Oppure ridurre logging
|
||||||
|
PROGRESS_LOG_INTERVAL=20000
|
||||||
|
|
||||||
# Oppure ottimizzare MySQL
|
# Oppure ottimizzare MySQL
|
||||||
mysql> FLUSH PRIVILEGES;
|
mysql> FLUSH PRIVILEGES;
|
||||||
|
|||||||
17
README.md
17
README.md
@@ -54,22 +54,24 @@ Modificare `.env` con i tuoi dettagli:
|
|||||||
MYSQL_HOST=localhost
|
MYSQL_HOST=localhost
|
||||||
MYSQL_PORT=3306
|
MYSQL_PORT=3306
|
||||||
MYSQL_USER=root
|
MYSQL_USER=root
|
||||||
MYSQL_PASSWORD=your_password
|
MYSQL_PASSWORD=your_mysql_password
|
||||||
MYSQL_DATABASE=your_database
|
MYSQL_DATABASE=your_database_name
|
||||||
|
|
||||||
# PostgreSQL Target Database (container Incus)
|
# PostgreSQL Target Database (container Incus)
|
||||||
POSTGRES_HOST=localhost
|
POSTGRES_HOST=localhost
|
||||||
POSTGRES_PORT=5432
|
POSTGRES_PORT=5432
|
||||||
POSTGRES_USER=postgres
|
POSTGRES_USER=postgres
|
||||||
POSTGRES_PASSWORD=your_password
|
POSTGRES_PASSWORD=your_postgres_password
|
||||||
POSTGRES_DATABASE=migrated_db
|
POSTGRES_DATABASE=migrated_db
|
||||||
|
|
||||||
# Migration Settings
|
# Migration Settings
|
||||||
BATCH_SIZE=10000
|
|
||||||
LOG_LEVEL=INFO
|
LOG_LEVEL=INFO
|
||||||
DRY_RUN=false
|
DRY_RUN=false
|
||||||
|
CONSOLIDATION_GROUP_LIMIT=40000
|
||||||
|
PROGRESS_LOG_INTERVAL=10000
|
||||||
|
|
||||||
# Benchmark Settings
|
# Performance Testing
|
||||||
|
BENCHMARK_OUTPUT_DIR=benchmark_results
|
||||||
BENCHMARK_ITERATIONS=5
|
BENCHMARK_ITERATIONS=5
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -401,7 +403,7 @@ mysql2postgres/
|
|||||||
- Verificare credenziali: `psql -h localhost -U postgres`
|
- Verificare credenziali: `psql -h localhost -U postgres`
|
||||||
|
|
||||||
### Timeout durante migrazione
|
### Timeout durante migrazione
|
||||||
- Aumentare `BATCH_SIZE` in `.env` (default: 10000)
|
- Aumentare `CONSOLIDATION_GROUP_LIMIT` in `.env` (default: 40000)
|
||||||
- Verificare performance di rete tra MySQL e PostgreSQL
|
- Verificare performance di rete tra MySQL e PostgreSQL
|
||||||
|
|
||||||
### "No previous migration found" (incremental)
|
### "No previous migration found" (incremental)
|
||||||
@@ -427,7 +429,8 @@ mysql2postgres/
|
|||||||
## Performance Tips
|
## Performance Tips
|
||||||
|
|
||||||
1. **Migration**
|
1. **Migration**
|
||||||
- Aumentare `BATCH_SIZE` per meno transazioni (es. 50000)
|
- Aumentare `CONSOLIDATION_GROUP_LIMIT` per processare più chiavi per batch (default: 40000)
|
||||||
|
- Aumentare `PROGRESS_LOG_INTERVAL` per ridurre logging (default: 10000)
|
||||||
- Disabilitare indici durante migrazione se possibile (non implementato)
|
- Disabilitare indici durante migrazione se possibile (non implementato)
|
||||||
|
|
||||||
2. **Queries on JSONB**
|
2. **Queries on JSONB**
|
||||||
|
|||||||
Reference in New Issue
Block a user