initial working
This commit is contained in:
381
vm1/src/refactory_scripts/TODO_TS_PINI.md
Normal file
381
vm1/src/refactory_scripts/TODO_TS_PINI.md
Normal file
@@ -0,0 +1,381 @@
|
||||
# TS Pini Loader - TODO for Complete Refactoring
|
||||
|
||||
## Status: Essential Refactoring Complete ✅
|
||||
|
||||
**Current Implementation**: 508 lines
|
||||
**Legacy Script**: 2,587 lines
|
||||
**Reduction**: 80% (from monolithic to modular)
|
||||
|
||||
---
|
||||
|
||||
## ✅ Implemented Features
|
||||
|
||||
### Core Functionality
|
||||
- [x] Async/await architecture with aiomysql
|
||||
- [x] Multiple station type support (Leica, Trimble S7, S9, S7-inverted)
|
||||
- [x] Coordinate system transformations:
|
||||
- [x] CH1903 (Old Swiss system)
|
||||
- [x] CH1903+ / LV95 (New Swiss system via EPSG)
|
||||
- [x] UTM (Universal Transverse Mercator)
|
||||
- [x] Lat/Lon (direct)
|
||||
- [x] Project/folder name mapping (16 special cases)
|
||||
- [x] CSV parsing for different station formats
|
||||
- [x] ELABDATAUPGEO data insertion
|
||||
- [x] Basic mira (target point) lookup
|
||||
- [x] Proper logging and error handling
|
||||
- [x] Type hints and comprehensive docstrings
|
||||
|
||||
---
|
||||
|
||||
## ⏳ TODO: High Priority
|
||||
|
||||
### 1. Mira Creation Logic
|
||||
**File**: `ts_pini_loader.py`, method `_get_or_create_mira()`
|
||||
**Lines in legacy**: 138-160
|
||||
|
||||
**Current Status**: Stub implementation
|
||||
**What's needed**:
|
||||
```python
|
||||
async def _get_or_create_mira(self, mira_name: str, lavoro_id: int, site_id: int) -> int | None:
|
||||
# 1. Check if mira already exists (DONE)
|
||||
|
||||
# 2. If not, check company mira limits
|
||||
query = """
|
||||
SELECT c.id, c.upgeo_numero_mire, c.upgeo_numero_mireTot
|
||||
FROM companies as c
|
||||
JOIN sites as s ON c.id = s.company_id
|
||||
WHERE s.id = %s
|
||||
"""
|
||||
|
||||
# 3. If under limit, create mira
|
||||
if upgeo_numero_mire < upgeo_numero_mireTot:
|
||||
# INSERT INTO upgeo_mire
|
||||
# UPDATE companies mira counter
|
||||
|
||||
# 4. Return mira_id
|
||||
```
|
||||
|
||||
**Complexity**: Medium
|
||||
**Estimated time**: 30 minutes
|
||||
|
||||
---
|
||||
|
||||
### 2. Multi-Level Alarm System
|
||||
**File**: `ts_pini_loader.py`, method `_process_thresholds_and_alarms()`
|
||||
**Lines in legacy**: 174-1500+ (most of the script!)
|
||||
|
||||
**Current Status**: Stub with warning message
|
||||
**What's needed**:
|
||||
|
||||
#### 2.1 Threshold Configuration Loading
|
||||
```python
|
||||
class ThresholdConfig:
|
||||
"""Threshold configuration for a monitored point."""
|
||||
|
||||
# 5 dimensions x 3 levels = 15 thresholds
|
||||
attention_N: float | None
|
||||
intervention_N: float | None
|
||||
immediate_N: float | None
|
||||
|
||||
attention_E: float | None
|
||||
intervention_E: float | None
|
||||
immediate_E: float | None
|
||||
|
||||
attention_H: float | None
|
||||
intervention_H: float | None
|
||||
immediate_H: float | None
|
||||
|
||||
attention_R2D: float | None
|
||||
intervention_R2D: float | None
|
||||
immediate_R2D: float | None
|
||||
|
||||
attention_R3D: float | None
|
||||
intervention_R3D: float | None
|
||||
immediate_R3D: float | None
|
||||
|
||||
# Notification settings (3 levels x 5 dimensions x 2 channels)
|
||||
email_level_1_N: bool
|
||||
sms_level_1_N: bool
|
||||
# ... (30 fields total)
|
||||
```
|
||||
|
||||
#### 2.2 Displacement Calculation
|
||||
```python
|
||||
async def _calculate_displacements(self, mira_id: int) -> dict:
|
||||
"""
|
||||
Calculate displacements in all dimensions.
|
||||
|
||||
Returns dict with:
|
||||
- dN: displacement in North
|
||||
- dE: displacement in East
|
||||
- dH: displacement in Height
|
||||
- dR2D: 2D displacement (sqrt(dN² + dE²))
|
||||
- dR3D: 3D displacement (sqrt(dN² + dE² + dH²))
|
||||
- timestamp: current measurement time
|
||||
- previous_timestamp: baseline measurement time
|
||||
"""
|
||||
```
|
||||
|
||||
#### 2.3 Alarm Creation
|
||||
```python
|
||||
async def _create_alarm_if_threshold_exceeded(
|
||||
self,
|
||||
mira_id: int,
|
||||
dimension: str, # 'N', 'E', 'H', 'R2D', 'R3D'
|
||||
level: int, # 1, 2, 3
|
||||
value: float,
|
||||
threshold: float,
|
||||
config: ThresholdConfig
|
||||
) -> None:
|
||||
"""Create alarm in database if not already exists."""
|
||||
|
||||
# Check if alarm already exists for this mira/dimension/level
|
||||
# If not, INSERT INTO alarms
|
||||
# Send email/SMS based on config
|
||||
```
|
||||
|
||||
**Complexity**: High
|
||||
**Estimated time**: 4-6 hours
|
||||
**Dependencies**: Email/SMS sending infrastructure
|
||||
|
||||
---
|
||||
|
||||
### 3. Multiple Date Range Support
|
||||
**Lines in legacy**: Throughout alarm processing
|
||||
|
||||
**Current Status**: Not implemented
|
||||
**What's needed**:
|
||||
- Parse `multipleDateRange` JSON field from mira config
|
||||
- Apply different thresholds for different time periods
|
||||
- Handle overlapping ranges
|
||||
|
||||
**Complexity**: Medium
|
||||
**Estimated time**: 1-2 hours
|
||||
|
||||
---
|
||||
|
||||
## ⏳ TODO: Medium Priority
|
||||
|
||||
### 4. Additional Monitoring Types
|
||||
|
||||
#### 4.1 Railway Monitoring
|
||||
**Lines in legacy**: 1248-1522
|
||||
**What it does**: Special monitoring for railway tracks (binari)
|
||||
- Groups miras by railway identifier
|
||||
- Calculates transverse displacements
|
||||
- Different threshold logic
|
||||
|
||||
#### 4.2 Wall Monitoring (Muri)
|
||||
**Lines in legacy**: ~500-800
|
||||
**What it does**: Wall-specific monitoring with paired points
|
||||
|
||||
#### 4.3 Truss Monitoring (Tralicci)
|
||||
**Lines in legacy**: ~300-500
|
||||
**What it does**: Truss structure monitoring
|
||||
|
||||
**Approach**: Create separate classes:
|
||||
```python
|
||||
class RailwayMonitor:
|
||||
async def process(self, lavoro_id: int, miras: list[int]) -> None:
|
||||
...
|
||||
|
||||
class WallMonitor:
|
||||
async def process(self, lavoro_id: int, miras: list[int]) -> None:
|
||||
...
|
||||
|
||||
class TrussMonitor:
|
||||
async def process(self, lavoro_id: int, miras: list[int]) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
**Complexity**: High
|
||||
**Estimated time**: 3-4 hours each
|
||||
|
||||
---
|
||||
|
||||
### 5. Time-Series Analysis
|
||||
**Lines in legacy**: Multiple occurrences with `find_nearest_element()`
|
||||
|
||||
**Current Status**: Helper functions not ported
|
||||
**What's needed**:
|
||||
- Find nearest measurement in time series
|
||||
- Compare current vs. historical values
|
||||
- Detect trend changes
|
||||
|
||||
**Complexity**: Low-Medium
|
||||
**Estimated time**: 1 hour
|
||||
|
||||
---
|
||||
|
||||
## ⏳ TODO: Low Priority (Nice to Have)
|
||||
|
||||
### 6. Progressive Monitoring
|
||||
**Lines in legacy**: ~1100-1300
|
||||
**What it does**: Special handling for "progressive" type miras
|
||||
- Different calculation methods
|
||||
- Integration with externa data sources
|
||||
|
||||
**Complexity**: Medium
|
||||
**Estimated time**: 2 hours
|
||||
|
||||
---
|
||||
|
||||
### 7. Performance Optimizations
|
||||
|
||||
#### 7.1 Batch Operations
|
||||
Currently processes one point at a time. Could batch:
|
||||
- Coordinate transformations
|
||||
- Database inserts
|
||||
- Threshold checks
|
||||
|
||||
**Estimated speedup**: 2-3x
|
||||
|
||||
#### 7.2 Caching
|
||||
Cache frequently accessed data:
|
||||
- Threshold configurations
|
||||
- Company limits
|
||||
- Project metadata
|
||||
|
||||
**Estimated speedup**: 1.5-2x
|
||||
|
||||
---
|
||||
|
||||
### 8. Testing
|
||||
|
||||
#### 8.1 Unit Tests
|
||||
```python
|
||||
tests/test_ts_pini_loader.py:
|
||||
- test_coordinate_transformations()
|
||||
- test_station_type_parsing()
|
||||
- test_threshold_checking()
|
||||
- test_alarm_creation()
|
||||
```
|
||||
|
||||
#### 8.2 Integration Tests
|
||||
- Test with real CSV files
|
||||
- Test with mock database
|
||||
- Test coordinate edge cases (hemispheres, zones)
|
||||
|
||||
**Estimated time**: 3-4 hours
|
||||
|
||||
---
|
||||
|
||||
## 📋 Migration Strategy
|
||||
|
||||
### Phase 1: Core + Alarms (Recommended Next Step)
|
||||
1. Implement mira creation logic (30 min)
|
||||
2. Implement basic alarm system (4-6 hours)
|
||||
3. Test with real data
|
||||
4. Deploy alongside legacy script
|
||||
|
||||
**Total time**: ~1 working day
|
||||
**Value**: 80% of use cases covered
|
||||
|
||||
### Phase 2: Additional Monitoring
|
||||
5. Implement railway monitoring (3-4 hours)
|
||||
6. Implement wall monitoring (3-4 hours)
|
||||
7. Implement truss monitoring (3-4 hours)
|
||||
|
||||
**Total time**: 1.5-2 working days
|
||||
**Value**: 95% of use cases covered
|
||||
|
||||
### Phase 3: Polish & Optimization
|
||||
8. Add time-series analysis
|
||||
9. Performance optimizations
|
||||
10. Comprehensive testing
|
||||
11. Documentation updates
|
||||
|
||||
**Total time**: 1 working day
|
||||
**Value**: Production-ready, maintainable code
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Development Tips
|
||||
|
||||
### Working with Legacy Code
|
||||
The legacy script has:
|
||||
- **Deeply nested logic**: Up to 8 levels of indentation
|
||||
- **Repeated code**: Same patterns for 15 threshold checks
|
||||
- **Magic numbers**: Hardcoded values throughout
|
||||
- **Global state**: Variables used across 1000+ lines
|
||||
|
||||
**Refactoring approach**:
|
||||
1. Extract one feature at a time
|
||||
2. Write unit test first
|
||||
3. Refactor to pass test
|
||||
4. Integrate with main loader
|
||||
|
||||
### Testing Coordinate Transformations
|
||||
```python
|
||||
# Test data from legacy script
|
||||
test_cases = [
|
||||
# CH1903 (system 6)
|
||||
{"east": 2700000, "north": 1250000, "system": 6, "expected_lat": ..., "expected_lon": ...},
|
||||
|
||||
# UTM (system 7)
|
||||
{"east": 500000, "north": 5200000, "system": 7, "zone": "32N", "expected_lat": ..., "expected_lon": ...},
|
||||
|
||||
# CH1903+ (system 10)
|
||||
{"east": 2700000, "north": 1250000, "system": 10, "expected_lat": ..., "expected_lon": ...},
|
||||
]
|
||||
```
|
||||
|
||||
### Database Schema Understanding
|
||||
Key tables:
|
||||
- `ELABDATAUPGEO`: Survey measurements
|
||||
- `upgeo_mire`: Target points (miras)
|
||||
- `upgeo_lavori`: Projects/jobs
|
||||
- `upgeo_st`: Stations
|
||||
- `sites`: Sites with coordinate system info
|
||||
- `companies`: Company info with mira limits
|
||||
- `alarms`: Alarm records
|
||||
|
||||
---
|
||||
|
||||
## 📊 Complexity Comparison
|
||||
|
||||
| Feature | Legacy | Refactored | Reduction |
|
||||
|---------|--------|-----------|-----------|
|
||||
| **Lines of code** | 2,587 | 508 (+TODO) | 80% |
|
||||
| **Functions** | 5 (1 huge) | 10+ modular | +100% |
|
||||
| **Max nesting** | 8 levels | 3 levels | 63% |
|
||||
| **Type safety** | None | Full hints | ∞ |
|
||||
| **Testability** | Impossible | Easy | ∞ |
|
||||
| **Maintainability** | Very low | High | ∞ |
|
||||
|
||||
---
|
||||
|
||||
## 📚 References
|
||||
|
||||
### Coordinate Systems
|
||||
- **CH1903**: https://www.swisstopo.admin.ch/en/knowledge-facts/surveying-geodesy/reference-systems/local/lv03.html
|
||||
- **CH1903+/LV95**: https://www.swisstopo.admin.ch/en/knowledge-facts/surveying-geodesy/reference-systems/local/lv95.html
|
||||
- **UTM**: https://en.wikipedia.org/wiki/Universal_Transverse_Mercator_coordinate_system
|
||||
|
||||
### Libraries Used
|
||||
- **utm**: UTM <-> lat/lon conversions
|
||||
- **pyproj**: Swiss coordinate system transformations (EPSG:21781 -> EPSG:4326)
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Success Criteria
|
||||
|
||||
Phase 1 complete when:
|
||||
- [ ] All CSV files process without errors
|
||||
- [ ] Coordinate transformations match legacy output
|
||||
- [ ] Miras are created/updated correctly
|
||||
- [ ] Basic alarms are generated for threshold violations
|
||||
- [ ] No regressions in data quality
|
||||
|
||||
Full refactoring complete when:
|
||||
- [ ] All TODO items implemented
|
||||
- [ ] Test coverage > 80%
|
||||
- [ ] Performance >= legacy script
|
||||
- [ ] All additional monitoring types work
|
||||
- [ ] Legacy script can be retired
|
||||
|
||||
---
|
||||
|
||||
**Version**: 1.0 (Essential Refactoring)
|
||||
**Last Updated**: 2024-10-11
|
||||
**Status**: Ready for Phase 1 implementation
|
||||
Reference in New Issue
Block a user