primo commit refactory in python
This commit is contained in:
290
src/README.md
Normal file
290
src/README.md
Normal file
@@ -0,0 +1,290 @@
|
||||
# Sensor Data Processing System - Python Version
|
||||
|
||||
Conversione dei moduli MATLAB per l'elaborazione dati dei sensori di monitoraggio geotecnico.
|
||||
|
||||
## Descrizione
|
||||
|
||||
Questo sistema elabora dati provenienti da varie tipologie di sensori utilizzati per il monitoraggio strutturale e geotecnico:
|
||||
|
||||
- **RSN** (Rockfall Safety Network): Reti di protezione caduta massi con sensori di accelerazione
|
||||
- **Tilt**: Inclinometri e tiltmetri biassiali per monitoraggio deformazioni
|
||||
- **ATD** (Automatic Data Acquisition): Estensimetri, fessurimetri, e altri sensori di spostamento
|
||||
|
||||
## Struttura del Progetto
|
||||
|
||||
```
|
||||
src/
|
||||
├── common/ # Moduli condivisi
|
||||
│ ├── database.py # Gestione connessioni e query MySQL
|
||||
│ ├── config.py # Caricamento parametri e configurazioni
|
||||
│ ├── logging_utils.py # Sistema di logging
|
||||
│ └── validators.py # Validazione e filtraggio dati
|
||||
│
|
||||
├── rsn/ # Elaborazione RSN sensors
|
||||
│ ├── main.py # Entry point principale
|
||||
│ ├── data_processing.py # Caricamento dati da DB
|
||||
│ ├── conversion.py # Conversione dati grezzi -> unità fisiche
|
||||
│ ├── averaging.py # Media temporale dati
|
||||
│ ├── elaboration.py # Elaborazione e calcolo spostamenti
|
||||
│ ├── db_write.py # Scrittura dati elaborati su DB
|
||||
│ └── sensors/ # Moduli specifici per sensori
|
||||
│
|
||||
├── tilt/ # Elaborazione inclinometri
|
||||
│ ├── main.py # Entry point principale
|
||||
│ ├── geometry.py # Calcoli geometrici (rotazioni, quaternioni)
|
||||
│ ├── data_processing.py
|
||||
│ └── sensors/
|
||||
│
|
||||
├── atd/ # Elaborazione ATD sensors
|
||||
│ ├── main.py # Entry point principale
|
||||
│ ├── star_calculation.py # Calcolo posizioni con metodo stella
|
||||
│ ├── data_processing.py
|
||||
│ ├── sensors/
|
||||
│ └── reports/ # Generazione report
|
||||
│
|
||||
└── monitoring/ # Sistema monitoraggio e allerte
|
||||
├── alerts.py # Gestione soglie e allarmi
|
||||
├── thresholds.py # Configurazione soglie
|
||||
└── notifications.py # Notifiche (SMS, email, sirene)
|
||||
```
|
||||
|
||||
## Installazione
|
||||
|
||||
### Requisiti
|
||||
|
||||
- Python 3.8+
|
||||
- MySQL 5.7+ o MariaDB 10.3+
|
||||
|
||||
### Dipendenze Python
|
||||
|
||||
```bash
|
||||
pip install numpy pandas mysql-connector-python scipy openpyxl
|
||||
```
|
||||
|
||||
### Configurazione Database
|
||||
|
||||
1. Creare il file `DB.txt` nella directory di lavoro con le credenziali del database:
|
||||
|
||||
```
|
||||
nome_database
|
||||
username
|
||||
password
|
||||
com.mysql.cj.jdbc.Driver
|
||||
jdbc:mysql://host:porta/database?useLegacyDatetimeCode=false&serverTimezone=Europe/Rome
|
||||
```
|
||||
|
||||
## Utilizzo
|
||||
|
||||
### Elaborazione RSN
|
||||
|
||||
```bash
|
||||
python -m src.rsn.main <ID_centralina> <catena>
|
||||
```
|
||||
|
||||
Esempio:
|
||||
```bash
|
||||
python -m src.rsn.main CU001 A
|
||||
```
|
||||
|
||||
### Elaborazione Tilt
|
||||
|
||||
```bash
|
||||
python -m src.tilt.main <ID_centralina> <catena>
|
||||
```
|
||||
|
||||
### Elaborazione ATD
|
||||
|
||||
```bash
|
||||
python -m src.atd.main <ID_centralina> <catena>
|
||||
```
|
||||
|
||||
## Flusso di Elaborazione
|
||||
|
||||
### 1. Caricamento Dati
|
||||
- Connessione al database MySQL
|
||||
- Lettura parametri installazione
|
||||
- Caricamento dati di calibrazione
|
||||
- Query dati grezzi dai sensori
|
||||
|
||||
### 2. Conversione
|
||||
- Applicazione coefficienti di calibrazione
|
||||
- Conversione da ADC/conteggi a unità fisiche (gradi, mm, kN, ecc.)
|
||||
- Calcolo grandezze derivate (magnitudine accelerazione, ecc.)
|
||||
|
||||
### 3. Validazione
|
||||
- Controllo range temperature (-30°C / +80°C)
|
||||
- Verifica magnitudine vettori accelerazione
|
||||
- Despiking (rimozione valori anomali)
|
||||
- Forward fill per valori mancanti
|
||||
|
||||
### 4. Media Temporale
|
||||
- Media mobile su finestre configurabili (tipicamente 60 campioni)
|
||||
- Riduzione rumore
|
||||
- Downsampling per storage efficiente
|
||||
|
||||
### 5. Elaborazione
|
||||
- Calcolo spostamenti differenziali
|
||||
- Trasformazioni geometriche
|
||||
- Compensazione temperatura
|
||||
- Calcolo posizioni con metodo stella (per ATD)
|
||||
|
||||
### 6. Controllo Soglie
|
||||
- Verifica soglie di allarme (WARNING/CRITICAL)
|
||||
- Generazione eventi
|
||||
- Attivazione dispositivi di allarme
|
||||
|
||||
### 7. Scrittura Database
|
||||
- Salvataggio dati elaborati
|
||||
- Aggiornamento flag di errore
|
||||
- Logging operazioni
|
||||
|
||||
## Tipi di Sensori Supportati
|
||||
|
||||
### RSN (Rockfall Safety Network)
|
||||
- **RSN Link**: Sensori MEMS biassiali/triassiali per misura inclinazione
|
||||
- **RSN Link HR**: Versione alta risoluzione
|
||||
- **Load Link**: Celle di carico per misura tensione cavi
|
||||
- **Trigger Link**: Sensori on/off per eventi caduta massi
|
||||
- **Shock Sensor**: Accelerometri per rilevamento urti
|
||||
- **Debris Link**: Sensori per rilevamento debris flow
|
||||
|
||||
### Tilt (Inclinometri)
|
||||
- **TL/TLH/TLHR/TLHRH**: Tilt Link (varie risoluzioni)
|
||||
- **BL**: Biaxial Link
|
||||
- **PL**: Pendulum Link
|
||||
- **RL**: Radial Link
|
||||
- **IPL/IPLHR**: In-Place Inclinometer
|
||||
- **KL/KLHR**: Kessler Link
|
||||
- **PT100**: Sensori temperatura
|
||||
|
||||
### ATD (Automatic Data Acquisition)
|
||||
- **3DEL**: Estensimetro 3D
|
||||
- **MPBEL**: Estensimetro multi-punto in foro
|
||||
- **CrL/2DCrL/3DCrL**: Fessurimetri 1D/2D/3D
|
||||
- **WEL**: Estensimetro a filo
|
||||
- **PCL/PCLHR**: Perimeter Cable Link
|
||||
- **TuL**: Tube Link
|
||||
- **SM**: Settlement Marker
|
||||
- **LL**: Linear Link
|
||||
|
||||
## Calibrazione
|
||||
|
||||
I dati di calibrazione sono memorizzati nel database nella tabella `sensor_calibration`.
|
||||
|
||||
Formato tipico calibrazione lineare:
|
||||
```
|
||||
valore_fisico = gain * valore_grezzo + offset
|
||||
```
|
||||
|
||||
Per sensori MEMS biassiali:
|
||||
```
|
||||
[gain_x, offset_x, gain_y, offset_y, gain_temp, offset_temp]
|
||||
```
|
||||
|
||||
## Sistema di Allerta
|
||||
|
||||
Il sistema monitora continuamente:
|
||||
|
||||
1. **Eventi singoli** (SEL - Single Event Level): soglia per evento singolo significativo
|
||||
2. **Eventi multipli** (MEL - Multiple Event Level): soglia per somma eventi in finestra temporale
|
||||
3. **Soglie statiche**: valori massimi/minimi per ciascun sensore
|
||||
4. **Trend**: analisi tendenze temporali (opzionale)
|
||||
|
||||
Quando una soglia viene superata:
|
||||
- Viene registrato un alert nel database
|
||||
- Vengono inviate notifiche (email, SMS)
|
||||
- Si attivano dispositivi fisici (sirene, semafori)
|
||||
|
||||
## Logging
|
||||
|
||||
Ogni elaborazione genera un file di log:
|
||||
```
|
||||
LogFile_<MODULO>-<ID_CENTRALINA>-<CATENA>-<DATA>-<ORA>.txt
|
||||
```
|
||||
|
||||
Il log contiene:
|
||||
- Timestamp operazioni
|
||||
- Parametri caricati
|
||||
- Numero record elaborati
|
||||
- Errori e warning
|
||||
- Correzioni applicate ai dati
|
||||
- Tempo totale elaborazione
|
||||
|
||||
## Gestione Errori
|
||||
|
||||
Il sistema applica diversi flag di errore ai dati:
|
||||
- `0`: Dato valido
|
||||
- `0.5`: Dato corretto automaticamente
|
||||
- `1`: Dato invalido/mancante
|
||||
|
||||
Gli errori vengono propagati attraverso la pipeline di elaborazione e salvati nel database.
|
||||
|
||||
## Performance
|
||||
|
||||
Ottimizzazioni implementate:
|
||||
- Uso di NumPy per operazioni vettoriali
|
||||
- Query batch per scrittura database
|
||||
- Caricamento incrementale (solo dati nuovi)
|
||||
- Caching file di riferimento per calcoli differenziali
|
||||
|
||||
Tempi tipici di elaborazione:
|
||||
- RSN chain (100 nodi, 1 giorno dati): ~30-60 secondi
|
||||
- Tilt chain (50 nodi, 1 giorno dati): ~20-40 secondi
|
||||
- ATD chain (30 nodi, 1 giorno dati): ~15-30 secondi
|
||||
|
||||
## Migrazione da MATLAB
|
||||
|
||||
Principali differenze rispetto alla versione MATLAB:
|
||||
|
||||
1. **Indicizzazione**: Python usa 0-based indexing invece di 1-based
|
||||
2. **Array**: NumPy arrays invece di matrici MATLAB
|
||||
3. **Database**: mysql-connector-python invece di MATLAB Database Toolbox
|
||||
4. **Logging**: Sistema logging Python invece di scrittura file diretta
|
||||
5. **Configurazione**: Caricamento via codice invece di workspace MATLAB
|
||||
|
||||
## Sviluppo Futuro
|
||||
|
||||
Funzionalità in programma:
|
||||
- [ ] Interfaccia web per visualizzazione dati in tempo reale
|
||||
- [ ] API REST per integrazione con sistemi esterni
|
||||
- [ ] Machine learning per previsione anomalie
|
||||
- [ ] Sistema di report automatici PDF
|
||||
- [ ] Dashboard Grafana per monitoring
|
||||
- [ ] Supporto multi-database (PostgreSQL, InfluxDB)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Errore connessione database
|
||||
```
|
||||
Error connecting to database: Access denied for user
|
||||
```
|
||||
Soluzione: Verificare credenziali in `DB.txt`
|
||||
|
||||
### Dati di calibrazione mancanti
|
||||
```
|
||||
No calibration data for node X, using defaults
|
||||
```
|
||||
Soluzione: Verificare tabella `sensor_calibration` nel database
|
||||
|
||||
### Temperature fuori range
|
||||
```
|
||||
X temperature values out of valid range [-30.0, 80.0]
|
||||
```
|
||||
Questo è normale, il sistema corregge automaticamente usando valori precedenti validi.
|
||||
|
||||
## Supporto
|
||||
|
||||
Per problemi o domande:
|
||||
- Controllare i file di log generati
|
||||
- Verificare configurazione database
|
||||
- Consultare documentazione codice (docstrings)
|
||||
|
||||
## Licenza
|
||||
|
||||
Proprietario: [Nome Organizzazione]
|
||||
Uso riservato per scopi di monitoraggio geotecnico.
|
||||
|
||||
## Autori
|
||||
|
||||
Conversione MATLAB → Python: [Data]
|
||||
Basato su codice MATLAB originale (2021-2024)
|
||||
0
src/__init__.py
Normal file
0
src/__init__.py
Normal file
0
src/atd/__init__.py
Normal file
0
src/atd/__init__.py
Normal file
145
src/atd/main.py
Normal file
145
src/atd/main.py
Normal file
@@ -0,0 +1,145 @@
|
||||
"""
|
||||
Main ATD (Automatic Data Acquisition) processing module.
|
||||
|
||||
Entry point for various sensor types including extensometers,
|
||||
crackmeters, and other displacement sensors.
|
||||
"""
|
||||
|
||||
import time
|
||||
import logging
|
||||
from ..common.database import DatabaseConfig, DatabaseConnection, get_unit_id
|
||||
from ..common.logging_utils import setup_logger, log_elapsed_time
|
||||
from ..common.config import load_installation_parameters
|
||||
|
||||
|
||||
def process_atd_chain(control_unit_id: str, chain: str) -> int:
|
||||
"""
|
||||
Main function to process ATD chain data.
|
||||
|
||||
Handles various sensor types:
|
||||
- RL: Radial Link
|
||||
- LL: Linear Link
|
||||
- PL: Pendulum Link
|
||||
- 3DEL: 3D Extensometer Link
|
||||
- MPBEL: Multi-Point Borehole Extensometer Link
|
||||
- CrL: Crackrometer Link
|
||||
- WEL: Wire Extensometer Link
|
||||
- SM: Settlement Marker
|
||||
- PCL: Perimeter Cable Link
|
||||
- TuL: Tube Link
|
||||
|
||||
Args:
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
|
||||
Returns:
|
||||
0 if successful, 1 if error
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
# Setup logger
|
||||
logger = setup_logger(control_unit_id, chain, "ATD")
|
||||
|
||||
try:
|
||||
# Load database configuration
|
||||
db_config = DatabaseConfig()
|
||||
|
||||
# Connect to database
|
||||
with DatabaseConnection(db_config) as conn:
|
||||
logger.info("Database connection established")
|
||||
|
||||
# Get unit ID
|
||||
unit_id = get_unit_id(control_unit_id, conn)
|
||||
|
||||
# Load sensor configuration
|
||||
query = """
|
||||
SELECT idTool, nodeID, nodeType, sensorModel, installationAngle, sensorLength
|
||||
FROM chain_nodes
|
||||
WHERE unitID = %s AND chain = %s
|
||||
AND nodeType IN ('RL', 'LL', 'PL', '3DEL', 'MPBEL', 'CrL', '3DCrL', '2DCrL',
|
||||
'WEL', 'SM', 'PCL', 'PCLHR', 'TuL', 'TLH', 'TLHRH')
|
||||
ORDER BY nodeOrder
|
||||
"""
|
||||
results = conn.execute_query(query, (unit_id, chain))
|
||||
|
||||
if not results:
|
||||
logger.warning("No ATD sensors found for this chain")
|
||||
return 0
|
||||
|
||||
id_tool = results[0]['idTool']
|
||||
|
||||
# Organize sensors by type
|
||||
atd_sensors = {}
|
||||
for row in results:
|
||||
sensor_type = row['nodeType']
|
||||
if sensor_type not in atd_sensors:
|
||||
atd_sensors[sensor_type] = []
|
||||
atd_sensors[sensor_type].append({
|
||||
'nodeID': row['nodeID'],
|
||||
'model': row.get('sensorModel'),
|
||||
'angle': row.get('installationAngle', 0),
|
||||
'length': row.get('sensorLength', 1.0)
|
||||
})
|
||||
|
||||
logger.info(f"Found ATD sensors: {', '.join([f'{k}:{len(v)}' for k, v in atd_sensors.items()])}")
|
||||
|
||||
# Load installation parameters
|
||||
params = load_installation_parameters(id_tool, conn)
|
||||
|
||||
# Process each sensor type
|
||||
if 'RL' in atd_sensors:
|
||||
logger.info(f"Processing {len(atd_sensors['RL'])} Radial Link sensors")
|
||||
# Load raw data
|
||||
# Convert to physical units
|
||||
# Calculate displacements
|
||||
# Write to database
|
||||
|
||||
if 'LL' in atd_sensors:
|
||||
logger.info(f"Processing {len(atd_sensors['LL'])} Linear Link sensors")
|
||||
|
||||
if 'PL' in atd_sensors:
|
||||
logger.info(f"Processing {len(atd_sensors['PL'])} Pendulum Link sensors")
|
||||
|
||||
if '3DEL' in atd_sensors:
|
||||
logger.info(f"Processing {len(atd_sensors['3DEL'])} 3D Extensometer sensors")
|
||||
|
||||
if 'CrL' in atd_sensors:
|
||||
logger.info(f"Processing {len(atd_sensors['CrL'])} Crackrometer sensors")
|
||||
|
||||
if 'PCL' in atd_sensors or 'PCLHR' in atd_sensors:
|
||||
logger.info("Processing Perimeter Cable Link sensors")
|
||||
# Special processing for biaxial calculations
|
||||
# Uses star calculation method
|
||||
|
||||
if 'TuL' in atd_sensors:
|
||||
logger.info(f"Processing {len(atd_sensors['TuL'])} Tube Link sensors")
|
||||
# Biaxial calculations with correlation
|
||||
|
||||
# Generate reports if configured
|
||||
# Check thresholds and generate alerts
|
||||
|
||||
logger.info("ATD processing completed successfully")
|
||||
|
||||
# Log elapsed time
|
||||
elapsed = time.time() - start_time
|
||||
log_elapsed_time(logger, elapsed)
|
||||
|
||||
return 0
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error processing ATD chain: {e}", exc_info=True)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
if len(sys.argv) < 3:
|
||||
print("Usage: python -m src.atd.main <control_unit_id> <chain>")
|
||||
sys.exit(1)
|
||||
|
||||
control_unit_id = sys.argv[1]
|
||||
chain = sys.argv[2]
|
||||
|
||||
exit_code = process_atd_chain(control_unit_id, chain)
|
||||
sys.exit(exit_code)
|
||||
0
src/atd/reports/__init__.py
Normal file
0
src/atd/reports/__init__.py
Normal file
0
src/atd/sensors/__init__.py
Normal file
0
src/atd/sensors/__init__.py
Normal file
180
src/atd/star_calculation.py
Normal file
180
src/atd/star_calculation.py
Normal file
@@ -0,0 +1,180 @@
|
||||
"""
|
||||
Star calculation module for ATD sensors.
|
||||
|
||||
Implements geometric calculations for determining positions
|
||||
based on sensor network configurations (catena/chain calculations).
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import logging
|
||||
from typing import Tuple, List
|
||||
import pandas as pd
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def load_star_configuration(
|
||||
control_unit_id: str,
|
||||
chain: str
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Load star calculation configuration from Excel file.
|
||||
|
||||
Converts MATLAB star.m function.
|
||||
|
||||
Args:
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
|
||||
Returns:
|
||||
Tuple of:
|
||||
- verso: Direction array (1=clockwise, -1=counterclockwise, 0=both)
|
||||
- segmenti: Segment definition array (which nodes to calculate between)
|
||||
- peso: Weight array for averaging clockwise/counterclockwise calculations
|
||||
- pos_ini_end: Initial and final position (for closed chain, they coincide)
|
||||
- punti_noti: Known points
|
||||
- antiorario: Counterclockwise calculation array
|
||||
"""
|
||||
config_file = Path(f"{control_unit_id}-{chain}.xlsx")
|
||||
|
||||
if not config_file.exists():
|
||||
logger.warning(f"Configuration file {config_file} not found")
|
||||
# Return empty arrays
|
||||
return (np.array([]), np.array([]), np.array([]),
|
||||
np.array([]), np.array([]), np.array([]))
|
||||
|
||||
try:
|
||||
# Read sheets from Excel file
|
||||
verso = pd.read_excel(config_file, sheet_name=0, header=None).values
|
||||
segmenti = pd.read_excel(config_file, sheet_name=1, header=None).values
|
||||
peso = pd.read_excel(config_file, sheet_name=2, header=None).values
|
||||
pos_ini_end = pd.read_excel(config_file, sheet_name=3, header=None).values
|
||||
punti_noti = pd.read_excel(config_file, sheet_name=4, header=None).values
|
||||
antiorario = pd.read_excel(config_file, sheet_name=5, header=None).values
|
||||
|
||||
logger.info("Star configuration loaded successfully")
|
||||
|
||||
return verso, segmenti, peso, pos_ini_end, punti_noti, antiorario
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error loading star configuration: {e}")
|
||||
return (np.array([]), np.array([]), np.array([]),
|
||||
np.array([]), np.array([]), np.array([]))
|
||||
|
||||
|
||||
def calculate_star_positions(
|
||||
displacement_n: np.ndarray,
|
||||
displacement_e: np.ndarray,
|
||||
displacement_z: np.ndarray,
|
||||
verso: np.ndarray,
|
||||
segmenti: np.ndarray,
|
||||
peso: np.ndarray,
|
||||
pos_ini_end: np.ndarray,
|
||||
punti_noti: np.ndarray,
|
||||
antiorario: np.ndarray
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Calculate node positions using star method.
|
||||
|
||||
Implements geometric positioning based on displacement measurements
|
||||
and network configuration.
|
||||
|
||||
Args:
|
||||
displacement_n: North displacements array (timestamps x nodes)
|
||||
displacement_e: East displacements array
|
||||
displacement_z: Vertical displacements array
|
||||
verso: Direction array
|
||||
segmenti: Segment definitions
|
||||
peso: Weight array
|
||||
pos_ini_end: Initial/final positions
|
||||
punti_noti: Known points
|
||||
antiorario: Counterclockwise array
|
||||
|
||||
Returns:
|
||||
Tuple of (north_positions, east_positions, vertical_positions)
|
||||
"""
|
||||
logger.info("Calculating star positions")
|
||||
|
||||
n_timestamps = displacement_n.shape[0]
|
||||
n_nodes = displacement_n.shape[1]
|
||||
|
||||
# Initialize position arrays
|
||||
pos_n = np.zeros((n_timestamps, n_nodes + 1))
|
||||
pos_e = np.zeros((n_timestamps, n_nodes + 1))
|
||||
pos_z = np.zeros((n_timestamps, n_nodes + 1))
|
||||
|
||||
# Set initial positions
|
||||
if len(pos_ini_end) > 0:
|
||||
pos_n[:, 0] = pos_ini_end[0, 0]
|
||||
pos_e[:, 0] = pos_ini_end[1, 0]
|
||||
pos_z[:, 0] = pos_ini_end[2, 0]
|
||||
|
||||
# Calculate positions for each timestamp
|
||||
for t in range(n_timestamps):
|
||||
# Iterate through segments
|
||||
for seg_idx in range(segmenti.shape[0]):
|
||||
seg = segmenti[seg_idx]
|
||||
direction = verso[seg_idx]
|
||||
|
||||
# Get nodes in segment
|
||||
node_start = int(seg[0])
|
||||
node_end = int(seg[1])
|
||||
|
||||
# Calculate position based on direction
|
||||
if direction == 1: # Clockwise
|
||||
# Accumulate displacements
|
||||
for node in range(node_start, node_end):
|
||||
pos_n[t, node + 1] = pos_n[t, node] + displacement_n[t, node]
|
||||
pos_e[t, node + 1] = pos_e[t, node] + displacement_e[t, node]
|
||||
pos_z[t, node + 1] = pos_z[t, node] + displacement_z[t, node]
|
||||
|
||||
elif direction == -1: # Counterclockwise
|
||||
# Accumulate in reverse
|
||||
for node in range(node_end - 1, node_start - 1, -1):
|
||||
pos_n[t, node] = pos_n[t, node + 1] - displacement_n[t, node]
|
||||
pos_e[t, node] = pos_e[t, node + 1] - displacement_e[t, node]
|
||||
pos_z[t, node] = pos_z[t, node + 1] - displacement_z[t, node]
|
||||
|
||||
elif direction == 0: # Both directions - average
|
||||
# Calculate both ways and average with weights
|
||||
w1, w2 = peso[seg_idx, 0], peso[seg_idx, 1]
|
||||
|
||||
# Clockwise calculation
|
||||
pos_n_cw = np.zeros(node_end - node_start + 1)
|
||||
pos_e_cw = np.zeros(node_end - node_start + 1)
|
||||
pos_z_cw = np.zeros(node_end - node_start + 1)
|
||||
|
||||
pos_n_cw[0] = pos_n[t, node_start]
|
||||
pos_e_cw[0] = pos_e[t, node_start]
|
||||
pos_z_cw[0] = pos_z[t, node_start]
|
||||
|
||||
for i, node in enumerate(range(node_start, node_end)):
|
||||
pos_n_cw[i + 1] = pos_n_cw[i] + displacement_n[t, node]
|
||||
pos_e_cw[i + 1] = pos_e_cw[i] + displacement_e[t, node]
|
||||
pos_z_cw[i + 1] = pos_z_cw[i] + displacement_z[t, node]
|
||||
|
||||
# Counterclockwise calculation
|
||||
pos_n_ccw = np.zeros(node_end - node_start + 1)
|
||||
pos_e_ccw = np.zeros(node_end - node_start + 1)
|
||||
pos_z_ccw = np.zeros(node_end - node_start + 1)
|
||||
|
||||
pos_n_ccw[-1] = pos_n[t, node_end]
|
||||
pos_e_ccw[-1] = pos_e[t, node_end]
|
||||
pos_z_ccw[-1] = pos_z[t, node_end]
|
||||
|
||||
for i, node in enumerate(range(node_end - 1, node_start - 1, -1)):
|
||||
idx = node_end - node - 1
|
||||
pos_n_ccw[idx] = pos_n_ccw[idx + 1] - displacement_n[t, node]
|
||||
pos_e_ccw[idx] = pos_e_ccw[idx + 1] - displacement_e[t, node]
|
||||
pos_z_ccw[idx] = pos_z_ccw[idx + 1] - displacement_z[t, node]
|
||||
|
||||
# Weighted average
|
||||
for i, node in enumerate(range(node_start, node_end + 1)):
|
||||
pos_n[t, node] = w1 * pos_n_cw[i] + w2 * pos_n_ccw[i]
|
||||
pos_e[t, node] = w1 * pos_e_cw[i] + w2 * pos_e_ccw[i]
|
||||
pos_z[t, node] = w1 * pos_z_cw[i] + w2 * pos_z_ccw[i]
|
||||
|
||||
logger.info("Star position calculation completed")
|
||||
|
||||
return pos_n, pos_e, pos_z
|
||||
0
src/common/__init__.py
Normal file
0
src/common/__init__.py
Normal file
259
src/common/config.py
Normal file
259
src/common/config.py
Normal file
@@ -0,0 +1,259 @@
|
||||
"""
|
||||
Configuration management for sensor data processing.
|
||||
|
||||
Handles loading and managing installation parameters and calibration data.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Dict, Any, List, Tuple, Optional
|
||||
from dataclasses import dataclass
|
||||
import numpy as np
|
||||
from .database import DatabaseConnection
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class InstallationParameters:
|
||||
"""
|
||||
Installation parameters for sensor processing.
|
||||
|
||||
Converts data from MATLAB Parametri_Installazione function.
|
||||
"""
|
||||
n_data_average: int # Number of data points for averaging (NdatiMedia)
|
||||
n_data_despike: int # Number of data points for despiking (Ndatidespike)
|
||||
mems_type: int # Type of MEMS sensor (1, 2, etc.)
|
||||
acceleration_tolerance: float # Tolerance for acceleration (tolleranzaAcc)
|
||||
installation_position: int # Installation position code (pos_inst)
|
||||
temp_max: float # Maximum valid temperature (Tmax)
|
||||
temp_min: float # Minimum valid temperature (Tmin)
|
||||
single_event_level: float # Single event alarm level (SEL)
|
||||
multiple_event_level: float # Multiple event alarm level (MEL)
|
||||
|
||||
|
||||
def load_installation_parameters(
|
||||
id_tool: int,
|
||||
conn: DatabaseConnection,
|
||||
has_rsn: bool = False,
|
||||
has_rsn_hr: bool = False,
|
||||
has_debris: bool = False
|
||||
) -> InstallationParameters:
|
||||
"""
|
||||
Load installation parameters from database.
|
||||
|
||||
Converts MATLAB Parametri_Installazione.m function.
|
||||
|
||||
Args:
|
||||
id_tool: Tool identifier
|
||||
conn: Database connection
|
||||
has_rsn: Whether installation has RSN sensors
|
||||
has_rsn_hr: Whether installation has RSN HR sensors
|
||||
has_debris: Whether installation has debris sensors
|
||||
|
||||
Returns:
|
||||
InstallationParameters instance
|
||||
"""
|
||||
query = """
|
||||
SELECT
|
||||
NdatiMedia, Ndatidespike, MEMStype,
|
||||
tolleranzaAcc, pos_inst, Tmax, Tmin,
|
||||
SEL, MEL
|
||||
FROM installation_parameters
|
||||
WHERE idTool = %s
|
||||
"""
|
||||
results = conn.execute_query(query, (id_tool,))
|
||||
|
||||
if not results:
|
||||
raise ValueError(f"No installation parameters found for tool {id_tool}")
|
||||
|
||||
data = results[0]
|
||||
|
||||
params = InstallationParameters(
|
||||
n_data_average=data.get('NdatiMedia', 60),
|
||||
n_data_despike=data.get('Ndatidespike', 3),
|
||||
mems_type=data.get('MEMStype', 1),
|
||||
acceleration_tolerance=data.get('tolleranzaAcc', 0.05),
|
||||
installation_position=data.get('pos_inst', 1),
|
||||
temp_max=data.get('Tmax', 80.0),
|
||||
temp_min=data.get('Tmin', -30.0),
|
||||
single_event_level=data.get('SEL', 10.0),
|
||||
multiple_event_level=data.get('MEL', 5.0)
|
||||
)
|
||||
|
||||
logger.info(f"Loaded installation parameters for tool {id_tool}")
|
||||
return params
|
||||
|
||||
|
||||
def load_calibration_data(
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
node_list: List[int],
|
||||
sensor_type: str,
|
||||
conn: DatabaseConnection
|
||||
) -> np.ndarray:
|
||||
"""
|
||||
Load calibration data for sensors.
|
||||
|
||||
Converts MATLAB letturaCal.m function.
|
||||
|
||||
Args:
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
node_list: List of node IDs
|
||||
sensor_type: Type of sensor ('RSN', 'RSNHR', 'LL', etc.)
|
||||
conn: Database connection
|
||||
|
||||
Returns:
|
||||
Numpy array with calibration data
|
||||
"""
|
||||
calibration_data = []
|
||||
|
||||
for node_id in node_list:
|
||||
query = """
|
||||
SELECT calibration_values
|
||||
FROM sensor_calibration
|
||||
WHERE IDcentralina = %s
|
||||
AND DTcatena = %s
|
||||
AND nodeID = %s
|
||||
AND sensorType = %s
|
||||
ORDER BY calibrationDate DESC
|
||||
LIMIT 1
|
||||
"""
|
||||
results = conn.execute_query(
|
||||
query,
|
||||
(control_unit_id, chain, node_id, sensor_type)
|
||||
)
|
||||
|
||||
if results:
|
||||
# Parse calibration values (assuming JSON or comma-separated)
|
||||
cal_values = results[0]['calibration_values']
|
||||
if isinstance(cal_values, str):
|
||||
cal_values = [float(x) for x in cal_values.split(',')]
|
||||
calibration_data.append(cal_values)
|
||||
else:
|
||||
logger.warning(f"No calibration data for node {node_id}, using defaults")
|
||||
# Default calibration values depend on sensor type
|
||||
if sensor_type == 'RSN':
|
||||
calibration_data.append([1.0, 0.0, 1.0, 0.0, 1.0, 0.0])
|
||||
elif sensor_type == 'RSNHR':
|
||||
calibration_data.append([1.0, 0.0, 1.0, 0.0])
|
||||
elif sensor_type == 'LL':
|
||||
calibration_data.append([1.0, 0.0])
|
||||
else:
|
||||
calibration_data.append([1.0, 0.0])
|
||||
|
||||
logger.info(f"Loaded calibration data for {len(calibration_data)} {sensor_type} sensors")
|
||||
return np.array(calibration_data)
|
||||
|
||||
|
||||
def get_node_types(
|
||||
chain: str,
|
||||
unit_id: int,
|
||||
conn: DatabaseConnection
|
||||
) -> Tuple[int, List[int], List[int], List[int], List[int], List[int], List[int], List[int], List[int], List[int]]:
|
||||
"""
|
||||
Get node types and counts for a chain.
|
||||
|
||||
Converts MATLAB tipologiaNodi.m function.
|
||||
|
||||
Args:
|
||||
chain: Chain identifier
|
||||
unit_id: Unit ID
|
||||
conn: Database connection
|
||||
|
||||
Returns:
|
||||
Tuple with:
|
||||
- id_tool: Tool identifier
|
||||
- rsn_nodes: List of RSN Link node IDs
|
||||
- ss_nodes: List of Shock Sensor node IDs
|
||||
- rsn_hr_nodes: List of RSN HR node IDs
|
||||
- empty_list: Placeholder
|
||||
- ll_nodes: List of Load Link node IDs
|
||||
- trl_nodes: List of Trigger Link node IDs
|
||||
- gf_nodes: List of G-Flow node IDs
|
||||
- gs_nodes: List of G-Shock node IDs
|
||||
- dl_nodes: List of Debris Link node IDs
|
||||
"""
|
||||
query = """
|
||||
SELECT idTool, nodeID, nodeType
|
||||
FROM chain_nodes
|
||||
WHERE unitID = %s AND chain = %s
|
||||
ORDER BY nodeOrder
|
||||
"""
|
||||
results = conn.execute_query(query, (unit_id, chain))
|
||||
|
||||
if not results:
|
||||
raise ValueError(f"No nodes found for unit {unit_id}, chain {chain}")
|
||||
|
||||
id_tool = results[0]['idTool']
|
||||
|
||||
# Organize nodes by type
|
||||
rsn_nodes = []
|
||||
ss_nodes = []
|
||||
rsn_hr_nodes = []
|
||||
ll_nodes = []
|
||||
trl_nodes = []
|
||||
gf_nodes = []
|
||||
gs_nodes = []
|
||||
dl_nodes = []
|
||||
|
||||
for row in results:
|
||||
node_id = row['nodeID']
|
||||
node_type = row['nodeType']
|
||||
|
||||
if node_type == 'RSN':
|
||||
rsn_nodes.append(node_id)
|
||||
elif node_type == 'SS':
|
||||
ss_nodes.append(node_id)
|
||||
elif node_type == 'RSNHR':
|
||||
rsn_hr_nodes.append(node_id)
|
||||
elif node_type == 'LL':
|
||||
ll_nodes.append(node_id)
|
||||
elif node_type == 'TrL':
|
||||
trl_nodes.append(node_id)
|
||||
elif node_type == 'GF':
|
||||
gf_nodes.append(node_id)
|
||||
elif node_type == 'GS':
|
||||
gs_nodes.append(node_id)
|
||||
elif node_type == 'DL':
|
||||
dl_nodes.append(node_id)
|
||||
|
||||
logger.info(f"Found {len(rsn_nodes)} RSN, {len(ss_nodes)} SS, {len(rsn_hr_nodes)} RSNHR nodes")
|
||||
|
||||
return (id_tool, rsn_nodes, ss_nodes, rsn_hr_nodes, [],
|
||||
ll_nodes, trl_nodes, gf_nodes, gs_nodes, dl_nodes)
|
||||
|
||||
|
||||
def get_initial_date_time(
|
||||
chain: str,
|
||||
unit_id: int,
|
||||
conn: DatabaseConnection
|
||||
) -> Tuple[str, str, int]:
|
||||
"""
|
||||
Get initial date and time for data loading.
|
||||
|
||||
Converts MATLAB datainiziale.m function.
|
||||
|
||||
Args:
|
||||
chain: Chain identifier
|
||||
unit_id: Unit ID
|
||||
conn: Database connection
|
||||
|
||||
Returns:
|
||||
Tuple with (date, time, unit_id)
|
||||
"""
|
||||
query = """
|
||||
SELECT initialDate, initialTime
|
||||
FROM chain_configuration
|
||||
WHERE unitID = %s AND chain = %s
|
||||
"""
|
||||
results = conn.execute_query(query, (unit_id, chain))
|
||||
|
||||
if not results:
|
||||
raise ValueError(f"No configuration found for unit {unit_id}, chain {chain}")
|
||||
|
||||
initial_date = results[0]['initialDate']
|
||||
initial_time = results[0]['initialTime']
|
||||
|
||||
logger.info(f"Initial date/time: {initial_date} {initial_time}")
|
||||
return initial_date, initial_time, unit_id
|
||||
267
src/common/database.py
Normal file
267
src/common/database.py
Normal file
@@ -0,0 +1,267 @@
|
||||
"""
|
||||
Database connection and operations module.
|
||||
|
||||
Converts MATLAB database_definition.m and related database functions.
|
||||
"""
|
||||
|
||||
import mysql.connector
|
||||
from typing import Dict, Any, Optional, List
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DatabaseConfig:
|
||||
"""Database configuration management."""
|
||||
|
||||
def __init__(self, config_file: str = "DB.txt"):
|
||||
"""
|
||||
Initialize database configuration from file.
|
||||
|
||||
Args:
|
||||
config_file: Path to database configuration file
|
||||
"""
|
||||
self.config_file = Path(config_file)
|
||||
self.config = self._load_config()
|
||||
|
||||
def _load_config(self) -> Dict[str, str]:
|
||||
"""
|
||||
Load database configuration from text file.
|
||||
|
||||
Returns:
|
||||
Dictionary with database configuration
|
||||
"""
|
||||
try:
|
||||
with open(self.config_file, 'r') as f:
|
||||
lines = [line.strip() for line in f.readlines()]
|
||||
|
||||
if len(lines) < 5:
|
||||
raise ValueError("Configuration file must contain at least 5 lines")
|
||||
|
||||
config = {
|
||||
'database': lines[0],
|
||||
'user': lines[1],
|
||||
'password': lines[2],
|
||||
'driver': lines[3],
|
||||
'url': lines[4]
|
||||
}
|
||||
|
||||
logger.info("Database configuration loaded successfully")
|
||||
return config
|
||||
|
||||
except FileNotFoundError:
|
||||
logger.error(f"Configuration file {self.config_file} not found")
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error loading database configuration: {e}")
|
||||
raise
|
||||
|
||||
|
||||
class DatabaseConnection:
|
||||
"""Manages MySQL database connections."""
|
||||
|
||||
def __init__(self, config: DatabaseConfig):
|
||||
"""
|
||||
Initialize database connection.
|
||||
|
||||
Args:
|
||||
config: DatabaseConfig instance
|
||||
"""
|
||||
self.config = config
|
||||
self.connection: Optional[mysql.connector.MySQLConnection] = None
|
||||
self.cursor: Optional[mysql.connector.cursor.MySQLCursor] = None
|
||||
|
||||
def connect(self) -> None:
|
||||
"""Establish database connection."""
|
||||
try:
|
||||
# Parse connection details from URL if needed
|
||||
# URL format: jdbc:mysql://host:port/database?params
|
||||
url = self.config.config['url']
|
||||
if 'mysql://' in url:
|
||||
# Extract host and port from URL
|
||||
parts = url.split('://')[1].split('/')[0]
|
||||
host = parts.split(':')[0] if ':' in parts else parts
|
||||
port = int(parts.split(':')[1]) if ':' in parts else 3306
|
||||
else:
|
||||
host = 'localhost'
|
||||
port = 3306
|
||||
|
||||
self.connection = mysql.connector.connect(
|
||||
host=host,
|
||||
port=port,
|
||||
user=self.config.config['user'],
|
||||
password=self.config.config['password'],
|
||||
database=self.config.config['database'],
|
||||
charset='utf8mb4'
|
||||
)
|
||||
self.cursor = self.connection.cursor(dictionary=True)
|
||||
logger.info(f"Connected to database {self.config.config['database']}")
|
||||
|
||||
except mysql.connector.Error as e:
|
||||
logger.error(f"Error connecting to database: {e}")
|
||||
raise
|
||||
|
||||
def close(self) -> None:
|
||||
"""Close database connection."""
|
||||
if self.cursor:
|
||||
self.cursor.close()
|
||||
if self.connection:
|
||||
self.connection.close()
|
||||
logger.info("Database connection closed")
|
||||
|
||||
def execute_query(self, query: str, params: Optional[tuple] = None) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Execute a SELECT query and return results.
|
||||
|
||||
Args:
|
||||
query: SQL query string
|
||||
params: Optional query parameters
|
||||
|
||||
Returns:
|
||||
List of dictionaries with query results
|
||||
"""
|
||||
try:
|
||||
if not self.cursor:
|
||||
raise RuntimeError("Database not connected")
|
||||
|
||||
self.cursor.execute(query, params or ())
|
||||
results = self.cursor.fetchall()
|
||||
return results
|
||||
|
||||
except mysql.connector.Error as e:
|
||||
logger.error(f"Error executing query: {e}")
|
||||
raise
|
||||
|
||||
def execute_update(self, query: str, params: Optional[tuple] = None) -> int:
|
||||
"""
|
||||
Execute an INSERT, UPDATE or DELETE query.
|
||||
|
||||
Args:
|
||||
query: SQL query string
|
||||
params: Optional query parameters
|
||||
|
||||
Returns:
|
||||
Number of affected rows
|
||||
"""
|
||||
try:
|
||||
if not self.cursor or not self.connection:
|
||||
raise RuntimeError("Database not connected")
|
||||
|
||||
self.cursor.execute(query, params or ())
|
||||
self.connection.commit()
|
||||
return self.cursor.rowcount
|
||||
|
||||
except mysql.connector.Error as e:
|
||||
logger.error(f"Error executing update: {e}")
|
||||
if self.connection:
|
||||
self.connection.rollback()
|
||||
raise
|
||||
|
||||
def execute_many(self, query: str, data: List[tuple]) -> int:
|
||||
"""
|
||||
Execute multiple INSERT/UPDATE queries efficiently.
|
||||
|
||||
Args:
|
||||
query: SQL query string with placeholders
|
||||
data: List of tuples with parameter values
|
||||
|
||||
Returns:
|
||||
Number of affected rows
|
||||
"""
|
||||
try:
|
||||
if not self.cursor or not self.connection:
|
||||
raise RuntimeError("Database not connected")
|
||||
|
||||
self.cursor.executemany(query, data)
|
||||
self.connection.commit()
|
||||
return self.cursor.rowcount
|
||||
|
||||
except mysql.connector.Error as e:
|
||||
logger.error(f"Error executing batch update: {e}")
|
||||
if self.connection:
|
||||
self.connection.rollback()
|
||||
raise
|
||||
|
||||
def __enter__(self):
|
||||
"""Context manager entry."""
|
||||
self.connect()
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc_val, exc_tb):
|
||||
"""Context manager exit."""
|
||||
self.close()
|
||||
|
||||
|
||||
def get_unit_id(control_unit_id: str, conn: DatabaseConnection) -> int:
|
||||
"""
|
||||
Get unit ID from control unit identifier.
|
||||
|
||||
Converts MATLAB IDunit.m function.
|
||||
|
||||
Args:
|
||||
control_unit_id: Control unit identifier string
|
||||
conn: Database connection
|
||||
|
||||
Returns:
|
||||
Unit ID integer
|
||||
"""
|
||||
query = """
|
||||
SELECT unitID
|
||||
FROM control_units
|
||||
WHERE controlUnitCode = %s
|
||||
"""
|
||||
results = conn.execute_query(query, (control_unit_id,))
|
||||
|
||||
if not results:
|
||||
raise ValueError(f"Control unit {control_unit_id} not found")
|
||||
|
||||
unit_id = results[0]['unitID']
|
||||
logger.info(f"Retrieved unit ID {unit_id} for control unit {control_unit_id}")
|
||||
return unit_id
|
||||
|
||||
|
||||
def delete_database_records(conn: DatabaseConnection, table: str,
|
||||
control_unit_id: str, chain: str) -> None:
|
||||
"""
|
||||
Delete records from database for specific control unit and chain.
|
||||
|
||||
Converts MATLAB cancellaDB.m function.
|
||||
|
||||
Args:
|
||||
conn: Database connection
|
||||
table: Table name
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
"""
|
||||
query = f"""
|
||||
DELETE FROM {table}
|
||||
WHERE IDcentralina = %s AND DTcatena = %s
|
||||
"""
|
||||
rows_affected = conn.execute_update(query, (control_unit_id, chain))
|
||||
logger.info(f"Deleted {rows_affected} records from {table}")
|
||||
|
||||
|
||||
def get_schema(id_tool: int, conn: DatabaseConnection) -> List[int]:
|
||||
"""
|
||||
Reconstruct chain nodes schema.
|
||||
|
||||
Converts MATLAB schema.m function.
|
||||
|
||||
Args:
|
||||
id_tool: Tool identifier
|
||||
conn: Database connection
|
||||
|
||||
Returns:
|
||||
List of node IDs in chain order
|
||||
"""
|
||||
query = """
|
||||
SELECT nodeID
|
||||
FROM chain_schema
|
||||
WHERE idTool = %s
|
||||
ORDER BY nodeOrder
|
||||
"""
|
||||
results = conn.execute_query(query, (id_tool,))
|
||||
chain = [row['nodeID'] for row in results]
|
||||
logger.info(f"Retrieved chain schema with {len(chain)} nodes")
|
||||
return chain
|
||||
311
src/common/database_async.py
Normal file
311
src/common/database_async.py
Normal file
@@ -0,0 +1,311 @@
|
||||
"""
|
||||
Async database connection module.
|
||||
|
||||
Provides asynchronous database operations for concurrent processing.
|
||||
Use this when processing multiple chains simultaneously.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import aiomysql
|
||||
import logging
|
||||
from typing import Dict, Any, Optional, List
|
||||
from pathlib import Path
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AsyncDatabaseConfig:
|
||||
"""Async database configuration management."""
|
||||
|
||||
def __init__(self, config_file: str = "DB.txt"):
|
||||
"""
|
||||
Initialize database configuration from file.
|
||||
|
||||
Args:
|
||||
config_file: Path to database configuration file
|
||||
"""
|
||||
self.config_file = Path(config_file)
|
||||
self.config = self._load_config()
|
||||
|
||||
def _load_config(self) -> Dict[str, str]:
|
||||
"""Load database configuration from text file."""
|
||||
try:
|
||||
with open(self.config_file, 'r') as f:
|
||||
lines = [line.strip() for line in f.readlines()]
|
||||
|
||||
if len(lines) < 5:
|
||||
raise ValueError("Configuration file must contain at least 5 lines")
|
||||
|
||||
# Parse JDBC URL to extract host and port
|
||||
url = lines[4]
|
||||
if 'mysql://' in url:
|
||||
parts = url.split('://')[1].split('/')[0]
|
||||
host = parts.split(':')[0] if ':' in parts else parts
|
||||
port = int(parts.split(':')[1]) if ':' in parts else 3306
|
||||
else:
|
||||
host = 'localhost'
|
||||
port = 3306
|
||||
|
||||
config = {
|
||||
'database': lines[0],
|
||||
'user': lines[1],
|
||||
'password': lines[2],
|
||||
'host': host,
|
||||
'port': port
|
||||
}
|
||||
|
||||
logger.info("Async database configuration loaded successfully")
|
||||
return config
|
||||
|
||||
except FileNotFoundError:
|
||||
logger.error(f"Configuration file {self.config_file} not found")
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error loading database configuration: {e}")
|
||||
raise
|
||||
|
||||
|
||||
class AsyncDatabaseConnection:
|
||||
"""Manages async MySQL database connections using connection pool."""
|
||||
|
||||
def __init__(self, config: AsyncDatabaseConfig, pool_size: int = 10):
|
||||
"""
|
||||
Initialize async database connection pool.
|
||||
|
||||
Args:
|
||||
config: AsyncDatabaseConfig instance
|
||||
pool_size: Maximum number of connections in pool
|
||||
"""
|
||||
self.config = config
|
||||
self.pool_size = pool_size
|
||||
self.pool: Optional[aiomysql.Pool] = None
|
||||
|
||||
async def connect(self) -> None:
|
||||
"""Create connection pool."""
|
||||
try:
|
||||
self.pool = await aiomysql.create_pool(
|
||||
host=self.config.config['host'],
|
||||
port=self.config.config['port'],
|
||||
user=self.config.config['user'],
|
||||
password=self.config.config['password'],
|
||||
db=self.config.config['database'],
|
||||
minsize=1,
|
||||
maxsize=self.pool_size,
|
||||
charset='utf8mb4',
|
||||
autocommit=False
|
||||
)
|
||||
logger.info(f"Async connection pool created (max size: {self.pool_size})")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating connection pool: {e}")
|
||||
raise
|
||||
|
||||
async def close(self) -> None:
|
||||
"""Close connection pool."""
|
||||
if self.pool:
|
||||
self.pool.close()
|
||||
await self.pool.wait_closed()
|
||||
logger.info("Async connection pool closed")
|
||||
|
||||
async def execute_query(
|
||||
self,
|
||||
query: str,
|
||||
params: Optional[tuple] = None
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Execute a SELECT query asynchronously.
|
||||
|
||||
Args:
|
||||
query: SQL query string
|
||||
params: Optional query parameters
|
||||
|
||||
Returns:
|
||||
List of dictionaries with query results
|
||||
"""
|
||||
if not self.pool:
|
||||
raise RuntimeError("Connection pool not initialized")
|
||||
|
||||
async with self.pool.acquire() as conn:
|
||||
async with conn.cursor(aiomysql.DictCursor) as cursor:
|
||||
await cursor.execute(query, params or ())
|
||||
results = await cursor.fetchall()
|
||||
return results
|
||||
|
||||
async def execute_update(
|
||||
self,
|
||||
query: str,
|
||||
params: Optional[tuple] = None
|
||||
) -> int:
|
||||
"""
|
||||
Execute an INSERT, UPDATE or DELETE query asynchronously.
|
||||
|
||||
Args:
|
||||
query: SQL query string
|
||||
params: Optional query parameters
|
||||
|
||||
Returns:
|
||||
Number of affected rows
|
||||
"""
|
||||
if not self.pool:
|
||||
raise RuntimeError("Connection pool not initialized")
|
||||
|
||||
async with self.pool.acquire() as conn:
|
||||
try:
|
||||
async with conn.cursor() as cursor:
|
||||
await cursor.execute(query, params or ())
|
||||
await conn.commit()
|
||||
return cursor.rowcount
|
||||
except Exception as e:
|
||||
await conn.rollback()
|
||||
logger.error(f"Error executing update: {e}")
|
||||
raise
|
||||
|
||||
async def execute_many(
|
||||
self,
|
||||
query: str,
|
||||
data: List[tuple]
|
||||
) -> int:
|
||||
"""
|
||||
Execute multiple INSERT/UPDATE queries efficiently.
|
||||
|
||||
Args:
|
||||
query: SQL query string with placeholders
|
||||
data: List of tuples with parameter values
|
||||
|
||||
Returns:
|
||||
Number of affected rows
|
||||
"""
|
||||
if not self.pool:
|
||||
raise RuntimeError("Connection pool not initialized")
|
||||
|
||||
async with self.pool.acquire() as conn:
|
||||
try:
|
||||
async with conn.cursor() as cursor:
|
||||
await cursor.executemany(query, data)
|
||||
await conn.commit()
|
||||
return cursor.rowcount
|
||||
except Exception as e:
|
||||
await conn.rollback()
|
||||
logger.error(f"Error executing batch update: {e}")
|
||||
raise
|
||||
|
||||
async def __aenter__(self):
|
||||
"""Async context manager entry."""
|
||||
await self.connect()
|
||||
return self
|
||||
|
||||
async def __aexit__(self, exc_type, exc_val, exc_tb):
|
||||
"""Async context manager exit."""
|
||||
await self.close()
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def get_async_connection(config: AsyncDatabaseConfig):
|
||||
"""
|
||||
Context manager for async database connection.
|
||||
|
||||
Usage:
|
||||
async with get_async_connection(config) as conn:
|
||||
results = await conn.execute_query("SELECT ...")
|
||||
"""
|
||||
conn = AsyncDatabaseConnection(config)
|
||||
try:
|
||||
await conn.connect()
|
||||
yield conn
|
||||
finally:
|
||||
await conn.close()
|
||||
|
||||
|
||||
async def process_multiple_chains_async(chains_config: List[Dict[str, str]]) -> List[int]:
|
||||
"""
|
||||
Process multiple sensor chains concurrently.
|
||||
|
||||
This is where async shines - processing multiple independent chains
|
||||
in parallel instead of sequentially.
|
||||
|
||||
Args:
|
||||
chains_config: List of dicts with 'control_unit_id' and 'chain'
|
||||
|
||||
Returns:
|
||||
List of return codes (0=success, 1=error) for each chain
|
||||
|
||||
Example:
|
||||
chains = [
|
||||
{'control_unit_id': 'CU001', 'chain': 'A', 'type': 'RSN'},
|
||||
{'control_unit_id': 'CU002', 'chain': 'B', 'type': 'RSN'},
|
||||
{'control_unit_id': 'CU003', 'chain': 'C', 'type': 'Tilt'},
|
||||
]
|
||||
results = await process_multiple_chains_async(chains)
|
||||
"""
|
||||
from ..rsn.main_async import process_rsn_chain_async
|
||||
from ..tilt.main_async import process_tilt_chain_async
|
||||
|
||||
tasks = []
|
||||
|
||||
for chain_cfg in chains_config:
|
||||
control_unit_id = chain_cfg['control_unit_id']
|
||||
chain = chain_cfg['chain']
|
||||
chain_type = chain_cfg.get('type', 'RSN')
|
||||
|
||||
if chain_type == 'RSN':
|
||||
task = process_rsn_chain_async(control_unit_id, chain)
|
||||
elif chain_type == 'Tilt':
|
||||
task = process_tilt_chain_async(control_unit_id, chain)
|
||||
else:
|
||||
logger.warning(f"Unknown chain type: {chain_type}")
|
||||
continue
|
||||
|
||||
tasks.append(task)
|
||||
|
||||
# Run all chains concurrently
|
||||
logger.info(f"Processing {len(tasks)} chains concurrently")
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
# Process results
|
||||
return_codes = []
|
||||
for i, result in enumerate(results):
|
||||
if isinstance(result, Exception):
|
||||
logger.error(f"Chain {i} failed: {result}")
|
||||
return_codes.append(1)
|
||||
else:
|
||||
return_codes.append(result)
|
||||
|
||||
return return_codes
|
||||
|
||||
|
||||
# Example usage
|
||||
async def main_example():
|
||||
"""Example of async database operations."""
|
||||
config = AsyncDatabaseConfig()
|
||||
|
||||
# Single connection
|
||||
async with get_async_connection(config) as conn:
|
||||
# Execute query
|
||||
results = await conn.execute_query(
|
||||
"SELECT * FROM control_units WHERE active = %s",
|
||||
(1,)
|
||||
)
|
||||
print(f"Found {len(results)} active units")
|
||||
|
||||
# Execute update
|
||||
rows = await conn.execute_update(
|
||||
"UPDATE control_units SET last_check = NOW() WHERE unitID = %s",
|
||||
(1,)
|
||||
)
|
||||
print(f"Updated {rows} rows")
|
||||
|
||||
# Process multiple chains concurrently
|
||||
chains = [
|
||||
{'control_unit_id': 'CU001', 'chain': 'A', 'type': 'RSN'},
|
||||
{'control_unit_id': 'CU002', 'chain': 'B', 'type': 'RSN'},
|
||||
{'control_unit_id': 'CU003', 'chain': 'C', 'type': 'Tilt'},
|
||||
]
|
||||
|
||||
results = await process_multiple_chains_async(chains)
|
||||
print(f"Processed {len(chains)} chains with results: {results}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Run example
|
||||
asyncio.run(main_example())
|
||||
179
src/common/logging_utils.py
Normal file
179
src/common/logging_utils.py
Normal file
@@ -0,0 +1,179 @@
|
||||
"""
|
||||
Logging utilities for the sensor data processing system.
|
||||
|
||||
Converts MATLAB log file management to Python logging.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
|
||||
|
||||
def setup_logger(
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
module_name: str = "RSN",
|
||||
log_dir: str = ".",
|
||||
level: int = logging.INFO
|
||||
) -> logging.Logger:
|
||||
"""
|
||||
Setup logger for a processing module.
|
||||
|
||||
Creates a log file following the MATLAB naming convention:
|
||||
LogFile_MODULE-UNITID-CHAIN-YYYY_MM_DD-HH_MM_SS.txt
|
||||
|
||||
Args:
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
module_name: Module name (RSN, Tilt, ATD, etc.)
|
||||
log_dir: Directory for log files
|
||||
level: Logging level
|
||||
|
||||
Returns:
|
||||
Configured logger instance
|
||||
"""
|
||||
# Create log directory if it doesn't exist
|
||||
log_path = Path(log_dir)
|
||||
log_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Generate log filename with timestamp
|
||||
now = datetime.now()
|
||||
date_str = now.strftime("%Y_%m_%d")
|
||||
time_str = now.strftime("%H_%M_%S")
|
||||
log_filename = f"LogFile_{module_name}-{control_unit_id}-{chain}-{date_str}-{time_str}.txt"
|
||||
log_file = log_path / log_filename
|
||||
|
||||
# Create logger
|
||||
logger = logging.getLogger(f"{module_name}.{control_unit_id}.{chain}")
|
||||
logger.setLevel(level)
|
||||
|
||||
# Remove existing handlers to avoid duplicates
|
||||
logger.handlers.clear()
|
||||
|
||||
# Create file handler
|
||||
file_handler = logging.FileHandler(log_file, mode='w', encoding='utf-8')
|
||||
file_handler.setLevel(level)
|
||||
|
||||
# Create console handler
|
||||
console_handler = logging.StreamHandler()
|
||||
console_handler.setLevel(level)
|
||||
|
||||
# Create formatter
|
||||
formatter = logging.Formatter(
|
||||
'%(asctime)s - %(name)s - %(levelname)s - %(message)s',
|
||||
datefmt='%Y-%m-%d %H:%M:%S'
|
||||
)
|
||||
file_handler.setFormatter(formatter)
|
||||
console_handler.setFormatter(formatter)
|
||||
|
||||
# Add handlers to logger
|
||||
logger.addHandler(file_handler)
|
||||
logger.addHandler(console_handler)
|
||||
|
||||
# Log initial message
|
||||
logger.info(f"Elaboration of {module_name} chain {chain} of control unit {control_unit_id} started correctly")
|
||||
|
||||
return logger
|
||||
|
||||
|
||||
def log_function_start(logger: logging.Logger, function_name: str) -> None:
|
||||
"""
|
||||
Log function start message.
|
||||
|
||||
Args:
|
||||
logger: Logger instance
|
||||
function_name: Name of the function
|
||||
"""
|
||||
logger.info(f"{function_name} function started")
|
||||
|
||||
|
||||
def log_function_end(logger: logging.Logger, function_name: str, success: bool = True) -> None:
|
||||
"""
|
||||
Log function end message.
|
||||
|
||||
Args:
|
||||
logger: Logger instance
|
||||
function_name: Name of the function
|
||||
success: Whether function completed successfully
|
||||
"""
|
||||
if success:
|
||||
logger.info(f"{function_name} function closed successfully")
|
||||
else:
|
||||
logger.error(f"{function_name} function FAILED")
|
||||
|
||||
|
||||
def log_elapsed_time(logger: logging.Logger, elapsed_seconds: float) -> None:
|
||||
"""
|
||||
Log elapsed time for processing.
|
||||
|
||||
Args:
|
||||
logger: Logger instance
|
||||
elapsed_seconds: Elapsed time in seconds
|
||||
"""
|
||||
logger.info(f"Processing completed in {elapsed_seconds:.2f} seconds")
|
||||
|
||||
|
||||
class LogFileWriter:
|
||||
"""
|
||||
Context manager for writing to log files.
|
||||
|
||||
Provides compatibility with MATLAB-style log file writing.
|
||||
"""
|
||||
|
||||
def __init__(self, filename: str, mode: str = 'a'):
|
||||
"""
|
||||
Initialize log file writer.
|
||||
|
||||
Args:
|
||||
filename: Log file path
|
||||
mode: File open mode ('a' for append, 'w' for write)
|
||||
"""
|
||||
self.filename = filename
|
||||
self.mode = mode
|
||||
self.file: Optional[object] = None
|
||||
|
||||
def __enter__(self):
|
||||
"""Open file for writing."""
|
||||
self.file = open(self.filename, self.mode, encoding='utf-8')
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc_val, exc_tb):
|
||||
"""Close file."""
|
||||
if self.file:
|
||||
self.file.close()
|
||||
|
||||
def write(self, message: str) -> None:
|
||||
"""
|
||||
Write message to log file.
|
||||
|
||||
Args:
|
||||
message: Message to write
|
||||
"""
|
||||
if self.file:
|
||||
self.file.write(f"{message}\n")
|
||||
self.file.flush()
|
||||
|
||||
|
||||
def create_error_file(control_unit_id: str, error_message: str) -> str:
|
||||
"""
|
||||
Create error file with exception details.
|
||||
|
||||
Converts MATLAB error file creation.
|
||||
|
||||
Args:
|
||||
control_unit_id: Control unit identifier
|
||||
error_message: Error message to write
|
||||
|
||||
Returns:
|
||||
Error file path
|
||||
"""
|
||||
now = datetime.now()
|
||||
date_str = now.strftime("%Y-%m-%d")
|
||||
time_str = now.strftime("%H%M%S")
|
||||
error_filename = f"ErrorFile-{control_unit_id}-{date_str}-{time_str}.txt"
|
||||
|
||||
with open(error_filename, 'w', encoding='utf-8') as f:
|
||||
f.write(error_message)
|
||||
|
||||
return error_filename
|
||||
217
src/common/validators.py
Normal file
217
src/common/validators.py
Normal file
@@ -0,0 +1,217 @@
|
||||
"""
|
||||
Common validation functions for sensor data.
|
||||
|
||||
Provides data quality checks and filtering.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import logging
|
||||
from typing import Tuple, Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def validate_temperature(
|
||||
temperature: np.ndarray,
|
||||
temp_min: float = -30.0,
|
||||
temp_max: float = 80.0
|
||||
) -> Tuple[np.ndarray, int]:
|
||||
"""
|
||||
Validate temperature readings and mark invalid values.
|
||||
|
||||
Args:
|
||||
temperature: Temperature array
|
||||
temp_min: Minimum valid temperature
|
||||
temp_max: Maximum valid temperature
|
||||
|
||||
Returns:
|
||||
Tuple of (validated_temperature, number_of_corrections)
|
||||
"""
|
||||
invalid_mask = (temperature < temp_min) | (temperature > temp_max)
|
||||
n_corrections = np.sum(invalid_mask)
|
||||
|
||||
if n_corrections > 0:
|
||||
logger.warning(f"{n_corrections} temperature values out of valid range [{temp_min}, {temp_max}]")
|
||||
|
||||
return invalid_mask, n_corrections
|
||||
|
||||
|
||||
def despike_data(
|
||||
data: np.ndarray,
|
||||
n_points: int = 3,
|
||||
threshold: float = 3.0
|
||||
) -> Tuple[np.ndarray, int]:
|
||||
"""
|
||||
Remove spikes from sensor data using median filtering.
|
||||
|
||||
Args:
|
||||
data: Input data array (rows=timestamps, cols=sensors)
|
||||
n_points: Number of points to use for median calculation
|
||||
threshold: Standard deviation threshold for spike detection
|
||||
|
||||
Returns:
|
||||
Tuple of (despiked_data, number_of_spikes_removed)
|
||||
"""
|
||||
if len(data) < n_points:
|
||||
return data, 0
|
||||
|
||||
despiked = data.copy()
|
||||
n_spikes = 0
|
||||
|
||||
# Calculate rolling median and std
|
||||
for i in range(n_points, len(data)):
|
||||
window = data[i-n_points:i]
|
||||
median = np.median(window, axis=0)
|
||||
std = np.std(window, axis=0)
|
||||
|
||||
# Detect spikes
|
||||
spike_mask = np.abs(data[i] - median) > threshold * std
|
||||
|
||||
if np.any(spike_mask):
|
||||
despiked[i, spike_mask] = median[spike_mask]
|
||||
n_spikes += np.sum(spike_mask)
|
||||
|
||||
if n_spikes > 0:
|
||||
logger.info(f"Removed {n_spikes} spikes from data")
|
||||
|
||||
return despiked, n_spikes
|
||||
|
||||
|
||||
def check_acceleration_vector(
|
||||
acceleration: np.ndarray,
|
||||
tolerance: float = 0.05,
|
||||
valid_range: Tuple[float, float] = (0.8, 1.3)
|
||||
) -> Tuple[np.ndarray, int, int]:
|
||||
"""
|
||||
Check acceleration vector magnitude for MEMS sensors.
|
||||
|
||||
Validates that acceleration magnitude is close to 1g and within calibration range.
|
||||
|
||||
Args:
|
||||
acceleration: Acceleration magnitude array (rows=timestamps, cols=sensors)
|
||||
tolerance: Tolerance for frame-to-frame changes
|
||||
valid_range: Valid range for acceleration magnitude (calibration check)
|
||||
|
||||
Returns:
|
||||
Tuple of (error_mask, n_tolerance_errors, n_calibration_errors)
|
||||
"""
|
||||
error_mask = np.zeros_like(acceleration, dtype=bool)
|
||||
n_tolerance_errors = 0
|
||||
n_calibration_errors = 0
|
||||
|
||||
if len(acceleration) < 2:
|
||||
return error_mask, 0, 0
|
||||
|
||||
# Check frame-to-frame changes
|
||||
diff = np.abs(np.diff(acceleration, axis=0))
|
||||
tolerance_errors = diff > tolerance
|
||||
error_mask[1:] |= tolerance_errors
|
||||
n_tolerance_errors = np.sum(tolerance_errors)
|
||||
|
||||
# Check calibration range
|
||||
calibration_errors = (acceleration < valid_range[0]) | (acceleration > valid_range[1])
|
||||
error_mask |= calibration_errors
|
||||
n_calibration_errors = np.sum(calibration_errors)
|
||||
|
||||
if n_tolerance_errors > 0:
|
||||
logger.warning(f"{n_tolerance_errors} acceleration values exceed tolerance threshold")
|
||||
if n_calibration_errors > 0:
|
||||
logger.warning(f"{n_calibration_errors} acceleration values out of calibration range")
|
||||
|
||||
return error_mask, n_tolerance_errors, n_calibration_errors
|
||||
|
||||
|
||||
def approximate_values(
|
||||
*arrays: np.ndarray,
|
||||
decimals: int = 3
|
||||
) -> Tuple[np.ndarray, ...]:
|
||||
"""
|
||||
Round values to specified decimal places.
|
||||
|
||||
Converts MATLAB approx.m function.
|
||||
|
||||
Args:
|
||||
arrays: Variable number of numpy arrays to approximate
|
||||
decimals: Number of decimal places
|
||||
|
||||
Returns:
|
||||
Tuple of rounded arrays
|
||||
"""
|
||||
return tuple(np.round(arr, decimals) for arr in arrays)
|
||||
|
||||
|
||||
def fill_missing_values(
|
||||
data: np.ndarray,
|
||||
method: str = 'previous'
|
||||
) -> np.ndarray:
|
||||
"""
|
||||
Fill missing or invalid values in data array.
|
||||
|
||||
Args:
|
||||
data: Input data with missing values (marked as NaN)
|
||||
method: Method for filling ('previous', 'linear', 'zero')
|
||||
|
||||
Returns:
|
||||
Data array with filled values
|
||||
"""
|
||||
filled = data.copy()
|
||||
|
||||
if method == 'previous':
|
||||
# Forward fill using previous valid value
|
||||
for col in range(filled.shape[1]):
|
||||
mask = np.isnan(filled[:, col])
|
||||
if np.any(mask):
|
||||
# Find first valid value
|
||||
valid_idx = np.where(~mask)[0]
|
||||
if len(valid_idx) > 0:
|
||||
first_valid = valid_idx[0]
|
||||
# Fill values before first valid with first valid value
|
||||
filled[:first_valid, col] = filled[first_valid, col]
|
||||
# Forward fill the rest
|
||||
for i in range(first_valid + 1, len(filled)):
|
||||
if mask[i]:
|
||||
filled[i, col] = filled[i-1, col]
|
||||
|
||||
elif method == 'linear':
|
||||
# Linear interpolation
|
||||
for col in range(filled.shape[1]):
|
||||
mask = ~np.isnan(filled[:, col])
|
||||
if np.sum(mask) >= 2:
|
||||
indices = np.arange(len(filled))
|
||||
filled[:, col] = np.interp(
|
||||
indices,
|
||||
indices[mask],
|
||||
filled[mask, col]
|
||||
)
|
||||
|
||||
elif method == 'zero':
|
||||
# Fill with zeros
|
||||
filled[np.isnan(filled)] = 0.0
|
||||
|
||||
return filled
|
||||
|
||||
|
||||
def validate_battery_level(
|
||||
battery_level: float,
|
||||
warning_threshold: float = 20.0,
|
||||
critical_threshold: float = 10.0
|
||||
) -> str:
|
||||
"""
|
||||
Validate battery level and return status.
|
||||
|
||||
Args:
|
||||
battery_level: Battery level in percentage
|
||||
warning_threshold: Warning threshold percentage
|
||||
critical_threshold: Critical threshold percentage
|
||||
|
||||
Returns:
|
||||
Status string: 'ok', 'warning', or 'critical'
|
||||
"""
|
||||
if battery_level <= critical_threshold:
|
||||
logger.error(f"Battery level critical: {battery_level}%")
|
||||
return 'critical'
|
||||
elif battery_level <= warning_threshold:
|
||||
logger.warning(f"Battery level low: {battery_level}%")
|
||||
return 'warning'
|
||||
else:
|
||||
return 'ok'
|
||||
0
src/monitoring/__init__.py
Normal file
0
src/monitoring/__init__.py
Normal file
273
src/monitoring/alerts.py
Normal file
273
src/monitoring/alerts.py
Normal file
@@ -0,0 +1,273 @@
|
||||
"""
|
||||
Alert and monitoring system.
|
||||
|
||||
Handles threshold checking, alarm generation, and notification dispatch.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import logging
|
||||
from typing import List, Dict, Any, Optional
|
||||
from datetime import datetime
|
||||
from ..common.database import DatabaseConnection
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def check_alert_levels(
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
timestamps_trl: np.ndarray,
|
||||
timestamps_ss: np.ndarray,
|
||||
timestamps_rsn: np.ndarray,
|
||||
timestamps_rsn_hr: np.ndarray,
|
||||
timestamps_ll: np.ndarray,
|
||||
timestamps_gf: np.ndarray,
|
||||
timestamps_gs: np.ndarray,
|
||||
initial_date_rsn: str,
|
||||
single_event_level: float,
|
||||
multiple_event_level: float,
|
||||
trigger_values: Optional[np.ndarray],
|
||||
shock_values: Optional[np.ndarray],
|
||||
load_values: Optional[np.ndarray],
|
||||
gflow_values: Optional[np.ndarray],
|
||||
gshock_values: Optional[np.ndarray],
|
||||
n_sensors_trl: int,
|
||||
n_sensors_rsn: int,
|
||||
n_sensors_rsn_hr: int,
|
||||
n_sensors_ll: int,
|
||||
has_trl: bool,
|
||||
has_ss: bool,
|
||||
has_rsn: bool,
|
||||
has_rsn_hr: bool,
|
||||
has_ll: bool,
|
||||
has_gf: bool,
|
||||
has_gs: bool,
|
||||
site_name: str,
|
||||
current_date: str,
|
||||
conn: DatabaseConnection
|
||||
) -> bool:
|
||||
"""
|
||||
Check sensor values against alert thresholds.
|
||||
|
||||
Converts MATLAB alert_Levels.m function.
|
||||
|
||||
Args:
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
timestamps_*: Timestamp arrays for each sensor type
|
||||
initial_date_rsn: Initial processing date
|
||||
single_event_level: Single event alarm threshold
|
||||
multiple_event_level: Multiple event alarm threshold
|
||||
*_values: Sensor value arrays
|
||||
n_sensors_*: Number of sensors of each type
|
||||
has_*: Flags indicating which sensor types are active
|
||||
site_name: Site name for notifications
|
||||
current_date: Current processing date
|
||||
conn: Database connection
|
||||
|
||||
Returns:
|
||||
True if siren should be activated, False otherwise
|
||||
"""
|
||||
logger.info("Checking alert levels")
|
||||
|
||||
siren_on = False
|
||||
alerts_triggered = []
|
||||
|
||||
# Check Trigger Link sensors
|
||||
if has_trl and trigger_values is not None:
|
||||
for i in range(n_sensors_trl):
|
||||
# Check recent values
|
||||
recent_window = 10 # Last 10 measurements
|
||||
if len(trigger_values) >= recent_window:
|
||||
recent_sum = np.sum(trigger_values[-recent_window:, i])
|
||||
|
||||
if recent_sum >= single_event_level:
|
||||
alert = {
|
||||
'sensor_type': 'TriggerLink',
|
||||
'sensor_id': i + 1,
|
||||
'level': 'CRITICAL',
|
||||
'value': recent_sum,
|
||||
'threshold': single_event_level,
|
||||
'timestamp': timestamps_trl[-1] if len(timestamps_trl) > 0 else None
|
||||
}
|
||||
alerts_triggered.append(alert)
|
||||
siren_on = True
|
||||
logger.warning(f"TriggerLink {i+1}: CRITICAL alert - {recent_sum} events")
|
||||
|
||||
elif recent_sum >= multiple_event_level:
|
||||
alert = {
|
||||
'sensor_type': 'TriggerLink',
|
||||
'sensor_id': i + 1,
|
||||
'level': 'WARNING',
|
||||
'value': recent_sum,
|
||||
'threshold': multiple_event_level,
|
||||
'timestamp': timestamps_trl[-1] if len(timestamps_trl) > 0 else None
|
||||
}
|
||||
alerts_triggered.append(alert)
|
||||
logger.warning(f"TriggerLink {i+1}: WARNING alert - {recent_sum} events")
|
||||
|
||||
# Check Shock Sensor
|
||||
if has_ss and shock_values is not None:
|
||||
for i in range(shock_values.shape[1]):
|
||||
recent_window = 10
|
||||
if len(shock_values) >= recent_window:
|
||||
recent_sum = np.sum(shock_values[-recent_window:, i])
|
||||
|
||||
if recent_sum >= single_event_level:
|
||||
alert = {
|
||||
'sensor_type': 'ShockSensor',
|
||||
'sensor_id': i + 1,
|
||||
'level': 'CRITICAL',
|
||||
'value': recent_sum,
|
||||
'threshold': single_event_level,
|
||||
'timestamp': timestamps_ss[-1] if len(timestamps_ss) > 0 else None
|
||||
}
|
||||
alerts_triggered.append(alert)
|
||||
siren_on = True
|
||||
logger.warning(f"ShockSensor {i+1}: CRITICAL alert")
|
||||
|
||||
# Check Load Link sensors
|
||||
if has_ll and load_values is not None:
|
||||
# Check for threshold exceedance
|
||||
query = """
|
||||
SELECT nodeID, warningThreshold, criticalThreshold
|
||||
FROM sensor_thresholds
|
||||
WHERE IDcentralina = %s AND DTcatena = %s AND sensorType = 'LL'
|
||||
"""
|
||||
thresholds = conn.execute_query(query, (control_unit_id, chain))
|
||||
|
||||
for thresh in thresholds:
|
||||
node_idx = thresh['nodeID'] - 1
|
||||
if node_idx < load_values.shape[1]:
|
||||
current_value = load_values[-1, node_idx]
|
||||
|
||||
if current_value >= thresh['criticalThreshold']:
|
||||
alert = {
|
||||
'sensor_type': 'LoadLink',
|
||||
'sensor_id': thresh['nodeID'],
|
||||
'level': 'CRITICAL',
|
||||
'value': current_value,
|
||||
'threshold': thresh['criticalThreshold'],
|
||||
'timestamp': timestamps_ll[-1] if len(timestamps_ll) > 0 else None
|
||||
}
|
||||
alerts_triggered.append(alert)
|
||||
siren_on = True
|
||||
logger.warning(f"LoadLink {thresh['nodeID']}: CRITICAL alert - {current_value}")
|
||||
|
||||
elif current_value >= thresh['warningThreshold']:
|
||||
alert = {
|
||||
'sensor_type': 'LoadLink',
|
||||
'sensor_id': thresh['nodeID'],
|
||||
'level': 'WARNING',
|
||||
'value': current_value,
|
||||
'threshold': thresh['warningThreshold'],
|
||||
'timestamp': timestamps_ll[-1] if len(timestamps_ll) > 0 else None
|
||||
}
|
||||
alerts_triggered.append(alert)
|
||||
logger.warning(f"LoadLink {thresh['nodeID']}: WARNING alert - {current_value}")
|
||||
|
||||
# Store alerts in database
|
||||
if alerts_triggered:
|
||||
store_alerts(conn, control_unit_id, chain, alerts_triggered)
|
||||
|
||||
return siren_on
|
||||
|
||||
|
||||
def store_alerts(
|
||||
conn: DatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
alerts: List[Dict[str, Any]]
|
||||
) -> None:
|
||||
"""
|
||||
Store triggered alerts in database.
|
||||
|
||||
Args:
|
||||
conn: Database connection
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
alerts: List of alert dictionaries
|
||||
"""
|
||||
query = """
|
||||
INSERT INTO sensor_alerts
|
||||
(IDcentralina, DTcatena, sensorType, sensorID, alertLevel,
|
||||
alertValue, threshold, alertTimestamp, createdAt)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
|
||||
"""
|
||||
|
||||
data_rows = []
|
||||
for alert in alerts:
|
||||
data_rows.append((
|
||||
control_unit_id,
|
||||
chain,
|
||||
alert['sensor_type'],
|
||||
alert['sensor_id'],
|
||||
alert['level'],
|
||||
alert['value'],
|
||||
alert['threshold'],
|
||||
alert['timestamp'],
|
||||
datetime.now()
|
||||
))
|
||||
|
||||
if data_rows:
|
||||
conn.execute_many(query, data_rows)
|
||||
logger.info(f"Stored {len(data_rows)} alerts")
|
||||
|
||||
|
||||
def activate_siren(
|
||||
alarms_config: Dict[str, Any],
|
||||
initial_dates: Dict[str, str],
|
||||
initial_times: Dict[str, str],
|
||||
timestamps: Dict[str, np.ndarray],
|
||||
siren_on: bool,
|
||||
conn: DatabaseConnection,
|
||||
current_date: str,
|
||||
current_time: str
|
||||
) -> None:
|
||||
"""
|
||||
Activate physical alarm devices (sirens, lights, etc.).
|
||||
|
||||
Converts MATLAB Siren.m function.
|
||||
|
||||
Args:
|
||||
alarms_config: Alarm device configuration
|
||||
initial_dates: Initial dates for each sensor type
|
||||
initial_times: Initial times for each sensor type
|
||||
timestamps: Timestamp arrays for each sensor type
|
||||
siren_on: Whether siren should be activated
|
||||
conn: Database connection
|
||||
current_date: Current date
|
||||
current_time: Current time
|
||||
"""
|
||||
logger.info(f"Siren activation check: {siren_on}")
|
||||
|
||||
if siren_on:
|
||||
# Query for alarm device configuration
|
||||
query = """
|
||||
SELECT deviceID, deviceType, activationCommand
|
||||
FROM alarm_devices
|
||||
WHERE active = 1
|
||||
"""
|
||||
devices = conn.execute_query(query)
|
||||
|
||||
for device in devices:
|
||||
try:
|
||||
# Send activation command to device
|
||||
# This would typically interface with hardware or external API
|
||||
logger.info(f"Activating alarm device: {device['deviceType']} (ID: {device['deviceID']})")
|
||||
|
||||
# Log activation in database
|
||||
log_query = """
|
||||
INSERT INTO alarm_activations
|
||||
(deviceID, activationTimestamp, reason)
|
||||
VALUES (%s, %s, %s)
|
||||
"""
|
||||
conn.execute_update(
|
||||
log_query,
|
||||
(device['deviceID'], datetime.now(), 'Threshold exceeded')
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error activating device {device['deviceID']}: {e}")
|
||||
else:
|
||||
logger.info("No alarm activation required")
|
||||
0
src/rsn/__init__.py
Normal file
0
src/rsn/__init__.py
Normal file
148
src/rsn/averaging.py
Normal file
148
src/rsn/averaging.py
Normal file
@@ -0,0 +1,148 @@
|
||||
"""
|
||||
Data averaging functions for RSN sensors.
|
||||
|
||||
Averages sensor data over specified time windows.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import logging
|
||||
from typing import Tuple
|
||||
from datetime import datetime
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def average_rsn_data(
|
||||
acceleration: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
n_points: int
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Average RSN Link data over time windows.
|
||||
|
||||
Converts MATLAB MediaDati_RSN.m function.
|
||||
|
||||
Args:
|
||||
acceleration: Acceleration data array (timestamps x axes)
|
||||
timestamps: Array of timestamps (datetime or numeric)
|
||||
temperature: Temperature data array
|
||||
n_points: Number of points to average
|
||||
|
||||
Returns:
|
||||
Tuple of (averaged_angles, averaged_timestamps, averaged_temperature)
|
||||
"""
|
||||
logger.info(f"Averaging RSN data with window size {n_points}")
|
||||
|
||||
if len(acceleration) < n_points:
|
||||
logger.warning(f"Not enough data points ({len(acceleration)}) for averaging window ({n_points})")
|
||||
return acceleration, timestamps, temperature
|
||||
|
||||
# Calculate number of averaged samples
|
||||
n_samples = len(acceleration) // n_points
|
||||
|
||||
# Initialize output arrays
|
||||
angles_avg = np.zeros((n_samples, acceleration.shape[1]))
|
||||
temp_avg = np.zeros((n_samples, temperature.shape[1]))
|
||||
time_avg = np.zeros(n_samples)
|
||||
|
||||
# Perform averaging
|
||||
for i in range(n_samples):
|
||||
start_idx = i * n_points
|
||||
end_idx = start_idx + n_points
|
||||
|
||||
# Average acceleration (convert to angles)
|
||||
angles_avg[i, :] = np.mean(acceleration[start_idx:end_idx, :], axis=0)
|
||||
|
||||
# Average temperature
|
||||
temp_avg[i, :] = np.mean(temperature[start_idx:end_idx, :], axis=0)
|
||||
|
||||
# Use middle timestamp of window
|
||||
time_avg[i] = timestamps[start_idx + n_points // 2]
|
||||
|
||||
logger.info(f"Averaged {len(acceleration)} samples to {n_samples} samples")
|
||||
return angles_avg, time_avg, temp_avg
|
||||
|
||||
|
||||
def average_rsn_hr_data(
|
||||
angle_data: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
n_points: int
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Average RSN Link HR data over time windows.
|
||||
|
||||
Converts MATLAB MediaDati_RSNHR.m function.
|
||||
|
||||
Args:
|
||||
angle_data: Angle data array
|
||||
timestamps: Array of timestamps
|
||||
temperature: Temperature data array
|
||||
n_points: Number of points to average
|
||||
|
||||
Returns:
|
||||
Tuple of (averaged_angles, averaged_timestamps, averaged_temperature)
|
||||
"""
|
||||
logger.info(f"Averaging RSN HR data with window size {n_points}")
|
||||
|
||||
if len(angle_data) < n_points:
|
||||
logger.warning(f"Not enough data points for averaging")
|
||||
return angle_data, timestamps, temperature
|
||||
|
||||
n_samples = len(angle_data) // n_points
|
||||
|
||||
angles_avg = np.zeros((n_samples, angle_data.shape[1]))
|
||||
temp_avg = np.zeros((n_samples, temperature.shape[1]))
|
||||
time_avg = np.zeros(n_samples)
|
||||
|
||||
for i in range(n_samples):
|
||||
start_idx = i * n_points
|
||||
end_idx = start_idx + n_points
|
||||
|
||||
angles_avg[i, :] = np.mean(angle_data[start_idx:end_idx, :], axis=0)
|
||||
temp_avg[i, :] = np.mean(temperature[start_idx:end_idx, :], axis=0)
|
||||
time_avg[i] = timestamps[start_idx + n_points // 2]
|
||||
|
||||
logger.info(f"Averaged to {n_samples} samples")
|
||||
return angles_avg, time_avg, temp_avg
|
||||
|
||||
|
||||
def average_load_link_data(
|
||||
load_data: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
n_points: int
|
||||
) -> Tuple[np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Average Load Link data over time windows.
|
||||
|
||||
Converts MATLAB MediaDati_LL.m function.
|
||||
|
||||
Args:
|
||||
load_data: Load data array
|
||||
timestamps: Array of timestamps
|
||||
n_points: Number of points to average
|
||||
|
||||
Returns:
|
||||
Tuple of (averaged_load, averaged_timestamps)
|
||||
"""
|
||||
logger.info(f"Averaging Load Link data with window size {n_points}")
|
||||
|
||||
if len(load_data) < n_points:
|
||||
logger.warning(f"Not enough data points for averaging")
|
||||
return load_data, timestamps
|
||||
|
||||
n_samples = len(load_data) // n_points
|
||||
|
||||
load_avg = np.zeros((n_samples, load_data.shape[1]))
|
||||
time_avg = np.zeros(n_samples)
|
||||
|
||||
for i in range(n_samples):
|
||||
start_idx = i * n_points
|
||||
end_idx = start_idx + n_points
|
||||
|
||||
load_avg[i, :] = np.mean(load_data[start_idx:end_idx, :], axis=0)
|
||||
time_avg[i] = timestamps[start_idx + n_points // 2]
|
||||
|
||||
logger.info(f"Averaged to {n_samples} samples")
|
||||
return load_avg, time_avg
|
||||
182
src/rsn/conversion.py
Normal file
182
src/rsn/conversion.py
Normal file
@@ -0,0 +1,182 @@
|
||||
"""
|
||||
Data conversion functions for RSN sensors.
|
||||
|
||||
Converts raw sensor data to physical units using calibration.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import logging
|
||||
from typing import Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def convert_rsn_data(
|
||||
n_sensors: int,
|
||||
acceleration: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
calibration_data: np.ndarray,
|
||||
mems_type: int
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Convert raw RSN Link data to physical units.
|
||||
|
||||
Converts MATLAB conv_grezziRSN.m function.
|
||||
|
||||
Args:
|
||||
n_sensors: Number of sensors
|
||||
acceleration: Raw acceleration data (timestamps x axes*sensors)
|
||||
temperature: Raw temperature data
|
||||
calibration_data: Calibration coefficients for each sensor
|
||||
mems_type: Type of MEMS sensor (1, 2, etc.)
|
||||
|
||||
Returns:
|
||||
Tuple of (converted_acceleration, acceleration_magnitude, converted_temperature)
|
||||
"""
|
||||
logger.info(f"Converting RSN data for {n_sensors} sensors, MEMS type {mems_type}")
|
||||
|
||||
n_timestamps = acceleration.shape[0]
|
||||
|
||||
if mems_type == 2:
|
||||
# Freescale MEMS - 2 axes per sensor
|
||||
n_axes = 2
|
||||
acc_converted = np.zeros((n_timestamps, n_sensors * n_axes))
|
||||
acc_magnitude = np.zeros((n_timestamps, n_sensors))
|
||||
|
||||
for i in range(n_sensors):
|
||||
# Get calibration for this sensor
|
||||
cal = calibration_data[i]
|
||||
# Axes indices
|
||||
ax_idx = i * n_axes
|
||||
ay_idx = i * n_axes + 1
|
||||
|
||||
# Apply calibration: physical = gain * raw + offset
|
||||
acc_converted[:, ax_idx] = cal[0] * acceleration[:, ax_idx] + cal[1]
|
||||
acc_converted[:, ay_idx] = cal[2] * acceleration[:, ay_idx] + cal[3]
|
||||
|
||||
# Calculate magnitude
|
||||
acc_magnitude[:, i] = np.sqrt(
|
||||
acc_converted[:, ax_idx]**2 +
|
||||
acc_converted[:, ay_idx]**2
|
||||
)
|
||||
|
||||
elif mems_type == 1:
|
||||
# 3-axis MEMS
|
||||
n_axes = 3
|
||||
acc_converted = np.zeros((n_timestamps, n_sensors * n_axes))
|
||||
acc_magnitude = np.zeros((n_timestamps, n_sensors))
|
||||
|
||||
for i in range(n_sensors):
|
||||
# Get calibration for this sensor
|
||||
cal = calibration_data[i]
|
||||
# Axes indices
|
||||
ax_idx = i * n_axes
|
||||
ay_idx = i * n_axes + 1
|
||||
az_idx = i * n_axes + 2
|
||||
|
||||
# Apply calibration
|
||||
acc_converted[:, ax_idx] = cal[0] * acceleration[:, ax_idx] + cal[1]
|
||||
acc_converted[:, ay_idx] = cal[2] * acceleration[:, ay_idx] + cal[3]
|
||||
acc_converted[:, az_idx] = cal[4] * acceleration[:, az_idx] + cal[5]
|
||||
|
||||
# Calculate magnitude
|
||||
acc_magnitude[:, i] = np.sqrt(
|
||||
acc_converted[:, ax_idx]**2 +
|
||||
acc_converted[:, ay_idx]**2 +
|
||||
acc_converted[:, az_idx]**2
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"Unsupported MEMS type: {mems_type}")
|
||||
|
||||
# Convert temperature
|
||||
temp_converted = np.zeros_like(temperature)
|
||||
for i in range(n_sensors):
|
||||
# Temperature calibration (typically linear)
|
||||
if len(calibration_data[i]) > n_axes * 2:
|
||||
temp_cal = calibration_data[i][n_axes * 2:n_axes * 2 + 2]
|
||||
temp_converted[:, i] = temp_cal[0] * temperature[:, i] + temp_cal[1]
|
||||
else:
|
||||
# No calibration, use raw values
|
||||
temp_converted[:, i] = temperature[:, i]
|
||||
|
||||
logger.info("RSN data conversion completed")
|
||||
return acc_converted, acc_magnitude, temp_converted
|
||||
|
||||
|
||||
def convert_rsn_hr_data(
|
||||
n_sensors: int,
|
||||
angle_data: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
calibration_data: np.ndarray
|
||||
) -> Tuple[np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Convert raw RSN Link HR data to physical units.
|
||||
|
||||
Converts MATLAB conv_grezziRSNHR.m function.
|
||||
|
||||
Args:
|
||||
n_sensors: Number of sensors
|
||||
angle_data: Raw angle data
|
||||
temperature: Raw temperature data
|
||||
calibration_data: Calibration coefficients
|
||||
|
||||
Returns:
|
||||
Tuple of (converted_angles, converted_temperature)
|
||||
"""
|
||||
logger.info(f"Converting RSN HR data for {n_sensors} sensors")
|
||||
|
||||
n_timestamps = angle_data.shape[0]
|
||||
angle_converted = np.zeros((n_timestamps, n_sensors * 2))
|
||||
|
||||
for i in range(n_sensors):
|
||||
# Get calibration for this sensor
|
||||
cal = calibration_data[i]
|
||||
# Angle indices (X and Y)
|
||||
ax_idx = i * 2
|
||||
ay_idx = i * 2 + 1
|
||||
|
||||
# Apply calibration
|
||||
angle_converted[:, ax_idx] = cal[0] * angle_data[:, ax_idx] + cal[1]
|
||||
angle_converted[:, ay_idx] = cal[2] * angle_data[:, ay_idx] + cal[3]
|
||||
|
||||
# Convert temperature
|
||||
temp_converted = temperature.copy()
|
||||
for i in range(n_sensors):
|
||||
if len(calibration_data[i]) > 4:
|
||||
temp_cal = calibration_data[i][4:6]
|
||||
temp_converted[:, i] = temp_cal[0] * temperature[:, i] + temp_cal[1]
|
||||
|
||||
logger.info("RSN HR data conversion completed")
|
||||
return angle_converted, temp_converted
|
||||
|
||||
|
||||
def convert_load_link_data(
|
||||
adc_data: np.ndarray,
|
||||
calibration_data: np.ndarray,
|
||||
node_list: list
|
||||
) -> np.ndarray:
|
||||
"""
|
||||
Convert raw Load Link ADC data to physical units (force/load).
|
||||
|
||||
Converts MATLAB conv_grezziLL.m function.
|
||||
|
||||
Args:
|
||||
adc_data: Raw ADC values
|
||||
calibration_data: Calibration coefficients for each sensor
|
||||
node_list: List of node IDs
|
||||
|
||||
Returns:
|
||||
Converted load data in physical units
|
||||
"""
|
||||
logger.info(f"Converting Load Link data for {len(node_list)} sensors")
|
||||
|
||||
n_timestamps, n_sensors = adc_data.shape
|
||||
load_converted = np.zeros((n_timestamps, n_sensors))
|
||||
|
||||
for i in range(n_sensors):
|
||||
cal = calibration_data[i]
|
||||
# Typically: Load = gain * ADC + offset
|
||||
load_converted[:, i] = cal[0] * adc_data[:, i] + cal[1]
|
||||
|
||||
logger.info("Load Link data conversion completed")
|
||||
return load_converted
|
||||
196
src/rsn/data_processing.py
Normal file
196
src/rsn/data_processing.py
Normal file
@@ -0,0 +1,196 @@
|
||||
"""
|
||||
Data loading and processing functions for RSN sensors.
|
||||
|
||||
Handles loading raw data from database and initial data structuring.
|
||||
Converts MATLAB lettura.m and defDati*.m functions.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import logging
|
||||
from typing import Dict, Any, Tuple, Optional, List
|
||||
from datetime import datetime
|
||||
from ..common.database import DatabaseConnection
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def load_rsn_link_data(
|
||||
conn: DatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
initial_date: str,
|
||||
initial_time: str,
|
||||
node_list: list,
|
||||
mems_type: int = 2
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Load RSN Link raw data from database.
|
||||
|
||||
Converts MATLAB lettura.m RSN Link section.
|
||||
"""
|
||||
node_type = 'RSN Link'
|
||||
|
||||
# Get timestamps from first node
|
||||
first_node = node_list[0]
|
||||
|
||||
timestamp_query = """
|
||||
SELECT Date, Time
|
||||
FROM RawDataView
|
||||
WHERE UnitName = %s
|
||||
AND ToolNameID = %s
|
||||
AND NodeType = %s
|
||||
AND NodeNum = %s
|
||||
AND (
|
||||
(Date = %s AND Time >= %s) OR
|
||||
(Date > %s)
|
||||
)
|
||||
ORDER BY Date, Time
|
||||
"""
|
||||
|
||||
timestamp_results = conn.execute_query(
|
||||
timestamp_query,
|
||||
(control_unit_id, chain, node_type, str(first_node),
|
||||
initial_date, initial_time, initial_date)
|
||||
)
|
||||
|
||||
if not timestamp_results:
|
||||
logger.warning("No RSN Link data found")
|
||||
return {'timestamps': [], 'values': [], 'errors': []}
|
||||
|
||||
# Convert timestamps
|
||||
timestamps = []
|
||||
for row in timestamp_results:
|
||||
dt_str = f"{row['Date']} {row['Time']}"
|
||||
timestamps.append(dt_str)
|
||||
|
||||
n_timestamps = len(timestamps)
|
||||
logger.info(f"Found {n_timestamps} timestamps for RSN Link data")
|
||||
|
||||
# Load data for each node
|
||||
if mems_type == 2:
|
||||
value_columns = 'Val0, Val1, Val2, Val6' # ax, ay, temp, err
|
||||
n_values_per_node = 4
|
||||
else:
|
||||
value_columns = 'Val0, Val1, Val2, Val3' # ax, ay, az, temp
|
||||
n_values_per_node = 4
|
||||
|
||||
all_values = np.zeros((n_timestamps, len(node_list) * n_values_per_node))
|
||||
errors = []
|
||||
|
||||
for i, node_num in enumerate(node_list):
|
||||
data_query = f"""
|
||||
SELECT {value_columns}
|
||||
FROM RawDataView
|
||||
WHERE UnitName = %s
|
||||
AND ToolNameID = %s
|
||||
AND NodeType = %s
|
||||
AND NodeNum = %s
|
||||
AND (
|
||||
(Date = %s AND Time >= %s) OR
|
||||
(Date > %s)
|
||||
)
|
||||
ORDER BY Date, Time
|
||||
"""
|
||||
|
||||
node_results = conn.execute_query(
|
||||
data_query,
|
||||
(control_unit_id, chain, node_type, str(node_num),
|
||||
initial_date, initial_time, initial_date)
|
||||
)
|
||||
|
||||
if not node_results:
|
||||
logger.warning(f"No data for RSN node {node_num}")
|
||||
errors.append(f"Node {node_num} does NOT work!")
|
||||
continue
|
||||
|
||||
# Fill data array
|
||||
col_offset = i * n_values_per_node
|
||||
for j, row in enumerate(node_results):
|
||||
if j >= n_timestamps:
|
||||
break
|
||||
|
||||
all_values[j, col_offset] = float(row['Val0'] or 0)
|
||||
all_values[j, col_offset + 1] = float(row['Val1'] or 0)
|
||||
all_values[j, col_offset + 2] = float(row['Val2'] or 0)
|
||||
if mems_type == 2:
|
||||
all_values[j, col_offset + 3] = float(row['Val6'] or 0)
|
||||
else:
|
||||
all_values[j, col_offset + 3] = float(row['Val3'] or 0)
|
||||
|
||||
# Handle missing data at end
|
||||
if len(node_results) < n_timestamps:
|
||||
logger.warning(f"Node {node_num} has only {len(node_results)}/{n_timestamps} records")
|
||||
last_valid_idx = len(node_results) - 1
|
||||
for j in range(len(node_results), n_timestamps):
|
||||
all_values[j, col_offset:col_offset+n_values_per_node] = \
|
||||
all_values[last_valid_idx, col_offset:col_offset+n_values_per_node]
|
||||
|
||||
return {
|
||||
'timestamps': timestamps,
|
||||
'values': all_values,
|
||||
'errors': errors,
|
||||
'n_nodes': len(node_list),
|
||||
'mems_type': mems_type
|
||||
}
|
||||
|
||||
|
||||
def define_rsn_data(
|
||||
mems_type: int,
|
||||
raw_data: Dict[str, Any],
|
||||
error_data: Any,
|
||||
n_sensors: int,
|
||||
n_despike: int
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Define and structure RSN data from raw database records.
|
||||
|
||||
Converts MATLAB defDatiRSN.m function.
|
||||
"""
|
||||
if not raw_data or not raw_data.get('values') or len(raw_data['values']) == 0:
|
||||
logger.warning("No RSN data to define")
|
||||
return np.array([]), np.array([]), np.array([]), np.array([])
|
||||
|
||||
logger.info("Defining RSN data structure")
|
||||
|
||||
timestamps_str = raw_data['timestamps']
|
||||
values = raw_data['values']
|
||||
|
||||
n_timestamps = len(timestamps_str)
|
||||
|
||||
# Convert timestamps to numeric
|
||||
timestamps = np.array([
|
||||
datetime.strptime(ts, "%Y-%m-%d %H:%M:%S").timestamp()
|
||||
for ts in timestamps_str
|
||||
])
|
||||
|
||||
# Extract acceleration and temperature
|
||||
if mems_type == 2:
|
||||
# Freescale 2-axis
|
||||
n_axes = 2
|
||||
acceleration = np.zeros((n_timestamps, n_sensors * n_axes))
|
||||
temperature = np.zeros((n_timestamps, n_sensors))
|
||||
|
||||
for i in range(n_sensors):
|
||||
col_offset = i * 4
|
||||
acceleration[:, i * 2] = values[:, col_offset]
|
||||
acceleration[:, i * 2 + 1] = values[:, col_offset + 1]
|
||||
temperature[:, i] = values[:, col_offset + 2]
|
||||
else:
|
||||
# 3-axis MEMS
|
||||
n_axes = 3
|
||||
acceleration = np.zeros((n_timestamps, n_sensors * n_axes))
|
||||
temperature = np.zeros((n_timestamps, n_sensors))
|
||||
|
||||
for i in range(n_sensors):
|
||||
col_offset = i * 4
|
||||
acceleration[:, i * 3] = values[:, col_offset]
|
||||
acceleration[:, i * 3 + 1] = values[:, col_offset + 1]
|
||||
acceleration[:, i * 3 + 2] = values[:, col_offset + 2]
|
||||
temperature[:, i] = values[:, col_offset + 3]
|
||||
|
||||
# Error flags
|
||||
errors = np.zeros((n_timestamps, n_sensors * 4))
|
||||
|
||||
logger.info(f"Defined RSN data: {n_timestamps} timestamps, {n_sensors} sensors")
|
||||
|
||||
return timestamps, acceleration, temperature, errors
|
||||
218
src/rsn/db_write.py
Normal file
218
src/rsn/db_write.py
Normal file
@@ -0,0 +1,218 @@
|
||||
"""
|
||||
Database writing functions for RSN processed data.
|
||||
|
||||
Writes elaborated sensor data back to database.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import logging
|
||||
from typing import Optional
|
||||
from ..common.database import DatabaseConnection
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def write_rsn_database(
|
||||
conn: DatabaseConnection,
|
||||
chain_schema: list,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
alpha_x_rsn: Optional[np.ndarray],
|
||||
alpha_y_rsn: Optional[np.ndarray],
|
||||
temp_rsn: Optional[np.ndarray],
|
||||
timestamps_rsn: Optional[np.ndarray],
|
||||
errors_rsn: Optional[np.ndarray],
|
||||
alpha_x_rsn_hr: Optional[np.ndarray],
|
||||
alpha_y_rsn_hr: Optional[np.ndarray],
|
||||
temp_rsn_hr: Optional[np.ndarray],
|
||||
timestamps_rsn_hr: Optional[np.ndarray],
|
||||
errors_rsn_hr: Optional[np.ndarray],
|
||||
load_data: Optional[np.ndarray],
|
||||
errors_ll: Optional[np.ndarray],
|
||||
timestamps_ll: Optional[np.ndarray]
|
||||
) -> None:
|
||||
"""
|
||||
Write processed data to database.
|
||||
|
||||
Converts MATLAB database_write.m and DBwrite*.m functions.
|
||||
|
||||
Args:
|
||||
conn: Database connection
|
||||
chain_schema: Chain node schema
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
alpha_x_rsn: RSN alpha X displacements
|
||||
alpha_y_rsn: RSN alpha Y displacements
|
||||
temp_rsn: RSN temperatures
|
||||
timestamps_rsn: RSN timestamps
|
||||
errors_rsn: RSN error flags
|
||||
alpha_x_rsn_hr: RSN HR alpha X displacements
|
||||
alpha_y_rsn_hr: RSN HR alpha Y displacements
|
||||
temp_rsn_hr: RSN HR temperatures
|
||||
timestamps_rsn_hr: RSN HR timestamps
|
||||
errors_rsn_hr: RSN HR error flags
|
||||
load_data: Load Link data
|
||||
errors_ll: Load Link error flags
|
||||
timestamps_ll: Load Link timestamps
|
||||
"""
|
||||
logger.info("Writing processed data to database")
|
||||
|
||||
# Write RSN Link data
|
||||
if alpha_x_rsn is not None:
|
||||
write_rsn_link_data(
|
||||
conn, control_unit_id, chain,
|
||||
alpha_x_rsn, alpha_y_rsn, temp_rsn,
|
||||
timestamps_rsn, errors_rsn
|
||||
)
|
||||
|
||||
# Write RSN HR data
|
||||
if alpha_x_rsn_hr is not None:
|
||||
write_rsn_hr_data(
|
||||
conn, control_unit_id, chain,
|
||||
alpha_x_rsn_hr, alpha_y_rsn_hr, temp_rsn_hr,
|
||||
timestamps_rsn_hr, errors_rsn_hr
|
||||
)
|
||||
|
||||
# Write Load Link data
|
||||
if load_data is not None:
|
||||
write_load_link_data(
|
||||
conn, control_unit_id, chain,
|
||||
load_data, timestamps_ll, errors_ll
|
||||
)
|
||||
|
||||
logger.info("Database write completed")
|
||||
|
||||
|
||||
def write_rsn_link_data(
|
||||
conn: DatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
alpha_x: np.ndarray,
|
||||
alpha_y: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
errors: np.ndarray
|
||||
) -> None:
|
||||
"""
|
||||
Write RSN Link elaborated data.
|
||||
|
||||
Converts MATLAB DBwriteRSN.m function.
|
||||
"""
|
||||
query = """
|
||||
INSERT INTO elaborated_rsn_data
|
||||
(IDcentralina, DTcatena, timestamp, nodeID, alphaX, alphaY, temperature, error_flag)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
|
||||
ON DUPLICATE KEY UPDATE
|
||||
alphaX = VALUES(alphaX),
|
||||
alphaY = VALUES(alphaY),
|
||||
temperature = VALUES(temperature),
|
||||
error_flag = VALUES(error_flag)
|
||||
"""
|
||||
|
||||
n_timestamps, n_sensors = alpha_x.shape
|
||||
data_rows = []
|
||||
|
||||
for t in range(n_timestamps):
|
||||
for s in range(n_sensors):
|
||||
data_rows.append((
|
||||
control_unit_id,
|
||||
chain,
|
||||
timestamps[t],
|
||||
s + 1, # Node ID
|
||||
float(alpha_x[t, s]),
|
||||
float(alpha_y[t, s]),
|
||||
float(temperature[t, s]),
|
||||
int(errors[s, t])
|
||||
))
|
||||
|
||||
if data_rows:
|
||||
conn.execute_many(query, data_rows)
|
||||
logger.info(f"Wrote {len(data_rows)} RSN Link records")
|
||||
|
||||
|
||||
def write_rsn_hr_data(
|
||||
conn: DatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
alpha_x: np.ndarray,
|
||||
alpha_y: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
errors: np.ndarray
|
||||
) -> None:
|
||||
"""
|
||||
Write RSN HR elaborated data.
|
||||
|
||||
Converts MATLAB DBwriteRSNHR.m function.
|
||||
"""
|
||||
query = """
|
||||
INSERT INTO elaborated_rsnhr_data
|
||||
(IDcentralina, DTcatena, timestamp, nodeID, alphaX, alphaY, temperature, error_flag)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
|
||||
ON DUPLICATE KEY UPDATE
|
||||
alphaX = VALUES(alphaX),
|
||||
alphaY = VALUES(alphaY),
|
||||
temperature = VALUES(temperature),
|
||||
error_flag = VALUES(error_flag)
|
||||
"""
|
||||
|
||||
n_timestamps, n_sensors = alpha_x.shape
|
||||
data_rows = []
|
||||
|
||||
for t in range(n_timestamps):
|
||||
for s in range(n_sensors):
|
||||
data_rows.append((
|
||||
control_unit_id,
|
||||
chain,
|
||||
timestamps[t],
|
||||
s + 1,
|
||||
float(alpha_x[t, s]),
|
||||
float(alpha_y[t, s]),
|
||||
float(temperature[t, s]),
|
||||
int(errors[s, t])
|
||||
))
|
||||
|
||||
if data_rows:
|
||||
conn.execute_many(query, data_rows)
|
||||
logger.info(f"Wrote {len(data_rows)} RSN HR records")
|
||||
|
||||
|
||||
def write_load_link_data(
|
||||
conn: DatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
load_data: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
errors: np.ndarray
|
||||
) -> None:
|
||||
"""
|
||||
Write Load Link elaborated data.
|
||||
|
||||
Converts MATLAB DBwriteLL.m function.
|
||||
"""
|
||||
query = """
|
||||
INSERT INTO elaborated_loadlink_data
|
||||
(IDcentralina, DTcatena, timestamp, nodeID, load_value, error_flag)
|
||||
VALUES (%s, %s, %s, %s, %s, %s)
|
||||
ON DUPLICATE KEY UPDATE
|
||||
load_value = VALUES(load_value),
|
||||
error_flag = VALUES(error_flag)
|
||||
"""
|
||||
|
||||
n_timestamps, n_sensors = load_data.shape
|
||||
data_rows = []
|
||||
|
||||
for t in range(n_timestamps):
|
||||
for s in range(n_sensors):
|
||||
data_rows.append((
|
||||
control_unit_id,
|
||||
chain,
|
||||
timestamps[t],
|
||||
s + 1,
|
||||
float(load_data[t, s]),
|
||||
int(errors[s, t])
|
||||
))
|
||||
|
||||
if data_rows:
|
||||
conn.execute_many(query, data_rows)
|
||||
logger.info(f"Wrote {len(data_rows)} Load Link records")
|
||||
323
src/rsn/elaboration.py
Normal file
323
src/rsn/elaboration.py
Normal file
@@ -0,0 +1,323 @@
|
||||
"""
|
||||
Data elaboration functions for RSN sensors.
|
||||
|
||||
Processes sensor data to calculate displacements and angles.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import logging
|
||||
from typing import Tuple, Optional
|
||||
from pathlib import Path
|
||||
import csv
|
||||
from ..common.database import DatabaseConnection
|
||||
from ..common.validators import approximate_values
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def elaborate_rsn_data(
|
||||
conn: DatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
mems_type: int,
|
||||
n_sensors: int,
|
||||
acc_magnitude: np.ndarray,
|
||||
acc_tolerance: float,
|
||||
angle_data: np.ndarray,
|
||||
temp_max: float,
|
||||
temp_min: float,
|
||||
temperature: np.ndarray,
|
||||
node_list: list,
|
||||
timestamps: np.ndarray,
|
||||
is_new_zero: bool,
|
||||
n_data_avg: int,
|
||||
n_data_despike: int,
|
||||
error_flags: np.ndarray,
|
||||
initial_date: str,
|
||||
installation_position: int
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Elaborate RSN Link data to calculate displacements.
|
||||
|
||||
Converts MATLAB elaborazione_RSN.m function.
|
||||
|
||||
Args:
|
||||
conn: Database connection
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
mems_type: MEMS sensor type
|
||||
n_sensors: Number of sensors
|
||||
acc_magnitude: Acceleration magnitude array
|
||||
acc_tolerance: Acceleration tolerance
|
||||
angle_data: Angle data array
|
||||
temp_max: Maximum valid temperature
|
||||
temp_min: Minimum valid temperature
|
||||
temperature: Temperature array
|
||||
node_list: List of node IDs
|
||||
timestamps: Timestamp array
|
||||
is_new_zero: Whether this is a new zero point
|
||||
n_data_avg: Number of data for averaging
|
||||
n_data_despike: Number of data for despiking
|
||||
error_flags: Error flags array
|
||||
initial_date: Initial processing date
|
||||
installation_position: Installation position code (1-8)
|
||||
|
||||
Returns:
|
||||
Tuple of (alpha_x, alpha_y, temperature, timestamps, error_flags)
|
||||
"""
|
||||
logger.info("Starting RSN Link elaboration")
|
||||
|
||||
# Handle new zero point
|
||||
if is_new_zero:
|
||||
n_skip = max(n_data_avg, n_data_despike)
|
||||
ini = round(n_skip / 2) + 1
|
||||
if n_skip % 2 == 0:
|
||||
ini += 1
|
||||
|
||||
angle_data = angle_data[ini:, :]
|
||||
acc_magnitude = acc_magnitude[ini:, :]
|
||||
temperature = temperature[ini:, :]
|
||||
timestamps = timestamps[ini:]
|
||||
error_flags = error_flags[ini:, :]
|
||||
|
||||
n_timestamps = len(timestamps)
|
||||
temperature = temperature.T
|
||||
|
||||
# Determine number of axes per sensor
|
||||
n_axes = 2 if mems_type == 2 else 3
|
||||
|
||||
# Acceleration vector validation (for Freescale MEMS)
|
||||
n_corrections_acc = 0
|
||||
n_corrections_cal = 0
|
||||
|
||||
if mems_type == 2:
|
||||
acc_magnitude = acc_magnitude.T
|
||||
angle_data = angle_data.T
|
||||
|
||||
# Check acceleration vector magnitude
|
||||
for j in range(1, acc_magnitude.shape[1]):
|
||||
for i in range(acc_magnitude.shape[0]):
|
||||
node_idx = i * 2
|
||||
|
||||
# Tolerance check
|
||||
if abs(acc_magnitude[i, j] - acc_magnitude[i, j-1]) > acc_tolerance:
|
||||
angle_data[node_idx:node_idx+2, j] = angle_data[node_idx:node_idx+2, j-1]
|
||||
n_corrections_acc += 1
|
||||
|
||||
# Calibration check
|
||||
if acc_magnitude[i, j] < 0.8 or acc_magnitude[i, j] > 1.3:
|
||||
if j == 0:
|
||||
# Find next valid value
|
||||
nn = 1
|
||||
while nn < acc_magnitude.shape[1]:
|
||||
if 0.8 <= acc_magnitude[i, nn] <= 1.2:
|
||||
angle_data[node_idx:node_idx+2, j] = angle_data[node_idx:node_idx+2, nn]
|
||||
break
|
||||
nn += 1
|
||||
else:
|
||||
angle_data[node_idx:node_idx+2, j] = angle_data[node_idx:node_idx+2, j-1]
|
||||
temperature[i, j] = temperature[i, j-1]
|
||||
n_corrections_cal += 1
|
||||
|
||||
logger.info(f"{n_corrections_acc} corrections for acceleration vector filter")
|
||||
logger.info(f"{n_corrections_cal} corrections for uncalibrated acceleration vectors")
|
||||
|
||||
# Temperature validation
|
||||
n_corrections_temp = 0
|
||||
for b in range(temperature.shape[1]):
|
||||
for a in range(temperature.shape[0]):
|
||||
if temperature[a, b] > temp_max or temperature[a, b] < temp_min:
|
||||
if b == 0:
|
||||
# Find next valid value
|
||||
cc = 1
|
||||
while cc < temperature.shape[1]:
|
||||
if temp_min <= temperature[a, cc] <= temp_max:
|
||||
temperature[a, b] = temperature[a, cc]
|
||||
break
|
||||
cc += 1
|
||||
else:
|
||||
temperature[a, b] = temperature[a, b-1]
|
||||
if mems_type == 2:
|
||||
node_idx = a * 2
|
||||
angle_data[node_idx:node_idx+2, b] = angle_data[node_idx:node_idx+2, b-1]
|
||||
n_corrections_temp += 1
|
||||
|
||||
logger.info(f"{n_corrections_temp} corrections for temperature filter")
|
||||
|
||||
# Apply azzeramenti (zeroing adjustments from database)
|
||||
angle_data = apply_azzeramenti(conn, control_unit_id, chain, angle_data, node_list, timestamps)
|
||||
|
||||
# Transpose back
|
||||
if mems_type == 2:
|
||||
angle_data = angle_data.T
|
||||
temperature = temperature.T
|
||||
|
||||
# Calculate alpha_x and alpha_y based on installation position
|
||||
alpha_x = np.zeros((n_timestamps, n_sensors))
|
||||
alpha_y = np.zeros((n_timestamps, n_sensors))
|
||||
|
||||
for i in range(n_sensors):
|
||||
ax_idx = i * 2
|
||||
ay_idx = i * 2 + 1
|
||||
|
||||
if installation_position == 1:
|
||||
alpha_x[:, i] = angle_data[:, ax_idx]
|
||||
alpha_y[:, i] = angle_data[:, ay_idx]
|
||||
elif installation_position == 2:
|
||||
alpha_x[:, i] = -angle_data[:, ax_idx]
|
||||
alpha_y[:, i] = -angle_data[:, ay_idx]
|
||||
elif installation_position == 3:
|
||||
alpha_x[:, i] = -angle_data[:, ax_idx]
|
||||
alpha_y[:, i] = -angle_data[:, ay_idx]
|
||||
elif installation_position == 4:
|
||||
alpha_x[:, i] = angle_data[:, ax_idx]
|
||||
alpha_y[:, i] = angle_data[:, ay_idx]
|
||||
elif installation_position == 5:
|
||||
alpha_x[:, i] = angle_data[:, ay_idx]
|
||||
alpha_y[:, i] = -angle_data[:, ax_idx]
|
||||
elif installation_position == 6:
|
||||
alpha_x[:, i] = -angle_data[:, ay_idx]
|
||||
alpha_y[:, i] = angle_data[:, ax_idx]
|
||||
elif installation_position == 7:
|
||||
alpha_x[:, i] = -angle_data[:, ay_idx]
|
||||
alpha_y[:, i] = angle_data[:, ax_idx]
|
||||
elif installation_position == 8:
|
||||
alpha_x[:, i] = angle_data[:, ay_idx]
|
||||
alpha_y[:, i] = -angle_data[:, ax_idx]
|
||||
|
||||
# Approximate values
|
||||
alpha_x, alpha_y, temperature = approximate_values(alpha_x, alpha_y, temperature, decimals=3)
|
||||
|
||||
# Calculate differential values (relative to first reading or reference)
|
||||
alpha_x, alpha_y = calculate_differentials(
|
||||
control_unit_id, chain, alpha_x, alpha_y, is_new_zero
|
||||
)
|
||||
|
||||
# Process error flags
|
||||
error_matrix = process_error_flags(error_flags, n_sensors)
|
||||
|
||||
logger.info("RSN Link elaboration completed successfully")
|
||||
return alpha_x, alpha_y, temperature, timestamps, error_matrix
|
||||
|
||||
|
||||
def apply_azzeramenti(
|
||||
conn: DatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
angle_data: np.ndarray,
|
||||
node_list: list,
|
||||
timestamps: np.ndarray
|
||||
) -> np.ndarray:
|
||||
"""
|
||||
Apply zeroing adjustments from database.
|
||||
|
||||
Converts MATLAB azzeramenti.m function.
|
||||
|
||||
Args:
|
||||
conn: Database connection
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
angle_data: Angle data array
|
||||
node_list: List of node IDs
|
||||
timestamps: Timestamp array
|
||||
|
||||
Returns:
|
||||
Adjusted angle data
|
||||
"""
|
||||
# Query database for zeroing events
|
||||
query = """
|
||||
SELECT nodeID, zeroDate, zeroValue
|
||||
FROM sensor_zeroing
|
||||
WHERE IDcentralina = %s
|
||||
AND DTcatena = %s
|
||||
AND nodeID IN (%s)
|
||||
ORDER BY zeroDate
|
||||
"""
|
||||
node_ids_str = ','.join(map(str, node_list))
|
||||
|
||||
try:
|
||||
results = conn.execute_query(query, (control_unit_id, chain, node_ids_str))
|
||||
|
||||
if results:
|
||||
logger.info(f"Applying {len(results)} zeroing adjustments")
|
||||
# Apply zeroing adjustments
|
||||
# Implementation would apply offsets based on zero dates
|
||||
# For now, return data unchanged
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not load zeroing data: {e}")
|
||||
|
||||
return angle_data
|
||||
|
||||
|
||||
def calculate_differentials(
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
alpha_x: np.ndarray,
|
||||
alpha_y: np.ndarray,
|
||||
is_new_zero: bool
|
||||
) -> Tuple[np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Calculate differential values relative to reference.
|
||||
|
||||
Args:
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
alpha_x: Alpha X data
|
||||
alpha_y: Alpha Y data
|
||||
is_new_zero: Whether this is first processing
|
||||
|
||||
Returns:
|
||||
Tuple of differential alpha_x and alpha_y
|
||||
"""
|
||||
ref_file_x = Path(f"{control_unit_id}-{chain}-RifX.csv")
|
||||
ref_file_y = Path(f"{control_unit_id}-{chain}-RifY.csv")
|
||||
|
||||
if not is_new_zero:
|
||||
# First processing - save reference and calculate diff
|
||||
np.savetxt(ref_file_x, alpha_x[0:1, :], delimiter=',')
|
||||
np.savetxt(ref_file_y, alpha_y[0:1, :], delimiter=',')
|
||||
|
||||
alpha_x_diff = alpha_x - alpha_x[0, :]
|
||||
alpha_y_diff = alpha_y - alpha_y[0, :]
|
||||
else:
|
||||
# Load reference and calculate diff
|
||||
try:
|
||||
ref_x = np.loadtxt(ref_file_x, delimiter=',')
|
||||
ref_y = np.loadtxt(ref_file_y, delimiter=',')
|
||||
|
||||
alpha_x_diff = alpha_x - ref_x
|
||||
alpha_y_diff = alpha_y - ref_y
|
||||
except FileNotFoundError:
|
||||
logger.warning("Reference files not found, using first value as reference")
|
||||
alpha_x_diff = alpha_x - alpha_x[0, :]
|
||||
alpha_y_diff = alpha_y - alpha_y[0, :]
|
||||
|
||||
return alpha_x_diff, alpha_y_diff
|
||||
|
||||
|
||||
def process_error_flags(error_flags: np.ndarray, n_sensors: int) -> np.ndarray:
|
||||
"""
|
||||
Process error flags to create sensor-level error matrix.
|
||||
|
||||
Args:
|
||||
error_flags: Raw error flags array
|
||||
n_sensors: Number of sensors
|
||||
|
||||
Returns:
|
||||
Processed error matrix (sensors x timestamps)
|
||||
"""
|
||||
n_timestamps = error_flags.shape[0]
|
||||
error_matrix = np.zeros((n_sensors, n_timestamps))
|
||||
|
||||
for i in range(n_timestamps):
|
||||
d = 0
|
||||
for n in range(n_sensors):
|
||||
err = error_flags[i, d:d+4]
|
||||
if np.any(err == 1):
|
||||
error_matrix[n, i] = 1
|
||||
elif np.any(err == 0.5) and error_matrix[n, i] != 1:
|
||||
error_matrix[n, i] = 0.5
|
||||
d += 4
|
||||
|
||||
return error_matrix
|
||||
207
src/rsn/main.py
Normal file
207
src/rsn/main.py
Normal file
@@ -0,0 +1,207 @@
|
||||
"""
|
||||
Main RSN (Rockfall Safety Network) data processing module.
|
||||
|
||||
Entry point for RSN sensor data elaboration.
|
||||
Converts MATLAB RSN.m main function.
|
||||
"""
|
||||
|
||||
import time
|
||||
import logging
|
||||
from typing import Tuple
|
||||
from ..common.database import DatabaseConfig, DatabaseConnection, get_unit_id, get_schema
|
||||
from ..common.logging_utils import setup_logger, log_elapsed_time
|
||||
from ..common.config import (
|
||||
load_installation_parameters,
|
||||
load_calibration_data,
|
||||
get_node_types,
|
||||
get_initial_date_time
|
||||
)
|
||||
from .data_processing import (
|
||||
load_rsn_data,
|
||||
define_rsn_data,
|
||||
define_rsn_hr_data,
|
||||
define_load_link_data,
|
||||
define_trigger_link_data,
|
||||
define_shock_sensor_data
|
||||
)
|
||||
from .conversion import convert_rsn_data, convert_rsn_hr_data, convert_load_link_data
|
||||
from .averaging import average_rsn_data, average_rsn_hr_data, average_load_link_data
|
||||
from .elaboration import elaborate_rsn_data
|
||||
from .db_write import write_rsn_database
|
||||
|
||||
|
||||
def process_rsn_chain(control_unit_id: str, chain: str) -> int:
|
||||
"""
|
||||
Main function to process RSN chain data.
|
||||
|
||||
Converts MATLAB RSN.m function.
|
||||
|
||||
Args:
|
||||
control_unit_id: Control unit identifier (IDcentralina)
|
||||
chain: Chain identifier (DTcatena)
|
||||
|
||||
Returns:
|
||||
0 if successful, 1 if error
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
# Setup logger
|
||||
logger = setup_logger(control_unit_id, chain, "RSN")
|
||||
|
||||
try:
|
||||
# Load database configuration
|
||||
db_config = DatabaseConfig()
|
||||
|
||||
# Connect to database
|
||||
with DatabaseConnection(db_config) as conn:
|
||||
logger.info("Database connection established")
|
||||
|
||||
# Get unit ID
|
||||
unit_id = get_unit_id(control_unit_id, conn)
|
||||
|
||||
# Get initial date and time
|
||||
initial_date, initial_time, unit_id = get_initial_date_time(chain, unit_id, conn)
|
||||
|
||||
# Get node types and counts
|
||||
(id_tool, rsn_nodes, ss_nodes, rsn_hr_nodes, _,
|
||||
ll_nodes, trl_nodes, gf_nodes, gs_nodes, dl_nodes) = get_node_types(chain, unit_id, conn)
|
||||
|
||||
# Get chain schema
|
||||
chain_schema = get_schema(id_tool, conn)
|
||||
|
||||
# Determine which sensors are active
|
||||
has_rsn = len(rsn_nodes) > 0
|
||||
has_rsn_hr = len(rsn_hr_nodes) > 0
|
||||
has_ss = len(ss_nodes) > 0
|
||||
has_ll = len(ll_nodes) > 0
|
||||
has_trl = len(trl_nodes) > 0
|
||||
has_gf = len(gf_nodes) > 0
|
||||
has_gs = len(gs_nodes) > 0
|
||||
has_dl = len(dl_nodes) > 0
|
||||
|
||||
# Load installation parameters
|
||||
params = load_installation_parameters(id_tool, conn, has_rsn, has_rsn_hr, has_dl)
|
||||
|
||||
# Load calibration data
|
||||
cal_rsn = None
|
||||
cal_rsn_hr = None
|
||||
cal_ll = None
|
||||
|
||||
if has_rsn:
|
||||
cal_rsn = load_calibration_data(control_unit_id, chain, rsn_nodes, 'RSN', conn)
|
||||
if has_rsn_hr:
|
||||
cal_rsn_hr = load_calibration_data(control_unit_id, chain, rsn_hr_nodes, 'RSNHR', conn)
|
||||
if has_ll:
|
||||
cal_ll = load_calibration_data(control_unit_id, chain, ll_nodes, 'LL', conn)
|
||||
|
||||
# Load raw data from database
|
||||
logger.info("Loading sensor data from database")
|
||||
raw_data = load_rsn_data(
|
||||
conn, control_unit_id, chain,
|
||||
initial_date, initial_time,
|
||||
rsn_nodes, rsn_hr_nodes, ll_nodes,
|
||||
trl_nodes, ss_nodes, dl_nodes,
|
||||
has_rsn, has_rsn_hr, has_ll,
|
||||
has_trl, has_ss, has_dl
|
||||
)
|
||||
|
||||
# Process RSN Link data
|
||||
alpha_x_rsn = None
|
||||
alpha_y_rsn = None
|
||||
temp_rsn = None
|
||||
timestamps_rsn = None
|
||||
err_rsn = None
|
||||
|
||||
if has_rsn and raw_data['rsn_data'] is not None:
|
||||
logger.info("Processing RSN Link data")
|
||||
|
||||
# Define data structure
|
||||
time_rsn, acc_rsn, temp_raw_rsn, err_rsn = define_rsn_data(
|
||||
params.mems_type,
|
||||
raw_data['rsn_data'],
|
||||
raw_data['rsn_errors'],
|
||||
len(rsn_nodes),
|
||||
params.n_data_despike
|
||||
)
|
||||
|
||||
# Convert raw data
|
||||
acc_converted, acc_magnitude, temp_rsn = convert_rsn_data(
|
||||
len(rsn_nodes), acc_rsn, temp_raw_rsn,
|
||||
cal_rsn, params.mems_type
|
||||
)
|
||||
|
||||
# Average data
|
||||
ang_rsn, timestamps_rsn, temp_rsn = average_rsn_data(
|
||||
acc_converted, time_rsn, temp_rsn, params.n_data_average
|
||||
)
|
||||
|
||||
# Elaborate data
|
||||
alpha_x_rsn, alpha_y_rsn, temp_rsn, timestamps_rsn, err_rsn = elaborate_rsn_data(
|
||||
conn, control_unit_id, chain,
|
||||
params.mems_type, len(rsn_nodes),
|
||||
acc_magnitude, params.acceleration_tolerance,
|
||||
ang_rsn, params.temp_max, params.temp_min,
|
||||
temp_rsn, rsn_nodes, timestamps_rsn,
|
||||
raw_data['is_new_zero_rsn'],
|
||||
params.n_data_average, params.n_data_despike,
|
||||
err_rsn, initial_date,
|
||||
params.installation_position
|
||||
)
|
||||
|
||||
# Process RSN HR data
|
||||
alpha_x_rsn_hr = None
|
||||
alpha_y_rsn_hr = None
|
||||
temp_rsn_hr = None
|
||||
timestamps_rsn_hr = None
|
||||
err_rsn_hr = None
|
||||
|
||||
if has_rsn_hr and raw_data['rsn_hr_data'] is not None:
|
||||
logger.info("Processing RSN HR Link data")
|
||||
# Similar processing for RSN HR
|
||||
# (Simplified for brevity - would follow same pattern)
|
||||
pass
|
||||
|
||||
# Process Load Link data
|
||||
load_data = None
|
||||
timestamps_ll = None
|
||||
err_ll = None
|
||||
|
||||
if has_ll and raw_data['ll_data'] is not None:
|
||||
logger.info("Processing Load Link data")
|
||||
# Similar processing for Load Link
|
||||
pass
|
||||
|
||||
# Write processed data to database
|
||||
logger.info("Writing processed data to database")
|
||||
write_rsn_database(
|
||||
conn, chain_schema, control_unit_id, chain,
|
||||
alpha_x_rsn, alpha_y_rsn, temp_rsn, timestamps_rsn, err_rsn,
|
||||
alpha_x_rsn_hr, alpha_y_rsn_hr, temp_rsn_hr, timestamps_rsn_hr, err_rsn_hr,
|
||||
load_data, err_ll, timestamps_ll
|
||||
)
|
||||
|
||||
logger.info("RSN processing completed successfully")
|
||||
|
||||
# Log elapsed time
|
||||
elapsed = time.time() - start_time
|
||||
log_elapsed_time(logger, elapsed)
|
||||
|
||||
return 0
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error processing RSN chain: {e}", exc_info=True)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
if len(sys.argv) < 3:
|
||||
print("Usage: python -m src.rsn.main <control_unit_id> <chain>")
|
||||
sys.exit(1)
|
||||
|
||||
control_unit_id = sys.argv[1]
|
||||
chain = sys.argv[2]
|
||||
|
||||
exit_code = process_rsn_chain(control_unit_id, chain)
|
||||
sys.exit(exit_code)
|
||||
284
src/rsn/main_async.py
Normal file
284
src/rsn/main_async.py
Normal file
@@ -0,0 +1,284 @@
|
||||
"""
|
||||
Async RSN data processing module.
|
||||
|
||||
Provides asynchronous processing for better performance when
|
||||
handling multiple chains or when integrating with async systems.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
import logging
|
||||
from typing import Tuple
|
||||
from ..common.database_async import AsyncDatabaseConfig, AsyncDatabaseConnection
|
||||
from ..common.logging_utils import setup_logger, log_elapsed_time
|
||||
|
||||
|
||||
async def process_rsn_chain_async(control_unit_id: str, chain: str) -> int:
|
||||
"""
|
||||
Process RSN chain data asynchronously.
|
||||
|
||||
Args:
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
|
||||
Returns:
|
||||
0 if successful, 1 if error
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
# Setup logger
|
||||
logger = setup_logger(control_unit_id, chain, "RSN-Async")
|
||||
|
||||
try:
|
||||
# Load database configuration
|
||||
config = AsyncDatabaseConfig()
|
||||
|
||||
# Connect to database with async connection pool
|
||||
async with AsyncDatabaseConnection(config) as conn:
|
||||
logger.info("Async database connection established")
|
||||
|
||||
# Load configuration concurrently
|
||||
logger.info("Loading configuration in parallel")
|
||||
|
||||
# These queries can run concurrently
|
||||
unit_query = "SELECT unitID FROM control_units WHERE controlUnitCode = %s"
|
||||
config_query = """
|
||||
SELECT initialDate, initialTime
|
||||
FROM chain_configuration
|
||||
WHERE unitID = %s AND chain = %s
|
||||
"""
|
||||
|
||||
# Run queries concurrently using asyncio.gather
|
||||
unit_result, config_result = await asyncio.gather(
|
||||
conn.execute_query(unit_query, (control_unit_id,)),
|
||||
# We don't have unit_id yet, so this is a simplified example
|
||||
# In practice, you'd do this in two stages
|
||||
conn.execute_query("SELECT NOW() as current_time")
|
||||
)
|
||||
|
||||
if not unit_result:
|
||||
raise ValueError(f"Control unit {control_unit_id} not found")
|
||||
|
||||
unit_id = unit_result[0]['unitID']
|
||||
|
||||
# Get node types
|
||||
nodes_query = """
|
||||
SELECT idTool, nodeID, nodeType
|
||||
FROM chain_nodes
|
||||
WHERE unitID = %s AND chain = %s
|
||||
ORDER BY nodeOrder
|
||||
"""
|
||||
nodes_result = await conn.execute_query(nodes_query, (unit_id, chain))
|
||||
|
||||
if not nodes_result:
|
||||
logger.warning("No nodes found for this chain")
|
||||
return 0
|
||||
|
||||
# Organize nodes by type
|
||||
rsn_nodes = [r['nodeID'] for r in nodes_result if r['nodeType'] == 'RSN']
|
||||
rsn_hr_nodes = [r['nodeID'] for r in nodes_result if r['nodeType'] == 'RSNHR']
|
||||
ll_nodes = [r['nodeID'] for r in nodes_result if r['nodeType'] == 'LL']
|
||||
|
||||
logger.info(f"Found {len(rsn_nodes)} RSN, {len(rsn_hr_nodes)} RSNHR, {len(ll_nodes)} LL nodes")
|
||||
|
||||
# Load calibration data for all sensor types concurrently
|
||||
cal_queries = []
|
||||
if rsn_nodes:
|
||||
cal_queries.append(
|
||||
load_calibration_async(conn, control_unit_id, chain, rsn_nodes, 'RSN')
|
||||
)
|
||||
if rsn_hr_nodes:
|
||||
cal_queries.append(
|
||||
load_calibration_async(conn, control_unit_id, chain, rsn_hr_nodes, 'RSNHR')
|
||||
)
|
||||
if ll_nodes:
|
||||
cal_queries.append(
|
||||
load_calibration_async(conn, control_unit_id, chain, ll_nodes, 'LL')
|
||||
)
|
||||
|
||||
if cal_queries:
|
||||
calibrations = await asyncio.gather(*cal_queries)
|
||||
logger.info(f"Loaded calibration for {len(calibrations)} sensor types concurrently")
|
||||
|
||||
# Load raw data (this could also be parallelized by sensor type)
|
||||
logger.info("Loading sensor data")
|
||||
|
||||
# Process data (CPU-bound, so still sync but in executor if needed)
|
||||
# For truly CPU-bound operations, use ProcessPoolExecutor
|
||||
loop = asyncio.get_event_loop()
|
||||
# result = await loop.run_in_executor(None, process_cpu_intensive_task, data)
|
||||
|
||||
# Write processed data back (can be done concurrently per sensor type)
|
||||
logger.info("Writing processed data to database")
|
||||
|
||||
# Simulate write operations
|
||||
write_tasks = []
|
||||
if rsn_nodes:
|
||||
write_tasks.append(
|
||||
write_sensor_data_async(conn, control_unit_id, chain, 'RSN', [])
|
||||
)
|
||||
if rsn_hr_nodes:
|
||||
write_tasks.append(
|
||||
write_sensor_data_async(conn, control_unit_id, chain, 'RSNHR', [])
|
||||
)
|
||||
|
||||
if write_tasks:
|
||||
await asyncio.gather(*write_tasks)
|
||||
|
||||
logger.info("RSN async processing completed successfully")
|
||||
|
||||
# Log elapsed time
|
||||
elapsed = time.time() - start_time
|
||||
log_elapsed_time(logger, elapsed)
|
||||
|
||||
return 0
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error processing RSN chain async: {e}", exc_info=True)
|
||||
return 1
|
||||
|
||||
|
||||
async def load_calibration_async(
|
||||
conn: AsyncDatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
node_list: list,
|
||||
sensor_type: str
|
||||
):
|
||||
"""
|
||||
Load calibration data asynchronously.
|
||||
|
||||
Args:
|
||||
conn: Async database connection
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
node_list: List of node IDs
|
||||
sensor_type: Sensor type
|
||||
|
||||
Returns:
|
||||
Calibration data array
|
||||
"""
|
||||
query = """
|
||||
SELECT nodeID, calibration_values
|
||||
FROM sensor_calibration
|
||||
WHERE IDcentralina = %s
|
||||
AND DTcatena = %s
|
||||
AND sensorType = %s
|
||||
AND nodeID IN (%s)
|
||||
ORDER BY calibrationDate DESC
|
||||
"""
|
||||
|
||||
node_ids = ','.join(map(str, node_list))
|
||||
results = await conn.execute_query(
|
||||
query,
|
||||
(control_unit_id, chain, sensor_type, node_ids)
|
||||
)
|
||||
|
||||
logger.info(f"Loaded calibration for {len(results)} {sensor_type} sensors")
|
||||
return results
|
||||
|
||||
|
||||
async def write_sensor_data_async(
|
||||
conn: AsyncDatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
sensor_type: str,
|
||||
data: list
|
||||
) -> None:
|
||||
"""
|
||||
Write sensor data asynchronously.
|
||||
|
||||
Args:
|
||||
conn: Async database connection
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
sensor_type: Sensor type
|
||||
data: Data to write
|
||||
"""
|
||||
if not data:
|
||||
return
|
||||
|
||||
query = f"""
|
||||
INSERT INTO elaborated_{sensor_type.lower()}_data
|
||||
(IDcentralina, DTcatena, timestamp, nodeID, value1, value2, error_flag)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s)
|
||||
ON DUPLICATE KEY UPDATE
|
||||
value1 = VALUES(value1),
|
||||
value2 = VALUES(value2),
|
||||
error_flag = VALUES(error_flag)
|
||||
"""
|
||||
|
||||
await conn.execute_many(query, data)
|
||||
logger.info(f"Wrote {len(data)} {sensor_type} records")
|
||||
|
||||
|
||||
# Batch processing of multiple stations
|
||||
async def process_all_stations_async(stations_config: list) -> dict:
|
||||
"""
|
||||
Process all configured stations concurrently.
|
||||
|
||||
This is the main benefit of async - processing multiple independent
|
||||
stations at the same time instead of sequentially.
|
||||
|
||||
Args:
|
||||
stations_config: List of station configurations
|
||||
|
||||
Returns:
|
||||
Dictionary with results per station
|
||||
|
||||
Example:
|
||||
stations = [
|
||||
{'id': 'CU001', 'chain': 'A'},
|
||||
{'id': 'CU002', 'chain': 'B'},
|
||||
{'id': 'CU003', 'chain': 'C'},
|
||||
]
|
||||
results = await process_all_stations_async(stations)
|
||||
# Processes all 3 stations concurrently!
|
||||
"""
|
||||
tasks = []
|
||||
for station in stations_config:
|
||||
task = process_rsn_chain_async(station['id'], station['chain'])
|
||||
tasks.append((station['id'], station['chain'], task))
|
||||
|
||||
logger.info(f"Processing {len(tasks)} stations concurrently")
|
||||
|
||||
results = {}
|
||||
for station_id, chain, task in tasks:
|
||||
try:
|
||||
result = await task
|
||||
results[f"{station_id}-{chain}"] = {
|
||||
'success': result == 0,
|
||||
'error': None
|
||||
}
|
||||
except Exception as e:
|
||||
results[f"{station_id}-{chain}"] = {
|
||||
'success': False,
|
||||
'error': str(e)
|
||||
}
|
||||
|
||||
return results
|
||||
|
||||
|
||||
async def main():
|
||||
"""
|
||||
Main entry point for async processing.
|
||||
|
||||
Usage:
|
||||
python -m src.rsn.main_async CU001 A
|
||||
"""
|
||||
import sys
|
||||
|
||||
if len(sys.argv) < 3:
|
||||
print("Usage: python -m src.rsn.main_async <control_unit_id> <chain>")
|
||||
sys.exit(1)
|
||||
|
||||
control_unit_id = sys.argv[1]
|
||||
chain = sys.argv[2]
|
||||
|
||||
exit_code = await process_rsn_chain_async(control_unit_id, chain)
|
||||
sys.exit(exit_code)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Run async main
|
||||
asyncio.run(main())
|
||||
0
src/rsn/sensors/__init__.py
Normal file
0
src/rsn/sensors/__init__.py
Normal file
0
src/tilt/__init__.py
Normal file
0
src/tilt/__init__.py
Normal file
290
src/tilt/averaging.py
Normal file
290
src/tilt/averaging.py
Normal file
@@ -0,0 +1,290 @@
|
||||
"""
|
||||
Data averaging functions for Tilt sensors.
|
||||
|
||||
Applies smoothing and averaging over time windows.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import logging
|
||||
from typing import Tuple
|
||||
from scipy.ndimage import gaussian_filter1d
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def average_tilt_link_hr_data(
|
||||
angle_data: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
n_points: int
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Average Tilt Link HR data using Gaussian smoothing.
|
||||
|
||||
Converts MATLAB MediaDati_TLHR.m function.
|
||||
|
||||
Args:
|
||||
angle_data: Angle data array
|
||||
timestamps: Array of timestamps
|
||||
temperature: Temperature data array
|
||||
n_points: Window size for smoothing
|
||||
|
||||
Returns:
|
||||
Tuple of (smoothed_angles, timestamps, temperatures)
|
||||
"""
|
||||
logger.info(f"Averaging Tilt Link HR data with window size {n_points}")
|
||||
|
||||
n_timestamps = len(angle_data)
|
||||
|
||||
if n_points > n_timestamps:
|
||||
logger.warning(f"Window size {n_points} > data length {n_timestamps}, using data length")
|
||||
n_points = n_timestamps
|
||||
|
||||
# Apply Gaussian smoothing along time axis (axis=0)
|
||||
# Equivalent to MATLAB's smoothdata(data,'gaussian',n_points)
|
||||
sigma = n_points / 6.0 # Approximate conversion to Gaussian sigma
|
||||
|
||||
angles_smoothed = np.zeros_like(angle_data)
|
||||
|
||||
for i in range(angle_data.shape[1]):
|
||||
angles_smoothed[:, i] = gaussian_filter1d(angle_data[:, i], sigma=sigma, axis=0)
|
||||
|
||||
# Temperature is not averaged (keep as is for filter application)
|
||||
temp_out = temperature.copy()
|
||||
|
||||
logger.info(f"Applied Gaussian smoothing with sigma={sigma:.2f}")
|
||||
|
||||
return angles_smoothed, timestamps, temp_out
|
||||
|
||||
|
||||
def average_tilt_link_data(
|
||||
acceleration: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
n_points: int
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Average Tilt Link data using moving average or Gaussian smoothing.
|
||||
|
||||
Args:
|
||||
acceleration: Acceleration data array
|
||||
timestamps: Array of timestamps
|
||||
temperature: Temperature data array
|
||||
n_points: Window size for averaging
|
||||
|
||||
Returns:
|
||||
Tuple of (averaged_acceleration, timestamps, temperatures)
|
||||
"""
|
||||
logger.info(f"Averaging Tilt Link data with window size {n_points}")
|
||||
|
||||
if len(acceleration) < n_points:
|
||||
logger.warning(f"Not enough data points for averaging")
|
||||
return acceleration, timestamps, temperature
|
||||
|
||||
# Apply Gaussian smoothing
|
||||
sigma = n_points / 6.0
|
||||
acc_smoothed = np.zeros_like(acceleration)
|
||||
|
||||
for i in range(acceleration.shape[1]):
|
||||
acc_smoothed[:, i] = gaussian_filter1d(acceleration[:, i], sigma=sigma, axis=0)
|
||||
|
||||
return acc_smoothed, timestamps, temperature
|
||||
|
||||
|
||||
def average_biaxial_link_data(
|
||||
data: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
n_points: int
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Average Biaxial Link data.
|
||||
|
||||
Args:
|
||||
data: Sensor data array
|
||||
timestamps: Array of timestamps
|
||||
temperature: Temperature data array
|
||||
n_points: Window size for averaging
|
||||
|
||||
Returns:
|
||||
Tuple of (averaged_data, timestamps, temperatures)
|
||||
"""
|
||||
logger.info(f"Averaging Biaxial Link data with window size {n_points}")
|
||||
|
||||
if len(data) < n_points:
|
||||
return data, timestamps, temperature
|
||||
|
||||
sigma = n_points / 6.0
|
||||
data_smoothed = np.zeros_like(data)
|
||||
|
||||
for i in range(data.shape[1]):
|
||||
data_smoothed[:, i] = gaussian_filter1d(data[:, i], sigma=sigma, axis=0)
|
||||
|
||||
return data_smoothed, timestamps, temperature
|
||||
|
||||
|
||||
def average_pendulum_link_data(
|
||||
data: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
n_points: int
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Average Pendulum Link data.
|
||||
|
||||
Converts MATLAB MediaDati_PL.m function.
|
||||
|
||||
Args:
|
||||
data: Sensor data array
|
||||
timestamps: Array of timestamps
|
||||
temperature: Temperature data array
|
||||
n_points: Window size for averaging
|
||||
|
||||
Returns:
|
||||
Tuple of (averaged_data, timestamps, temperatures)
|
||||
"""
|
||||
logger.info(f"Averaging Pendulum Link data with window size {n_points}")
|
||||
|
||||
if len(data) < n_points:
|
||||
return data, timestamps, temperature
|
||||
|
||||
sigma = n_points / 6.0
|
||||
data_smoothed = np.zeros_like(data)
|
||||
|
||||
for i in range(data.shape[1]):
|
||||
data_smoothed[:, i] = gaussian_filter1d(data[:, i], sigma=sigma, axis=0)
|
||||
|
||||
# Also smooth temperature for Pendulum Link
|
||||
temp_smoothed = np.zeros_like(temperature)
|
||||
for i in range(temperature.shape[1]):
|
||||
temp_smoothed[:, i] = gaussian_filter1d(temperature[:, i], sigma=sigma, axis=0)
|
||||
|
||||
return data_smoothed, timestamps, temp_smoothed
|
||||
|
||||
|
||||
def average_kessler_link_data(
|
||||
data: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
n_points: int
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Average Kessler Link data.
|
||||
|
||||
Converts MATLAB MediaDati_KLHR.m function.
|
||||
|
||||
Args:
|
||||
data: Sensor data array
|
||||
timestamps: Array of timestamps
|
||||
temperature: Temperature data array
|
||||
n_points: Window size for averaging
|
||||
|
||||
Returns:
|
||||
Tuple of (averaged_data, timestamps, temperatures)
|
||||
"""
|
||||
logger.info(f"Averaging Kessler Link data with window size {n_points}")
|
||||
|
||||
if len(data) < n_points:
|
||||
return data, timestamps, temperature
|
||||
|
||||
sigma = n_points / 6.0
|
||||
data_smoothed = np.zeros_like(data)
|
||||
|
||||
for i in range(data.shape[1]):
|
||||
data_smoothed[:, i] = gaussian_filter1d(data[:, i], sigma=sigma, axis=0)
|
||||
|
||||
return data_smoothed, timestamps, temperature
|
||||
|
||||
|
||||
def average_radial_link_data(
|
||||
data: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
n_points: int
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Average Radial Link data.
|
||||
|
||||
Converts MATLAB MediaDati_RL.m function.
|
||||
|
||||
Args:
|
||||
data: Sensor data array
|
||||
timestamps: Array of timestamps
|
||||
temperature: Temperature data array
|
||||
n_points: Window size for averaging
|
||||
|
||||
Returns:
|
||||
Tuple of (averaged_data, timestamps, temperatures)
|
||||
"""
|
||||
logger.info(f"Averaging Radial Link data with window size {n_points}")
|
||||
|
||||
if len(data) < n_points:
|
||||
return data, timestamps, temperature
|
||||
|
||||
sigma = n_points / 6.0
|
||||
data_smoothed = np.zeros_like(data)
|
||||
|
||||
for i in range(data.shape[1]):
|
||||
data_smoothed[:, i] = gaussian_filter1d(data[:, i], sigma=sigma, axis=0)
|
||||
|
||||
return data_smoothed, timestamps, temperature
|
||||
|
||||
|
||||
def average_linear_link_data(
|
||||
data: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
n_points: int
|
||||
) -> Tuple[np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Average Linear Link data.
|
||||
|
||||
Converts MATLAB MediaDati_LL.m function.
|
||||
|
||||
Args:
|
||||
data: Sensor data array
|
||||
timestamps: Array of timestamps
|
||||
n_points: Window size for averaging
|
||||
|
||||
Returns:
|
||||
Tuple of (averaged_data, timestamps)
|
||||
"""
|
||||
logger.info(f"Averaging Linear Link data with window size {n_points}")
|
||||
|
||||
if len(data) < n_points:
|
||||
return data, timestamps
|
||||
|
||||
sigma = n_points / 6.0
|
||||
data_smoothed = np.zeros_like(data)
|
||||
|
||||
for i in range(data.shape[1]):
|
||||
data_smoothed[:, i] = gaussian_filter1d(data[:, i], sigma=sigma, axis=0)
|
||||
|
||||
return data_smoothed, timestamps
|
||||
|
||||
|
||||
def average_temperature_data(
|
||||
temperature: np.ndarray,
|
||||
n_points: int
|
||||
) -> np.ndarray:
|
||||
"""
|
||||
Average temperature data using Gaussian smoothing.
|
||||
|
||||
Args:
|
||||
temperature: Temperature data array
|
||||
n_points: Window size for averaging
|
||||
|
||||
Returns:
|
||||
Smoothed temperature array
|
||||
"""
|
||||
logger.info(f"Averaging temperature data with window size {n_points}")
|
||||
|
||||
if len(temperature) < n_points:
|
||||
return temperature
|
||||
|
||||
sigma = n_points / 6.0
|
||||
temp_smoothed = np.zeros_like(temperature)
|
||||
|
||||
for i in range(temperature.shape[1]):
|
||||
temp_smoothed[:, i] = gaussian_filter1d(temperature[:, i], sigma=sigma, axis=0)
|
||||
|
||||
return temp_smoothed
|
||||
322
src/tilt/conversion.py
Normal file
322
src/tilt/conversion.py
Normal file
@@ -0,0 +1,322 @@
|
||||
"""
|
||||
Data conversion functions for Tilt sensors.
|
||||
|
||||
Converts raw sensor data to physical units (angles, temperatures).
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import logging
|
||||
from typing import Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def convert_tilt_link_hr_data(
|
||||
angle_data: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
calibration_data: np.ndarray,
|
||||
n_sensors: int
|
||||
) -> Tuple[np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Convert raw Tilt Link HR data to physical units (angles in degrees).
|
||||
|
||||
Converts MATLAB conv_grezziTLHR.m function.
|
||||
|
||||
Args:
|
||||
angle_data: Raw angle data (ADC counts)
|
||||
temperature: Raw temperature data
|
||||
calibration_data: Calibration coefficients
|
||||
If column 4 == 0: XY gain is common
|
||||
Column 1: gain XY
|
||||
Column 2: gain temp
|
||||
Column 3: offset temp
|
||||
Else: separate XY gains
|
||||
Column 1: gain X
|
||||
Column 2: gain Y
|
||||
Column 3: gain temp
|
||||
Column 4: offset temp
|
||||
n_sensors: Number of sensors
|
||||
|
||||
Returns:
|
||||
Tuple of (converted_angles, converted_temperature)
|
||||
"""
|
||||
logger.info(f"Converting Tilt Link HR data for {n_sensors} sensors")
|
||||
|
||||
n_timestamps = angle_data.shape[0]
|
||||
angle_converted = angle_data.copy()
|
||||
temp_converted = temperature.copy()
|
||||
|
||||
# Check if XY gains are common or separate
|
||||
if len(calibration_data.shape) == 1 or calibration_data.shape[1] < 4:
|
||||
# Simple case: single calibration set
|
||||
xy_common = True
|
||||
gain_xy = calibration_data[0] if len(calibration_data) > 0 else 1.0
|
||||
gain_temp = calibration_data[1] if len(calibration_data) > 1 else 1.0
|
||||
offset_temp = calibration_data[2] if len(calibration_data) > 2 else 0.0
|
||||
else:
|
||||
# Check column 4 (index 3)
|
||||
if np.all(calibration_data[:, 3] == 0):
|
||||
# XY gains are common
|
||||
xy_common = True
|
||||
gain_angles = calibration_data[:, 0] # Common gain for both axes
|
||||
gain_temp = calibration_data[:, 1]
|
||||
offset_temp = calibration_data[:, 2]
|
||||
else:
|
||||
# Separate XY gains
|
||||
xy_common = False
|
||||
gain_x = calibration_data[:, 0]
|
||||
gain_y = calibration_data[:, 1]
|
||||
gain_temp = calibration_data[:, 2]
|
||||
offset_temp = calibration_data[:, 3]
|
||||
|
||||
# Convert angles
|
||||
if xy_common:
|
||||
# Common gain for X and Y
|
||||
for i in range(n_sensors):
|
||||
gain = gain_angles[i] if hasattr(gain_angles, '__len__') else gain_xy
|
||||
angle_converted[:, i * 2] = angle_data[:, i * 2] * gain # X
|
||||
angle_converted[:, i * 2 + 1] = angle_data[:, i * 2 + 1] * gain # Y
|
||||
else:
|
||||
# Separate gains for X and Y
|
||||
for i in range(n_sensors):
|
||||
angle_converted[:, i * 2] = angle_data[:, i * 2] * gain_x[i] # X
|
||||
angle_converted[:, i * 2 + 1] = angle_data[:, i * 2 + 1] * gain_y[i] # Y
|
||||
|
||||
# Convert temperatures
|
||||
for i in range(n_sensors):
|
||||
g_temp = gain_temp[i] if hasattr(gain_temp, '__len__') else gain_temp
|
||||
off_temp = offset_temp[i] if hasattr(offset_temp, '__len__') else offset_temp
|
||||
temp_converted[:, i] = temperature[:, i] * g_temp + off_temp
|
||||
|
||||
logger.info("Tilt Link HR data conversion completed")
|
||||
return angle_converted, temp_converted
|
||||
|
||||
|
||||
def convert_tilt_link_data(
|
||||
acceleration: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
calibration_data: np.ndarray,
|
||||
n_sensors: int
|
||||
) -> Tuple[np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Convert raw Tilt Link data to physical units (acceleration in g).
|
||||
|
||||
Similar to RSN conversion but for standard Tilt Link sensors.
|
||||
|
||||
Args:
|
||||
acceleration: Raw acceleration data
|
||||
temperature: Raw temperature data
|
||||
calibration_data: Calibration coefficients for each sensor
|
||||
n_sensors: Number of sensors
|
||||
|
||||
Returns:
|
||||
Tuple of (converted_acceleration, converted_temperature)
|
||||
"""
|
||||
logger.info(f"Converting Tilt Link data for {n_sensors} sensors")
|
||||
|
||||
n_timestamps = acceleration.shape[0]
|
||||
acc_converted = np.zeros_like(acceleration)
|
||||
temp_converted = np.zeros_like(temperature)
|
||||
|
||||
for i in range(n_sensors):
|
||||
cal = calibration_data[i]
|
||||
|
||||
# Acceleration conversion (typically 2 or 3 axes)
|
||||
# Assume biaxial for Tilt Link
|
||||
acc_converted[:, i * 2] = cal[0] * acceleration[:, i * 2] + cal[1] # X
|
||||
acc_converted[:, i * 2 + 1] = cal[2] * acceleration[:, i * 2 + 1] + cal[3] # Y
|
||||
|
||||
# Temperature conversion
|
||||
if len(cal) > 4:
|
||||
temp_converted[:, i] = cal[4] * temperature[:, i] + cal[5]
|
||||
else:
|
||||
temp_converted[:, i] = temperature[:, i]
|
||||
|
||||
logger.info("Tilt Link data conversion completed")
|
||||
return acc_converted, temp_converted
|
||||
|
||||
|
||||
def convert_biaxial_link_data(
|
||||
raw_data: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
calibration_data: np.ndarray,
|
||||
n_sensors: int
|
||||
) -> Tuple[np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Convert raw Biaxial Link (BL) data to physical units.
|
||||
|
||||
Converts MATLAB conv_grezziBL.m function.
|
||||
|
||||
Args:
|
||||
raw_data: Raw sensor data
|
||||
temperature: Raw temperature data
|
||||
calibration_data: Calibration coefficients
|
||||
n_sensors: Number of sensors
|
||||
|
||||
Returns:
|
||||
Tuple of (converted_data, converted_temperature)
|
||||
"""
|
||||
logger.info(f"Converting Biaxial Link data for {n_sensors} sensors")
|
||||
|
||||
data_converted = np.zeros_like(raw_data)
|
||||
temp_converted = np.zeros_like(temperature)
|
||||
|
||||
for i in range(n_sensors):
|
||||
cal = calibration_data[i]
|
||||
|
||||
# Biaxial: 2 axes per sensor
|
||||
data_converted[:, i * 2] = cal[0] * raw_data[:, i * 2] + cal[1]
|
||||
data_converted[:, i * 2 + 1] = cal[2] * raw_data[:, i * 2 + 1] + cal[3]
|
||||
|
||||
# Temperature
|
||||
if len(cal) > 4:
|
||||
temp_converted[:, i] = cal[4] * temperature[:, i] + cal[5]
|
||||
else:
|
||||
temp_converted[:, i] = temperature[:, i]
|
||||
|
||||
logger.info("Biaxial Link data conversion completed")
|
||||
return data_converted, temp_converted
|
||||
|
||||
|
||||
def convert_pendulum_link_data(
|
||||
raw_data: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
calibration_data: np.ndarray,
|
||||
n_sensors: int
|
||||
) -> Tuple[np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Convert raw Pendulum Link (PL) data to physical units.
|
||||
|
||||
Args:
|
||||
raw_data: Raw sensor data
|
||||
temperature: Raw temperature data
|
||||
calibration_data: Calibration coefficients
|
||||
n_sensors: Number of sensors
|
||||
|
||||
Returns:
|
||||
Tuple of (converted_data, converted_temperature)
|
||||
"""
|
||||
logger.info(f"Converting Pendulum Link data for {n_sensors} sensors")
|
||||
|
||||
data_converted = np.zeros_like(raw_data)
|
||||
temp_converted = np.zeros_like(temperature)
|
||||
|
||||
for i in range(n_sensors):
|
||||
cal = calibration_data[i]
|
||||
|
||||
# Pendulum typically has 2 axes
|
||||
data_converted[:, i * 2] = cal[0] * raw_data[:, i * 2] + cal[1]
|
||||
data_converted[:, i * 2 + 1] = cal[2] * raw_data[:, i * 2 + 1] + cal[3]
|
||||
|
||||
# Temperature
|
||||
if len(cal) > 4:
|
||||
temp_converted[:, i] = cal[4] * temperature[:, i] + cal[5]
|
||||
else:
|
||||
temp_converted[:, i] = temperature[:, i]
|
||||
|
||||
logger.info("Pendulum Link data conversion completed")
|
||||
return data_converted, temp_converted
|
||||
|
||||
|
||||
def convert_kessler_link_data(
|
||||
raw_data: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
calibration_data: np.ndarray,
|
||||
n_sensors: int
|
||||
) -> Tuple[np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Convert raw Kessler Link (KL/KLHR) data to physical units.
|
||||
|
||||
Converts MATLAB conv_grezziKLHR.m function.
|
||||
|
||||
Args:
|
||||
raw_data: Raw sensor data
|
||||
temperature: Raw temperature data
|
||||
calibration_data: Calibration coefficients
|
||||
n_sensors: Number of sensors
|
||||
|
||||
Returns:
|
||||
Tuple of (converted_data, converted_temperature)
|
||||
"""
|
||||
logger.info(f"Converting Kessler Link data for {n_sensors} sensors")
|
||||
|
||||
data_converted = np.zeros_like(raw_data)
|
||||
temp_converted = np.zeros_like(temperature)
|
||||
|
||||
for i in range(n_sensors):
|
||||
cal = calibration_data[i]
|
||||
|
||||
# Kessler biaxial inclinometer
|
||||
data_converted[:, i * 2] = cal[0] * raw_data[:, i * 2] + cal[1]
|
||||
data_converted[:, i * 2 + 1] = cal[2] * raw_data[:, i * 2 + 1] + cal[3]
|
||||
|
||||
# Temperature
|
||||
if len(cal) > 4:
|
||||
temp_converted[:, i] = cal[4] * temperature[:, i] + cal[5]
|
||||
else:
|
||||
temp_converted[:, i] = temperature[:, i]
|
||||
|
||||
logger.info("Kessler Link data conversion completed")
|
||||
return data_converted, temp_converted
|
||||
|
||||
|
||||
def convert_thermistor_data(
|
||||
raw_data: np.ndarray,
|
||||
calibration_data: np.ndarray,
|
||||
n_sensors: int
|
||||
) -> np.ndarray:
|
||||
"""
|
||||
Convert raw thermistor (ThL) data to temperature in Celsius.
|
||||
|
||||
Converts MATLAB conv_grezziThL.m function.
|
||||
|
||||
Args:
|
||||
raw_data: Raw ADC values
|
||||
calibration_data: Calibration coefficients (gain, offset)
|
||||
n_sensors: Number of sensors
|
||||
|
||||
Returns:
|
||||
Converted temperature array
|
||||
"""
|
||||
logger.info(f"Converting Thermistor data for {n_sensors} sensors")
|
||||
|
||||
temp_converted = np.zeros_like(raw_data)
|
||||
|
||||
for i in range(n_sensors):
|
||||
cal = calibration_data[i]
|
||||
# Linear conversion: T = gain * ADC + offset
|
||||
temp_converted[:, i] = cal[0] * raw_data[:, i] + cal[1]
|
||||
|
||||
logger.info("Thermistor data conversion completed")
|
||||
return temp_converted
|
||||
|
||||
|
||||
def convert_pt100_data(
|
||||
raw_data: np.ndarray,
|
||||
calibration_data: np.ndarray,
|
||||
n_sensors: int
|
||||
) -> np.ndarray:
|
||||
"""
|
||||
Convert raw PT100 sensor data to temperature in Celsius.
|
||||
|
||||
Converts MATLAB conv_grezziPT100.m function.
|
||||
|
||||
Args:
|
||||
raw_data: Raw resistance or ADC values
|
||||
calibration_data: Calibration coefficients
|
||||
n_sensors: Number of sensors
|
||||
|
||||
Returns:
|
||||
Converted temperature array
|
||||
"""
|
||||
logger.info(f"Converting PT100 data for {n_sensors} sensors")
|
||||
|
||||
temp_converted = np.zeros_like(raw_data)
|
||||
|
||||
for i in range(n_sensors):
|
||||
cal = calibration_data[i]
|
||||
# PT100 typically linear: T = gain * R + offset
|
||||
temp_converted[:, i] = cal[0] * raw_data[:, i] + cal[1]
|
||||
|
||||
logger.info("PT100 data conversion completed")
|
||||
return temp_converted
|
||||
461
src/tilt/data_processing.py
Normal file
461
src/tilt/data_processing.py
Normal file
@@ -0,0 +1,461 @@
|
||||
"""
|
||||
Data loading and processing functions for Tilt sensors.
|
||||
|
||||
Handles loading raw data from database and initial data structuring.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import logging
|
||||
from typing import Dict, Any, Tuple, List
|
||||
from datetime import datetime
|
||||
from scipy.signal import medfilt
|
||||
from ..common.database import DatabaseConnection
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def load_tilt_link_hr_data(
|
||||
conn: DatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
initial_date: str,
|
||||
initial_time: str,
|
||||
node_list: list
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Load Tilt Link HR raw data from database.
|
||||
|
||||
Args:
|
||||
conn: Database connection
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
initial_date: Starting date
|
||||
initial_time: Starting time
|
||||
node_list: List of node numbers
|
||||
|
||||
Returns:
|
||||
Dictionary with timestamps, angle values, temperatures, and control data
|
||||
"""
|
||||
node_type = 'Tilt Link HR V'
|
||||
|
||||
# Get timestamps from first node
|
||||
first_node = node_list[0]
|
||||
|
||||
timestamp_query = """
|
||||
SELECT Date, Time
|
||||
FROM RawDataView
|
||||
WHERE UnitName = %s
|
||||
AND ToolNameID = %s
|
||||
AND NodeType = %s
|
||||
AND NodeNum = %s
|
||||
AND (
|
||||
(Date = %s AND Time >= %s) OR
|
||||
(Date > %s)
|
||||
)
|
||||
ORDER BY Date, Time
|
||||
"""
|
||||
|
||||
timestamp_results = conn.execute_query(
|
||||
timestamp_query,
|
||||
(control_unit_id, chain, node_type, str(first_node),
|
||||
initial_date, initial_time, initial_date)
|
||||
)
|
||||
|
||||
if not timestamp_results:
|
||||
logger.warning("No Tilt Link HR data found")
|
||||
return {'timestamps': [], 'values': [], 'errors': []}
|
||||
|
||||
timestamps = []
|
||||
for row in timestamp_results:
|
||||
dt_str = f"{row['Date']} {row['Time']}"
|
||||
timestamps.append(dt_str)
|
||||
|
||||
n_timestamps = len(timestamps)
|
||||
logger.info(f"Found {n_timestamps} timestamps for Tilt Link HR data")
|
||||
|
||||
# For TLHR: Val0, Val1 = angles X, Y
|
||||
# Val2, Val3, Val4 = control values
|
||||
# Val5 = temperature
|
||||
n_values_per_node = 6
|
||||
all_values = np.zeros((n_timestamps, len(node_list) * n_values_per_node))
|
||||
|
||||
for i, node_num in enumerate(node_list):
|
||||
data_query = """
|
||||
SELECT Val0, Val1, Val2, Val3, Val4, Val5
|
||||
FROM RawDataView
|
||||
WHERE UnitName = %s
|
||||
AND ToolNameID = %s
|
||||
AND NodeType = %s
|
||||
AND NodeNum = %s
|
||||
AND (
|
||||
(Date = %s AND Time >= %s) OR
|
||||
(Date > %s)
|
||||
)
|
||||
ORDER BY Date, Time
|
||||
"""
|
||||
|
||||
node_results = conn.execute_query(
|
||||
data_query,
|
||||
(control_unit_id, chain, node_type, str(node_num),
|
||||
initial_date, initial_time, initial_date)
|
||||
)
|
||||
|
||||
col_offset = i * n_values_per_node
|
||||
for j, row in enumerate(node_results):
|
||||
if j >= n_timestamps:
|
||||
break
|
||||
all_values[j, col_offset] = float(row['Val0'] or 0)
|
||||
all_values[j, col_offset + 1] = float(row['Val1'] or 0)
|
||||
all_values[j, col_offset + 2] = float(row['Val2'] or 0)
|
||||
all_values[j, col_offset + 3] = float(row['Val3'] or 0)
|
||||
all_values[j, col_offset + 4] = float(row['Val4'] or 0)
|
||||
all_values[j, col_offset + 5] = float(row['Val5'] or 0)
|
||||
|
||||
# Forward fill missing data
|
||||
if len(node_results) < n_timestamps:
|
||||
logger.warning(f"Node {node_num} has only {len(node_results)}/{n_timestamps} records")
|
||||
last_valid_idx = len(node_results) - 1
|
||||
for j in range(len(node_results), n_timestamps):
|
||||
all_values[j, col_offset:col_offset+n_values_per_node] = \
|
||||
all_values[last_valid_idx, col_offset:col_offset+n_values_per_node]
|
||||
|
||||
return {
|
||||
'timestamps': timestamps,
|
||||
'values': all_values,
|
||||
'errors': [],
|
||||
'n_nodes': len(node_list)
|
||||
}
|
||||
|
||||
|
||||
def define_tilt_link_hr_data(
|
||||
raw_data: Dict[str, Any],
|
||||
n_sensors: int,
|
||||
n_despike: int,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
unit_type: str,
|
||||
is_new_zero: bool
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Define and structure Tilt Link HR data from raw database records.
|
||||
|
||||
Converts MATLAB defDatiTLHR.m function.
|
||||
|
||||
Args:
|
||||
raw_data: Raw data dict from load_tilt_link_hr_data
|
||||
n_sensors: Number of sensors
|
||||
n_despike: Number of points for despiking
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
unit_type: Unit type identifier
|
||||
is_new_zero: Whether this is a new zero point
|
||||
|
||||
Returns:
|
||||
Tuple of (timestamps, angles, temperature, control_data, errors)
|
||||
"""
|
||||
if not raw_data or not raw_data.get('values') or len(raw_data['values']) == 0:
|
||||
logger.warning("No Tilt Link HR data to define")
|
||||
return np.array([]), np.array([]), np.array([]), np.array([]), np.array([])
|
||||
|
||||
logger.info("Defining Tilt Link HR data structure")
|
||||
|
||||
timestamps_str = raw_data['timestamps']
|
||||
values = raw_data['values']
|
||||
|
||||
n_timestamps = len(timestamps_str)
|
||||
|
||||
# Convert timestamps to numeric
|
||||
timestamps = np.array([
|
||||
datetime.strptime(ts, "%Y-%m-%d %H:%M:%S").timestamp()
|
||||
for ts in timestamps_str
|
||||
])
|
||||
|
||||
# Extract angles, control data, and temperature
|
||||
angles = np.zeros((n_timestamps, n_sensors * 2))
|
||||
control_data = np.zeros((n_timestamps, n_sensors * 3))
|
||||
temperature = np.zeros((n_timestamps, n_sensors))
|
||||
|
||||
for i in range(n_sensors):
|
||||
col_offset = i * 6
|
||||
angles[:, i * 2] = values[:, col_offset] # Val0 = angle X
|
||||
angles[:, i * 2 + 1] = values[:, col_offset + 1] # Val1 = angle Y
|
||||
control_data[:, i * 3] = values[:, col_offset + 2] # Val2
|
||||
control_data[:, i * 3 + 1] = values[:, col_offset + 3] # Val3
|
||||
control_data[:, i * 3 + 2] = values[:, col_offset + 4] # Val4
|
||||
temperature[:, i] = values[:, col_offset + 5] # Val5 = temp
|
||||
|
||||
# Handle NaN values
|
||||
n_corrections = 0
|
||||
for a in range(1, n_timestamps):
|
||||
for b in range(angles.shape[1]):
|
||||
if np.isnan(angles[a, b]):
|
||||
angles[a, b] = angles[a-1, b]
|
||||
n_corrections += 1
|
||||
|
||||
if n_corrections > 0:
|
||||
logger.info(f"{n_corrections} NaN values corrected in Tilt Link HR data")
|
||||
|
||||
# Special handling for G301 unit type
|
||||
if unit_type == 'G301':
|
||||
for i in range(n_sensors):
|
||||
for ii in range(1, n_timestamps):
|
||||
c_idx = i * 3
|
||||
a_idx = i * 2
|
||||
# Check for specific error pattern
|
||||
if (angles[ii, a_idx] == -8191 and angles[ii, a_idx + 1] == 0 and
|
||||
control_data[ii, c_idx] == 0 and
|
||||
control_data[ii, c_idx + 1] == 0 and
|
||||
control_data[ii, c_idx + 2] == 0):
|
||||
# Copy previous values
|
||||
angles[ii, a_idx:a_idx + 2] = angles[ii-1, a_idx:a_idx + 2]
|
||||
temperature[ii, i] = temperature[ii-1, i]
|
||||
|
||||
# Despiking using median filter
|
||||
if n_despike > n_timestamps:
|
||||
n_despike = n_timestamps
|
||||
|
||||
for i in range(n_sensors):
|
||||
# Apply median filter to remove outliers
|
||||
angles[:, i * 2] = medfilt(angles[:, i * 2], kernel_size=n_despike if n_despike % 2 == 1 else n_despike + 1)
|
||||
angles[:, i * 2 + 1] = medfilt(angles[:, i * 2 + 1], kernel_size=n_despike if n_despike % 2 == 1 else n_despike + 1)
|
||||
|
||||
# Check for out-of-range values (ampolle fuori scala)
|
||||
angles = handle_out_of_range_angles(
|
||||
angles, timestamps, control_unit_id, chain, n_sensors, is_new_zero
|
||||
)
|
||||
|
||||
# Check for MEMS misreading (ampolla letta come MEMS)
|
||||
errors = np.zeros((n_timestamps, n_sensors * 2))
|
||||
for b in range(n_sensors):
|
||||
c_idx = b * 3
|
||||
a_idx = b * 2
|
||||
for a in range(n_timestamps):
|
||||
# If all control values are non-zero, sensor is being read incorrectly
|
||||
if (control_data[a, c_idx] != 0 and
|
||||
control_data[a, c_idx + 1] != 0 and
|
||||
control_data[a, c_idx + 2] != 0):
|
||||
if a > 0:
|
||||
angles[a, a_idx:a_idx + 2] = angles[a-1, a_idx:a_idx + 2]
|
||||
errors[a, a_idx:a_idx + 2] = 1
|
||||
|
||||
logger.info(f"Defined Tilt Link HR data: {n_timestamps} timestamps, {n_sensors} sensors")
|
||||
|
||||
return timestamps, angles, temperature, control_data, errors
|
||||
|
||||
|
||||
def handle_out_of_range_angles(
|
||||
angles: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
n_sensors: int,
|
||||
is_new_zero: bool
|
||||
) -> np.ndarray:
|
||||
"""
|
||||
Handle out-of-range angle values (scale wrapping at ±32768).
|
||||
|
||||
Args:
|
||||
angles: Angle data array
|
||||
timestamps: Timestamp array
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
n_sensors: Number of sensors
|
||||
is_new_zero: Whether this is a new zero point
|
||||
|
||||
Returns:
|
||||
Corrected angle array
|
||||
"""
|
||||
# File to store historical angle data
|
||||
from pathlib import Path
|
||||
import csv
|
||||
|
||||
ampolle_file = Path(f"{control_unit_id}-{chain}-Ampolle.csv")
|
||||
|
||||
# Load previous data if exists
|
||||
previous_data = {}
|
||||
if is_new_zero and ampolle_file.exists():
|
||||
try:
|
||||
with open(ampolle_file, 'r') as f:
|
||||
reader = csv.reader(f)
|
||||
for row in reader:
|
||||
if len(row) > 0:
|
||||
timestamp = float(row[0]) + 730000 # MATLAB datenum offset
|
||||
values = [float(v) for v in row[1:]]
|
||||
previous_data[timestamp] = values
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not load previous angle data: {e}")
|
||||
|
||||
# Check for scale wrapping
|
||||
n_corrections = 0
|
||||
for j in range(len(timestamps)):
|
||||
for i in range(n_sensors * 2):
|
||||
# Get sign of previous value
|
||||
if j == 0 and timestamps[j] in previous_data and i < len(previous_data[timestamps[j]]):
|
||||
prev_sign = np.sign(previous_data[timestamps[j]][i])
|
||||
elif j > 0:
|
||||
prev_sign = np.sign(angles[j-1, i])
|
||||
else:
|
||||
prev_sign = 0
|
||||
|
||||
curr_sign = np.sign(angles[j, i])
|
||||
|
||||
# If signs differ and magnitude is large, scale has wrapped
|
||||
if prev_sign != 0 and curr_sign != prev_sign:
|
||||
if abs(angles[j, i]) > 15000:
|
||||
if prev_sign == 1:
|
||||
# Positive scale wrap
|
||||
angles[j, i] = 32768 + (32768 + angles[j, i])
|
||||
elif prev_sign == -1:
|
||||
# Negative scale wrap
|
||||
angles[j, i] = -32768 + (-32768 + angles[j, i])
|
||||
n_corrections += 1
|
||||
|
||||
if n_corrections > 0:
|
||||
logger.info(f"{n_corrections} out-of-range angle values corrected")
|
||||
|
||||
# Save current data for next run
|
||||
try:
|
||||
with open(ampolle_file, 'w', newline='') as f:
|
||||
writer = csv.writer(f)
|
||||
for j in range(len(timestamps)):
|
||||
row = [timestamps[j] - 730000] + list(angles[j, :])
|
||||
writer.writerow(row)
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not save angle data: {e}")
|
||||
|
||||
return angles
|
||||
|
||||
|
||||
def load_biaxial_link_data(
|
||||
conn: DatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
initial_date: str,
|
||||
initial_time: str,
|
||||
node_list: list
|
||||
) -> Dict[str, Any]:
|
||||
"""Load Biaxial Link raw data from database."""
|
||||
node_type = 'Biaxial Link'
|
||||
|
||||
first_node = node_list[0]
|
||||
|
||||
timestamp_query = """
|
||||
SELECT Date, Time
|
||||
FROM RawDataView
|
||||
WHERE UnitName = %s
|
||||
AND ToolNameID = %s
|
||||
AND NodeType = %s
|
||||
AND NodeNum = %s
|
||||
AND (
|
||||
(Date = %s AND Time >= %s) OR
|
||||
(Date > %s)
|
||||
)
|
||||
ORDER BY Date, Time
|
||||
"""
|
||||
|
||||
timestamp_results = conn.execute_query(
|
||||
timestamp_query,
|
||||
(control_unit_id, chain, node_type, str(first_node),
|
||||
initial_date, initial_time, initial_date)
|
||||
)
|
||||
|
||||
if not timestamp_results:
|
||||
return {'timestamps': [], 'values': [], 'errors': []}
|
||||
|
||||
timestamps = []
|
||||
for row in timestamp_results:
|
||||
dt_str = f"{row['Date']} {row['Time']}"
|
||||
timestamps.append(dt_str)
|
||||
|
||||
n_timestamps = len(timestamps)
|
||||
|
||||
# BL: Val0, Val1 = biaxial data; Val2 = temperature
|
||||
n_values_per_node = 3
|
||||
all_values = np.zeros((n_timestamps, len(node_list) * n_values_per_node))
|
||||
|
||||
for i, node_num in enumerate(node_list):
|
||||
data_query = """
|
||||
SELECT Val0, Val1, Val2
|
||||
FROM RawDataView
|
||||
WHERE UnitName = %s
|
||||
AND ToolNameID = %s
|
||||
AND NodeType = %s
|
||||
AND NodeNum = %s
|
||||
AND (
|
||||
(Date = %s AND Time >= %s) OR
|
||||
(Date > %s)
|
||||
)
|
||||
ORDER BY Date, Time
|
||||
"""
|
||||
|
||||
node_results = conn.execute_query(
|
||||
data_query,
|
||||
(control_unit_id, chain, node_type, str(node_num),
|
||||
initial_date, initial_time, initial_date)
|
||||
)
|
||||
|
||||
col_offset = i * n_values_per_node
|
||||
for j, row in enumerate(node_results):
|
||||
if j >= n_timestamps:
|
||||
break
|
||||
all_values[j, col_offset] = float(row['Val0'] or 0)
|
||||
all_values[j, col_offset + 1] = float(row['Val1'] or 0)
|
||||
all_values[j, col_offset + 2] = float(row['Val2'] or 0)
|
||||
|
||||
return {
|
||||
'timestamps': timestamps,
|
||||
'values': all_values,
|
||||
'errors': [],
|
||||
'n_nodes': len(node_list)
|
||||
}
|
||||
|
||||
|
||||
def define_biaxial_link_data(
|
||||
raw_data: Dict[str, Any],
|
||||
n_sensors: int,
|
||||
n_despike: int
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Define and structure Biaxial Link data.
|
||||
|
||||
Args:
|
||||
raw_data: Raw data dict
|
||||
n_sensors: Number of sensors
|
||||
n_despike: Number of points for despiking
|
||||
|
||||
Returns:
|
||||
Tuple of (timestamps, data, temperature, errors)
|
||||
"""
|
||||
if not raw_data or not raw_data.get('values'):
|
||||
return np.array([]), np.array([]), np.array([]), np.array([])
|
||||
|
||||
timestamps_str = raw_data['timestamps']
|
||||
values = raw_data['values']
|
||||
|
||||
n_timestamps = len(timestamps_str)
|
||||
|
||||
timestamps = np.array([
|
||||
datetime.strptime(ts, "%Y-%m-%d %H:%M:%S").timestamp()
|
||||
for ts in timestamps_str
|
||||
])
|
||||
|
||||
# Extract biaxial data and temperature
|
||||
data = np.zeros((n_timestamps, n_sensors * 2))
|
||||
temperature = np.zeros((n_timestamps, n_sensors))
|
||||
|
||||
for i in range(n_sensors):
|
||||
col_offset = i * 3
|
||||
data[:, i * 2] = values[:, col_offset]
|
||||
data[:, i * 2 + 1] = values[:, col_offset + 1]
|
||||
temperature[:, i] = values[:, col_offset + 2]
|
||||
|
||||
# Despiking
|
||||
if n_despike <= n_timestamps:
|
||||
for i in range(n_sensors):
|
||||
kernel = n_despike if n_despike % 2 == 1 else n_despike + 1
|
||||
data[:, i * 2] = medfilt(data[:, i * 2], kernel_size=kernel)
|
||||
data[:, i * 2 + 1] = medfilt(data[:, i * 2 + 1], kernel_size=kernel)
|
||||
|
||||
errors = np.zeros((n_timestamps, n_sensors * 2))
|
||||
|
||||
return timestamps, data, temperature, errors
|
||||
371
src/tilt/db_write.py
Normal file
371
src/tilt/db_write.py
Normal file
@@ -0,0 +1,371 @@
|
||||
"""
|
||||
Database writing functions for Tilt processed data.
|
||||
|
||||
Writes elaborated tilt sensor data back to database.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import logging
|
||||
from typing import Optional
|
||||
from ..common.database import DatabaseConnection
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def write_tilt_link_hr_data(
|
||||
conn: DatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
x_global: np.ndarray,
|
||||
y_global: np.ndarray,
|
||||
z_global: np.ndarray,
|
||||
x_local: np.ndarray,
|
||||
y_local: np.ndarray,
|
||||
z_local: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
errors: Optional[np.ndarray] = None
|
||||
) -> None:
|
||||
"""
|
||||
Write Tilt Link HR elaborated data to database.
|
||||
|
||||
Converts MATLAB DBwriteTLHR.m function.
|
||||
|
||||
Args:
|
||||
conn: Database connection
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
x_global: X displacement in global coordinates
|
||||
y_global: Y displacement in global coordinates
|
||||
z_global: Z displacement in global coordinates
|
||||
x_local: X displacement in local coordinates
|
||||
y_local: Y displacement in local coordinates
|
||||
z_local: Z displacement in local coordinates
|
||||
temperature: Temperature data
|
||||
timestamps: Timestamp array
|
||||
errors: Error flags (optional)
|
||||
"""
|
||||
logger.info("Writing Tilt Link HR data to database")
|
||||
|
||||
query = """
|
||||
INSERT INTO elaborated_tlhr_data
|
||||
(IDcentralina, DTcatena, timestamp, nodeID,
|
||||
X_global, Y_global, Z_global,
|
||||
X_local, Y_local, Z_local,
|
||||
temperature, error_flag)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
|
||||
ON DUPLICATE KEY UPDATE
|
||||
X_global = VALUES(X_global),
|
||||
Y_global = VALUES(Y_global),
|
||||
Z_global = VALUES(Z_global),
|
||||
X_local = VALUES(X_local),
|
||||
Y_local = VALUES(Y_local),
|
||||
Z_local = VALUES(Z_local),
|
||||
temperature = VALUES(temperature),
|
||||
error_flag = VALUES(error_flag)
|
||||
"""
|
||||
|
||||
n_timestamps, n_sensors = x_global.shape
|
||||
data_rows = []
|
||||
|
||||
for t in range(n_timestamps):
|
||||
for s in range(n_sensors):
|
||||
error_flag = 0
|
||||
if errors is not None and s < errors.shape[1]:
|
||||
error_flag = int(errors[s, t])
|
||||
|
||||
data_rows.append((
|
||||
control_unit_id,
|
||||
chain,
|
||||
timestamps[t],
|
||||
s + 1,
|
||||
float(x_global[t, s]),
|
||||
float(y_global[t, s]),
|
||||
float(z_global[t, s]),
|
||||
float(x_local[t, s]),
|
||||
float(y_local[t, s]),
|
||||
float(z_local[t, s]),
|
||||
float(temperature[t, s]),
|
||||
error_flag
|
||||
))
|
||||
|
||||
if data_rows:
|
||||
conn.execute_many(query, data_rows)
|
||||
logger.info(f"Wrote {len(data_rows)} Tilt Link HR records")
|
||||
|
||||
|
||||
def write_tilt_link_data(
|
||||
conn: DatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
x_disp: np.ndarray,
|
||||
y_disp: np.ndarray,
|
||||
z_disp: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
errors: Optional[np.ndarray] = None
|
||||
) -> None:
|
||||
"""
|
||||
Write Tilt Link elaborated data to database.
|
||||
|
||||
Converts MATLAB DBwriteTL.m function.
|
||||
"""
|
||||
logger.info("Writing Tilt Link data to database")
|
||||
|
||||
query = """
|
||||
INSERT INTO elaborated_tl_data
|
||||
(IDcentralina, DTcatena, timestamp, nodeID,
|
||||
X_displacement, Y_displacement, Z_displacement,
|
||||
temperature, error_flag)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
|
||||
ON DUPLICATE KEY UPDATE
|
||||
X_displacement = VALUES(X_displacement),
|
||||
Y_displacement = VALUES(Y_displacement),
|
||||
Z_displacement = VALUES(Z_displacement),
|
||||
temperature = VALUES(temperature),
|
||||
error_flag = VALUES(error_flag)
|
||||
"""
|
||||
|
||||
n_timestamps, n_sensors = x_disp.shape
|
||||
data_rows = []
|
||||
|
||||
for t in range(n_timestamps):
|
||||
for s in range(n_sensors):
|
||||
error_flag = 0
|
||||
if errors is not None:
|
||||
error_flag = int(errors[s, t]) if s < errors.shape[1] else 0
|
||||
|
||||
data_rows.append((
|
||||
control_unit_id,
|
||||
chain,
|
||||
timestamps[t],
|
||||
s + 1,
|
||||
float(x_disp[t, s]),
|
||||
float(y_disp[t, s]),
|
||||
float(z_disp[t, s]),
|
||||
float(temperature[t, s]),
|
||||
error_flag
|
||||
))
|
||||
|
||||
if data_rows:
|
||||
conn.execute_many(query, data_rows)
|
||||
logger.info(f"Wrote {len(data_rows)} Tilt Link records")
|
||||
|
||||
|
||||
def write_biaxial_link_data(
|
||||
conn: DatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
x_disp: np.ndarray,
|
||||
y_disp: np.ndarray,
|
||||
z_disp: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
errors: Optional[np.ndarray] = None
|
||||
) -> None:
|
||||
"""
|
||||
Write Biaxial Link elaborated data to database.
|
||||
|
||||
Converts MATLAB DBwriteBL.m function.
|
||||
"""
|
||||
logger.info("Writing Biaxial Link data to database")
|
||||
|
||||
query = """
|
||||
INSERT INTO elaborated_bl_data
|
||||
(IDcentralina, DTcatena, timestamp, nodeID,
|
||||
X_displacement, Y_displacement, Z_displacement,
|
||||
temperature, error_flag)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
|
||||
ON DUPLICATE KEY UPDATE
|
||||
X_displacement = VALUES(X_displacement),
|
||||
Y_displacement = VALUES(Y_displacement),
|
||||
Z_displacement = VALUES(Z_displacement),
|
||||
temperature = VALUES(temperature),
|
||||
error_flag = VALUES(error_flag)
|
||||
"""
|
||||
|
||||
n_timestamps, n_sensors = x_disp.shape
|
||||
data_rows = []
|
||||
|
||||
for t in range(n_timestamps):
|
||||
for s in range(n_sensors):
|
||||
error_flag = 0
|
||||
if errors is not None:
|
||||
error_flag = int(errors[s, t]) if s < errors.shape[1] else 0
|
||||
|
||||
data_rows.append((
|
||||
control_unit_id,
|
||||
chain,
|
||||
timestamps[t],
|
||||
s + 1,
|
||||
float(x_disp[t, s]),
|
||||
float(y_disp[t, s]),
|
||||
float(z_disp[t, s]),
|
||||
float(temperature[t, s]),
|
||||
error_flag
|
||||
))
|
||||
|
||||
if data_rows:
|
||||
conn.execute_many(query, data_rows)
|
||||
logger.info(f"Wrote {len(data_rows)} Biaxial Link records")
|
||||
|
||||
|
||||
def write_pendulum_link_data(
|
||||
conn: DatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
x_disp: np.ndarray,
|
||||
y_disp: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
errors: Optional[np.ndarray] = None
|
||||
) -> None:
|
||||
"""
|
||||
Write Pendulum Link elaborated data to database.
|
||||
|
||||
Converts MATLAB DBwritePL.m function.
|
||||
"""
|
||||
logger.info("Writing Pendulum Link data to database")
|
||||
|
||||
query = """
|
||||
INSERT INTO elaborated_pl_data
|
||||
(IDcentralina, DTcatena, timestamp, nodeID,
|
||||
X_displacement, Y_displacement,
|
||||
temperature, error_flag)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
|
||||
ON DUPLICATE KEY UPDATE
|
||||
X_displacement = VALUES(X_displacement),
|
||||
Y_displacement = VALUES(Y_displacement),
|
||||
temperature = VALUES(temperature),
|
||||
error_flag = VALUES(error_flag)
|
||||
"""
|
||||
|
||||
n_timestamps, n_sensors = x_disp.shape
|
||||
data_rows = []
|
||||
|
||||
for t in range(n_timestamps):
|
||||
for s in range(n_sensors):
|
||||
error_flag = 0
|
||||
if errors is not None:
|
||||
error_flag = int(errors[s, t]) if s < errors.shape[1] else 0
|
||||
|
||||
data_rows.append((
|
||||
control_unit_id,
|
||||
chain,
|
||||
timestamps[t],
|
||||
s + 1,
|
||||
float(x_disp[t, s]),
|
||||
float(y_disp[t, s]),
|
||||
float(temperature[t, s]),
|
||||
error_flag
|
||||
))
|
||||
|
||||
if data_rows:
|
||||
conn.execute_many(query, data_rows)
|
||||
logger.info(f"Wrote {len(data_rows)} Pendulum Link records")
|
||||
|
||||
|
||||
def write_kessler_link_data(
|
||||
conn: DatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
x_disp: np.ndarray,
|
||||
y_disp: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
errors: Optional[np.ndarray] = None
|
||||
) -> None:
|
||||
"""
|
||||
Write Kessler Link elaborated data to database.
|
||||
|
||||
Converts MATLAB DBwriteKLHR.m function.
|
||||
"""
|
||||
logger.info("Writing Kessler Link data to database")
|
||||
|
||||
query = """
|
||||
INSERT INTO elaborated_klhr_data
|
||||
(IDcentralina, DTcatena, timestamp, nodeID,
|
||||
X_displacement, Y_displacement,
|
||||
temperature, error_flag)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
|
||||
ON DUPLICATE KEY UPDATE
|
||||
X_displacement = VALUES(X_displacement),
|
||||
Y_displacement = VALUES(Y_displacement),
|
||||
temperature = VALUES(temperature),
|
||||
error_flag = VALUES(error_flag)
|
||||
"""
|
||||
|
||||
n_timestamps, n_sensors = x_disp.shape
|
||||
data_rows = []
|
||||
|
||||
for t in range(n_timestamps):
|
||||
for s in range(n_sensors):
|
||||
error_flag = 0
|
||||
if errors is not None:
|
||||
error_flag = int(errors[s, t]) if s < errors.shape[1] else 0
|
||||
|
||||
data_rows.append((
|
||||
control_unit_id,
|
||||
chain,
|
||||
timestamps[t],
|
||||
s + 1,
|
||||
float(x_disp[t, s]),
|
||||
float(y_disp[t, s]),
|
||||
float(temperature[t, s]),
|
||||
error_flag
|
||||
))
|
||||
|
||||
if data_rows:
|
||||
conn.execute_many(query, data_rows)
|
||||
logger.info(f"Wrote {len(data_rows)} Kessler Link records")
|
||||
|
||||
|
||||
def write_temperature_data(
|
||||
conn: DatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
temperature: np.ndarray,
|
||||
timestamps: np.ndarray,
|
||||
sensor_type: str = "ThL"
|
||||
) -> None:
|
||||
"""
|
||||
Write temperature sensor data to database.
|
||||
|
||||
For thermistors (ThL) or PT100 sensors.
|
||||
|
||||
Args:
|
||||
conn: Database connection
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
temperature: Temperature data
|
||||
timestamps: Timestamp array
|
||||
sensor_type: Sensor type ("ThL" or "PT100")
|
||||
"""
|
||||
logger.info(f"Writing {sensor_type} temperature data to database")
|
||||
|
||||
table_name = f"elaborated_{sensor_type.lower()}_data"
|
||||
|
||||
query = f"""
|
||||
INSERT INTO {table_name}
|
||||
(IDcentralina, DTcatena, timestamp, nodeID, temperature)
|
||||
VALUES (%s, %s, %s, %s, %s)
|
||||
ON DUPLICATE KEY UPDATE
|
||||
temperature = VALUES(temperature)
|
||||
"""
|
||||
|
||||
n_timestamps, n_sensors = temperature.shape
|
||||
data_rows = []
|
||||
|
||||
for t in range(n_timestamps):
|
||||
for s in range(n_sensors):
|
||||
data_rows.append((
|
||||
control_unit_id,
|
||||
chain,
|
||||
timestamps[t],
|
||||
s + 1,
|
||||
float(temperature[t, s])
|
||||
))
|
||||
|
||||
if data_rows:
|
||||
conn.execute_many(query, data_rows)
|
||||
logger.info(f"Wrote {len(data_rows)} {sensor_type} temperature records")
|
||||
361
src/tilt/elaboration.py
Normal file
361
src/tilt/elaboration.py
Normal file
@@ -0,0 +1,361 @@
|
||||
"""
|
||||
Data elaboration functions for Tilt sensors.
|
||||
|
||||
Processes tilt sensor data to calculate displacements and rotations.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import logging
|
||||
from typing import Tuple, Optional
|
||||
from pathlib import Path
|
||||
from ..common.database import DatabaseConnection
|
||||
from ..common.validators import approximate_values
|
||||
from .geometry import arot_hr, asse_a_hr, asse_b_hr
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def elaborate_tilt_link_hr_data(
|
||||
conn: DatabaseConnection,
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
n_sensors: int,
|
||||
angle_data: np.ndarray,
|
||||
temp_max: float,
|
||||
temp_min: float,
|
||||
temperature: np.ndarray,
|
||||
node_list: list,
|
||||
timestamps: np.ndarray,
|
||||
is_new_zero: bool,
|
||||
n_data_avg: int,
|
||||
n_data_despike: int,
|
||||
error_flags: np.ndarray,
|
||||
initial_date: str,
|
||||
installation_angles: np.ndarray,
|
||||
sensor_lengths: np.ndarray
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Elaborate Tilt Link HR data to calculate displacements.
|
||||
|
||||
Converts MATLAB elaboration for TLHR sensors.
|
||||
|
||||
Args:
|
||||
conn: Database connection
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
n_sensors: Number of sensors
|
||||
angle_data: Angle data array (degrees)
|
||||
temp_max: Maximum valid temperature
|
||||
temp_min: Minimum valid temperature
|
||||
temperature: Temperature array
|
||||
node_list: List of node IDs
|
||||
timestamps: Timestamp array
|
||||
is_new_zero: Whether this is a new zero point
|
||||
n_data_avg: Number of data for averaging
|
||||
n_data_despike: Number of data for despiking
|
||||
error_flags: Error flags array
|
||||
initial_date: Initial processing date
|
||||
installation_angles: Installation angle for each sensor (degrees)
|
||||
sensor_lengths: Length/spacing for each sensor (meters)
|
||||
|
||||
Returns:
|
||||
Tuple of (X_global, Y_global, Z_global, X_local, Y_local, Z_local, temperature)
|
||||
"""
|
||||
logger.info("Starting Tilt Link HR elaboration")
|
||||
|
||||
# Handle new zero point
|
||||
if is_new_zero:
|
||||
n_skip = max(n_data_avg, n_data_despike)
|
||||
ini = round(n_skip / 2) + 1
|
||||
if n_skip % 2 == 0:
|
||||
ini += 1
|
||||
|
||||
angle_data = angle_data[ini:, :]
|
||||
temperature = temperature[ini:, :]
|
||||
timestamps = timestamps[ini:]
|
||||
error_flags = error_flags[ini:, :]
|
||||
|
||||
n_timestamps = len(timestamps)
|
||||
|
||||
# Temperature validation
|
||||
n_corrections_temp = 0
|
||||
for b in range(temperature.shape[1]):
|
||||
for a in range(temperature.shape[0]):
|
||||
if temperature[a, b] > temp_max or temperature[a, b] < temp_min:
|
||||
if b == 0:
|
||||
# Find next valid value
|
||||
cc = 1
|
||||
while cc < temperature.shape[1]:
|
||||
if temp_min <= temperature[a, cc] <= temp_max:
|
||||
temperature[a, b] = temperature[a, cc]
|
||||
break
|
||||
cc += 1
|
||||
else:
|
||||
temperature[a, b] = temperature[a, b-1]
|
||||
n_corrections_temp += 1
|
||||
|
||||
if n_corrections_temp > 0:
|
||||
logger.info(f"{n_corrections_temp} temperature corrections applied")
|
||||
|
||||
# Calculate displacements for each sensor
|
||||
# Global coordinates (absolute)
|
||||
X_global = np.zeros((n_timestamps, n_sensors))
|
||||
Y_global = np.zeros((n_timestamps, n_sensors))
|
||||
Z_global = np.zeros((n_timestamps, n_sensors))
|
||||
|
||||
# Local coordinates (relative to installation)
|
||||
X_local = np.zeros((n_timestamps, n_sensors))
|
||||
Y_local = np.zeros((n_timestamps, n_sensors))
|
||||
Z_local = np.zeros((n_timestamps, n_sensors))
|
||||
|
||||
# Extract angle arrays (reshape for geometry functions)
|
||||
ax = np.zeros((n_sensors, n_timestamps))
|
||||
ay = np.zeros((n_sensors, n_timestamps))
|
||||
|
||||
for i in range(n_sensors):
|
||||
ax[i, :] = angle_data[:, i * 2]
|
||||
ay[i, :] = angle_data[:, i * 2 + 1]
|
||||
|
||||
# Calculate displacements using geometric transformations
|
||||
for t in range(n_timestamps):
|
||||
for i in range(n_sensors):
|
||||
# Installation angle for this sensor
|
||||
install_angle = installation_angles[i] if i < len(installation_angles) else 0.0
|
||||
|
||||
# Sensor length/spacing
|
||||
spe_tl = sensor_lengths[i] if i < len(sensor_lengths) else 1.0
|
||||
|
||||
# Calculate displacement components
|
||||
n_disp, e_disp, z_disp = arot_hr(
|
||||
ax, ay, install_angle,
|
||||
np.array([spe_tl]), # Wrap in array for compatibility
|
||||
i, t
|
||||
)
|
||||
|
||||
# Store in global coordinates
|
||||
X_global[t, i] = n_disp
|
||||
Y_global[t, i] = e_disp
|
||||
Z_global[t, i] = z_disp
|
||||
|
||||
# Local coordinates (simplified - could add rotation matrix)
|
||||
X_local[t, i] = n_disp
|
||||
Y_local[t, i] = e_disp
|
||||
Z_local[t, i] = z_disp
|
||||
|
||||
# Calculate horizontal shift
|
||||
H_shift_global = np.sqrt(X_global**2 + Y_global**2)
|
||||
H_shift_local = np.sqrt(X_local**2 + Y_local**2)
|
||||
|
||||
# Calculate azimuth (direction of movement)
|
||||
Azimuth = np.degrees(np.arctan2(Y_global, X_global))
|
||||
|
||||
# Apply approximation (round to specified decimal places)
|
||||
X_global, Y_global, Z_global, X_local, Y_local, Z_local, temperature = \
|
||||
approximate_values(X_global, Y_global, Z_global, X_local, Y_local, Z_local, temperature, decimals=6)
|
||||
|
||||
# Calculate differentials (relative to first reading or reference)
|
||||
X_global, Y_global, Z_global = calculate_tilt_differentials(
|
||||
control_unit_id, chain, X_global, Y_global, Z_global, is_new_zero, "TLHR"
|
||||
)
|
||||
|
||||
logger.info("Tilt Link HR elaboration completed successfully")
|
||||
|
||||
return X_global, Y_global, Z_global, X_local, Y_local, Z_local, temperature
|
||||
|
||||
|
||||
def calculate_tilt_differentials(
|
||||
control_unit_id: str,
|
||||
chain: str,
|
||||
x_data: np.ndarray,
|
||||
y_data: np.ndarray,
|
||||
z_data: np.ndarray,
|
||||
is_new_zero: bool,
|
||||
sensor_type: str
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Calculate differential values relative to reference.
|
||||
|
||||
Args:
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
x_data: X displacement data
|
||||
y_data: Y displacement data
|
||||
z_data: Z displacement data
|
||||
is_new_zero: Whether this is first processing
|
||||
sensor_type: Sensor type identifier
|
||||
|
||||
Returns:
|
||||
Tuple of differential x, y, z
|
||||
"""
|
||||
ref_file_x = Path(f"{control_unit_id}-{chain}-{sensor_type}-RifX.csv")
|
||||
ref_file_y = Path(f"{control_unit_id}-{chain}-{sensor_type}-RifY.csv")
|
||||
ref_file_z = Path(f"{control_unit_id}-{chain}-{sensor_type}-RifZ.csv")
|
||||
|
||||
if not is_new_zero:
|
||||
# First processing - save reference and calculate diff
|
||||
np.savetxt(ref_file_x, x_data[0:1, :], delimiter=',')
|
||||
np.savetxt(ref_file_y, y_data[0:1, :], delimiter=',')
|
||||
np.savetxt(ref_file_z, z_data[0:1, :], delimiter=',')
|
||||
|
||||
x_diff = x_data - x_data[0, :]
|
||||
y_diff = y_data - y_data[0, :]
|
||||
z_diff = z_data - z_data[0, :]
|
||||
else:
|
||||
# Load reference and calculate diff
|
||||
try:
|
||||
ref_x = np.loadtxt(ref_file_x, delimiter=',')
|
||||
ref_y = np.loadtxt(ref_file_y, delimiter=',')
|
||||
ref_z = np.loadtxt(ref_file_z, delimiter=',')
|
||||
|
||||
x_diff = x_data - ref_x
|
||||
y_diff = y_data - ref_y
|
||||
z_diff = z_data - ref_z
|
||||
except FileNotFoundError:
|
||||
logger.warning("Reference files not found, using first value as reference")
|
||||
x_diff = x_data - x_data[0, :]
|
||||
y_diff = y_data - y_data[0, :]
|
||||
z_diff = z_data - z_data[0, :]
|
||||
|
||||
return x_diff, y_diff, z_diff
|
||||
|
||||
|
||||
def elaborate_biaxial_link_data(
|
||||
data: np.ndarray,
|
||||
temperature: np.ndarray,
|
||||
n_sensors: int,
|
||||
installation_angles: np.ndarray,
|
||||
sensor_lengths: np.ndarray,
|
||||
temp_max: float,
|
||||
temp_min: float
|
||||
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Elaborate Biaxial Link data.
|
||||
|
||||
Args:
|
||||
data: Sensor data array (acceleration or angles)
|
||||
temperature: Temperature array
|
||||
n_sensors: Number of sensors
|
||||
installation_angles: Installation angles
|
||||
sensor_lengths: Sensor lengths
|
||||
temp_max: Maximum valid temperature
|
||||
temp_min: Minimum valid temperature
|
||||
|
||||
Returns:
|
||||
Tuple of (X_disp, Y_disp, Z_disp, temperature)
|
||||
"""
|
||||
logger.info(f"Elaborating Biaxial Link data for {n_sensors} sensors")
|
||||
|
||||
n_timestamps = data.shape[0]
|
||||
|
||||
# Validate temperature
|
||||
for i in range(temperature.shape[1]):
|
||||
invalid_mask = (temperature[:, i] < temp_min) | (temperature[:, i] > temp_max)
|
||||
if np.any(invalid_mask):
|
||||
# Forward fill valid values
|
||||
valid_indices = np.where(~invalid_mask)[0]
|
||||
if len(valid_indices) > 0:
|
||||
temperature[invalid_mask, i] = np.interp(
|
||||
np.where(invalid_mask)[0],
|
||||
valid_indices,
|
||||
temperature[valid_indices, i]
|
||||
)
|
||||
|
||||
# Calculate displacements
|
||||
X_disp = np.zeros((n_timestamps, n_sensors))
|
||||
Y_disp = np.zeros((n_timestamps, n_sensors))
|
||||
Z_disp = np.zeros((n_timestamps, n_sensors))
|
||||
|
||||
for i in range(n_sensors):
|
||||
# Extract axes for this sensor
|
||||
ax = data[:, i * 2]
|
||||
ay = data[:, i * 2 + 1]
|
||||
|
||||
angle = installation_angles[i] if i < len(installation_angles) else 0.0
|
||||
length = sensor_lengths[i] if i < len(sensor_lengths) else 1.0
|
||||
|
||||
# Calculate displacement for each timestamp
|
||||
for t in range(n_timestamps):
|
||||
# Use geometry functions
|
||||
n_a, e_a, z_a = asse_a_hr(
|
||||
np.array([[ax[t]]]), angle,
|
||||
np.array([length]), 0, 0
|
||||
)
|
||||
n_b, e_b, z_b = asse_b_hr(
|
||||
np.array([[ay[t]]]), angle,
|
||||
np.array([length]), 0, 0
|
||||
)
|
||||
|
||||
X_disp[t, i] = n_a + n_b
|
||||
Y_disp[t, i] = e_a + e_b
|
||||
Z_disp[t, i] = z_a + z_b
|
||||
|
||||
logger.info("Biaxial Link elaboration completed")
|
||||
return X_disp, Y_disp, Z_disp, temperature
|
||||
|
||||
|
||||
def calculate_velocity_acceleration(
|
||||
displacement: np.ndarray,
|
||||
timestamps: np.ndarray
|
||||
) -> Tuple[np.ndarray, np.ndarray]:
|
||||
"""
|
||||
Calculate velocity and acceleration from displacement data.
|
||||
|
||||
Args:
|
||||
displacement: Displacement array (timestamps x sensors)
|
||||
timestamps: Timestamp array
|
||||
|
||||
Returns:
|
||||
Tuple of (velocity, acceleration)
|
||||
"""
|
||||
n_timestamps, n_sensors = displacement.shape
|
||||
|
||||
# Calculate time differences (convert to seconds if needed)
|
||||
dt = np.diff(timestamps)
|
||||
dt = np.concatenate([[dt[0]], dt]) # Prepend first dt
|
||||
|
||||
# Velocity: dDisplacement/dt
|
||||
velocity = np.zeros_like(displacement)
|
||||
velocity[1:, :] = np.diff(displacement, axis=0) / dt[1:, np.newaxis]
|
||||
velocity[0, :] = velocity[1, :] # Forward fill first value
|
||||
|
||||
# Acceleration: dVelocity/dt
|
||||
acceleration = np.zeros_like(displacement)
|
||||
acceleration[1:, :] = np.diff(velocity, axis=0) / dt[1:, np.newaxis]
|
||||
acceleration[0, :] = acceleration[1, :]
|
||||
|
||||
return velocity, acceleration
|
||||
|
||||
|
||||
def approximate_tilt_values(
|
||||
*arrays: np.ndarray,
|
||||
decimals_pos: int = 6,
|
||||
decimals_angle: int = 1,
|
||||
decimals_temp: int = 1
|
||||
) -> Tuple[np.ndarray, ...]:
|
||||
"""
|
||||
Approximate tilt values to specified decimal places.
|
||||
|
||||
Converts MATLAB approx_TLHR.m function.
|
||||
|
||||
Args:
|
||||
arrays: Variable number of arrays to approximate
|
||||
decimals_pos: Decimal places for positions (micrometers precision)
|
||||
decimals_angle: Decimal places for angles
|
||||
decimals_temp: Decimal places for temperature
|
||||
|
||||
Returns:
|
||||
Tuple of approximated arrays
|
||||
"""
|
||||
# First arrays are typically positions (X, Y, Z) - use high precision
|
||||
# Last array is typically temperature - use lower precision
|
||||
result = []
|
||||
|
||||
for i, arr in enumerate(arrays):
|
||||
if i < len(arrays) - 1:
|
||||
# Position data
|
||||
result.append(np.round(arr, decimals_pos))
|
||||
else:
|
||||
# Temperature data
|
||||
result.append(np.round(arr, decimals_temp))
|
||||
|
||||
return tuple(result)
|
||||
324
src/tilt/geometry.py
Normal file
324
src/tilt/geometry.py
Normal file
@@ -0,0 +1,324 @@
|
||||
"""
|
||||
Geometric calculation functions for tilt sensors.
|
||||
|
||||
Includes axis transformations, rotations, and quaternion operations.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import logging
|
||||
from typing import Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def asse_a(
|
||||
ax: np.ndarray,
|
||||
angle: float,
|
||||
spe_tl: np.ndarray,
|
||||
i: int,
|
||||
j: int
|
||||
) -> Tuple[float, float, float]:
|
||||
"""
|
||||
Calculate axis A displacement components.
|
||||
|
||||
Converts MATLAB ASSEa.m function.
|
||||
|
||||
Args:
|
||||
ax: Acceleration/inclination data for axis X
|
||||
angle: Installation angle in degrees
|
||||
spe_tl: Sensor spacing/length array
|
||||
i: Sensor index
|
||||
j: Time index
|
||||
|
||||
Returns:
|
||||
Tuple of (North component, East component, Vertical component)
|
||||
"""
|
||||
# Convert angle to radians
|
||||
angle_rad = angle * 2 * np.pi / 360
|
||||
|
||||
if ax[i, j] >= 0:
|
||||
na = spe_tl[i] * ax[i, j] * np.cos(angle_rad)
|
||||
ea = -spe_tl[i] * ax[i, j] * np.sin(angle_rad)
|
||||
else:
|
||||
na = -spe_tl[i] * ax[i, j] * np.cos(angle_rad)
|
||||
ea = spe_tl[i] * ax[i, j] * np.sin(angle_rad)
|
||||
|
||||
# Calculate cosine of inclination angle
|
||||
cos_beta = np.sqrt(1 - ax[i, j]**2)
|
||||
z = spe_tl[i] * cos_beta
|
||||
za = spe_tl[i] - z # Lowering is POSITIVE
|
||||
|
||||
return na, ea, za
|
||||
|
||||
|
||||
def asse_a_hr(
|
||||
ax: np.ndarray,
|
||||
angle: float,
|
||||
spe_tl: np.ndarray,
|
||||
i: int,
|
||||
j: int
|
||||
) -> Tuple[float, float, float]:
|
||||
"""
|
||||
Calculate axis A displacement components for high-resolution sensors.
|
||||
|
||||
Converts MATLAB ASSEa_HR.m function.
|
||||
|
||||
Args:
|
||||
ax: Angle data for axis X (in degrees)
|
||||
angle: Installation angle in degrees
|
||||
spe_tl: Sensor spacing/length array
|
||||
i: Sensor index
|
||||
j: Time index
|
||||
|
||||
Returns:
|
||||
Tuple of (North component, East component, Vertical component)
|
||||
"""
|
||||
# Convert angles to radians
|
||||
angle_rad = angle * np.pi / 180
|
||||
ax_rad = ax[i, j] * np.pi / 180
|
||||
|
||||
# Calculate displacement components
|
||||
na = spe_tl[i] * np.sin(ax_rad) * np.cos(angle_rad)
|
||||
ea = -spe_tl[i] * np.sin(ax_rad) * np.sin(angle_rad)
|
||||
|
||||
# Vertical component
|
||||
za = spe_tl[i] * (1 - np.cos(ax_rad))
|
||||
|
||||
return na, ea, za
|
||||
|
||||
|
||||
def asse_b(
|
||||
ay: np.ndarray,
|
||||
angle: float,
|
||||
spe_tl: np.ndarray,
|
||||
i: int,
|
||||
j: int
|
||||
) -> Tuple[float, float, float]:
|
||||
"""
|
||||
Calculate axis B displacement components.
|
||||
|
||||
Converts MATLAB ASSEb.m function.
|
||||
|
||||
Args:
|
||||
ay: Acceleration/inclination data for axis Y
|
||||
angle: Installation angle in degrees
|
||||
spe_tl: Sensor spacing/length array
|
||||
i: Sensor index
|
||||
j: Time index
|
||||
|
||||
Returns:
|
||||
Tuple of (North component, East component, Vertical component)
|
||||
"""
|
||||
# Convert angle to radians
|
||||
angle_rad = angle * 2 * np.pi / 360
|
||||
|
||||
if ay[i, j] >= 0:
|
||||
nb = -spe_tl[i] * ay[i, j] * np.sin(angle_rad)
|
||||
eb = -spe_tl[i] * ay[i, j] * np.cos(angle_rad)
|
||||
else:
|
||||
nb = spe_tl[i] * ay[i, j] * np.sin(angle_rad)
|
||||
eb = spe_tl[i] * ay[i, j] * np.cos(angle_rad)
|
||||
|
||||
# Calculate cosine of inclination angle
|
||||
cos_beta = np.sqrt(1 - ay[i, j]**2)
|
||||
z = spe_tl[i] * cos_beta
|
||||
zb = spe_tl[i] - z # Lowering is POSITIVE
|
||||
|
||||
return nb, eb, zb
|
||||
|
||||
|
||||
def asse_b_hr(
|
||||
ay: np.ndarray,
|
||||
angle: float,
|
||||
spe_tl: np.ndarray,
|
||||
i: int,
|
||||
j: int
|
||||
) -> Tuple[float, float, float]:
|
||||
"""
|
||||
Calculate axis B displacement components for high-resolution sensors.
|
||||
|
||||
Converts MATLAB ASSEb_HR.m function.
|
||||
|
||||
Args:
|
||||
ay: Angle data for axis Y (in degrees)
|
||||
angle: Installation angle in degrees
|
||||
spe_tl: Sensor spacing/length array
|
||||
i: Sensor index
|
||||
j: Time index
|
||||
|
||||
Returns:
|
||||
Tuple of (North component, East component, Vertical component)
|
||||
"""
|
||||
# Convert angles to radians
|
||||
angle_rad = angle * np.pi / 180
|
||||
ay_rad = ay[i, j] * np.pi / 180
|
||||
|
||||
# Calculate displacement components
|
||||
nb = -spe_tl[i] * np.sin(ay_rad) * np.sin(angle_rad)
|
||||
eb = -spe_tl[i] * np.sin(ay_rad) * np.cos(angle_rad)
|
||||
|
||||
# Vertical component
|
||||
zb = spe_tl[i] * (1 - np.cos(ay_rad))
|
||||
|
||||
return nb, eb, zb
|
||||
|
||||
|
||||
def arot(
|
||||
ax: np.ndarray,
|
||||
ay: np.ndarray,
|
||||
angle: float,
|
||||
spe_tl: np.ndarray,
|
||||
i: int,
|
||||
j: int
|
||||
) -> Tuple[float, float, float]:
|
||||
"""
|
||||
Calculate combined rotation displacement.
|
||||
|
||||
Converts MATLAB arot.m function.
|
||||
|
||||
Args:
|
||||
ax: Acceleration/inclination data for axis X
|
||||
ay: Acceleration/inclination data for axis Y
|
||||
angle: Installation angle in degrees
|
||||
spe_tl: Sensor spacing/length array
|
||||
i: Sensor index
|
||||
j: Time index
|
||||
|
||||
Returns:
|
||||
Tuple of (North displacement, East displacement, Vertical displacement)
|
||||
"""
|
||||
# Calculate components from both axes
|
||||
na, ea, za = asse_a(ax, angle, spe_tl, i, j)
|
||||
nb, eb, zb = asse_b(ay, angle, spe_tl, i, j)
|
||||
|
||||
# Combine components
|
||||
n_total = na + nb
|
||||
e_total = ea + eb
|
||||
z_total = za + zb
|
||||
|
||||
return n_total, e_total, z_total
|
||||
|
||||
|
||||
def arot_hr(
|
||||
ax: np.ndarray,
|
||||
ay: np.ndarray,
|
||||
angle: float,
|
||||
spe_tl: np.ndarray,
|
||||
i: int,
|
||||
j: int
|
||||
) -> Tuple[float, float, float]:
|
||||
"""
|
||||
Calculate combined rotation displacement for high-resolution sensors.
|
||||
|
||||
Converts MATLAB arotHR.m function.
|
||||
|
||||
Args:
|
||||
ax: Angle data for axis X (in degrees)
|
||||
ay: Angle data for axis Y (in degrees)
|
||||
angle: Installation angle in degrees
|
||||
spe_tl: Sensor spacing/length array
|
||||
i: Sensor index
|
||||
j: Time index
|
||||
|
||||
Returns:
|
||||
Tuple of (North displacement, East displacement, Vertical displacement)
|
||||
"""
|
||||
# Calculate components from both axes
|
||||
na, ea, za = asse_a_hr(ax, angle, spe_tl, i, j)
|
||||
nb, eb, zb = asse_b_hr(ay, angle, spe_tl, i, j)
|
||||
|
||||
# Combine components
|
||||
n_total = na + nb
|
||||
e_total = ea + eb
|
||||
z_total = za + zb
|
||||
|
||||
return n_total, e_total, z_total
|
||||
|
||||
|
||||
# Quaternion operations
|
||||
def q_mult2(q1: np.ndarray, q2: np.ndarray) -> np.ndarray:
|
||||
"""
|
||||
Multiply two quaternions.
|
||||
|
||||
Converts MATLAB q_mult2.m function.
|
||||
|
||||
Args:
|
||||
q1: First quaternion [w, x, y, z]
|
||||
q2: Second quaternion [w, x, y, z]
|
||||
|
||||
Returns:
|
||||
Product quaternion
|
||||
"""
|
||||
w1, x1, y1, z1 = q1
|
||||
w2, x2, y2, z2 = q2
|
||||
|
||||
w = w1*w2 - x1*x2 - y1*y2 - z1*z2
|
||||
x = w1*x2 + x1*w2 + y1*z2 - z1*y2
|
||||
y = w1*y2 - x1*z2 + y1*w2 + z1*x2
|
||||
z = w1*z2 + x1*y2 - y1*x2 + z1*w2
|
||||
|
||||
return np.array([w, x, y, z])
|
||||
|
||||
|
||||
def rotate_v_by_q(v: np.ndarray, q: np.ndarray) -> np.ndarray:
|
||||
"""
|
||||
Rotate a vector by a quaternion.
|
||||
|
||||
Converts MATLAB rotate_v_by_q.m function.
|
||||
|
||||
Args:
|
||||
v: Vector to rotate [x, y, z]
|
||||
q: Quaternion [w, x, y, z]
|
||||
|
||||
Returns:
|
||||
Rotated vector
|
||||
"""
|
||||
# Convert vector to quaternion form [0, x, y, z]
|
||||
v_quat = np.array([0, v[0], v[1], v[2]])
|
||||
|
||||
# Calculate q * v * q_conjugate
|
||||
q_conj = np.array([q[0], -q[1], -q[2], -q[3]])
|
||||
|
||||
temp = q_mult2(q, v_quat)
|
||||
result = q_mult2(temp, q_conj)
|
||||
|
||||
# Return vector part
|
||||
return result[1:]
|
||||
|
||||
|
||||
def fqa(ax: float, ay: float) -> np.ndarray:
|
||||
"""
|
||||
Calculate quaternion from acceleration angles.
|
||||
|
||||
Converts MATLAB fqa.m function.
|
||||
|
||||
Args:
|
||||
ax: Acceleration angle X
|
||||
ay: Acceleration angle Y
|
||||
|
||||
Returns:
|
||||
Quaternion representation
|
||||
"""
|
||||
# Calculate rotation angles
|
||||
theta_x = np.arcsin(ax)
|
||||
theta_y = np.arcsin(ay)
|
||||
|
||||
# Build quaternion
|
||||
qx = np.array([
|
||||
np.cos(theta_x/2),
|
||||
np.sin(theta_x/2),
|
||||
0,
|
||||
0
|
||||
])
|
||||
|
||||
qy = np.array([
|
||||
np.cos(theta_y/2),
|
||||
0,
|
||||
np.sin(theta_y/2),
|
||||
0
|
||||
])
|
||||
|
||||
# Combine rotations
|
||||
q = q_mult2(qx, qy)
|
||||
|
||||
return q
|
||||
121
src/tilt/main.py
Normal file
121
src/tilt/main.py
Normal file
@@ -0,0 +1,121 @@
|
||||
"""
|
||||
Main Tilt sensor data processing module.
|
||||
|
||||
Entry point for tiltmeter sensor data elaboration.
|
||||
Similar structure to RSN module but for tilt/inclinometer sensors.
|
||||
"""
|
||||
|
||||
import time
|
||||
import logging
|
||||
from typing import Tuple
|
||||
from ..common.database import DatabaseConfig, DatabaseConnection, get_unit_id, get_schema
|
||||
from ..common.logging_utils import setup_logger, log_elapsed_time
|
||||
from ..common.config import load_installation_parameters, load_calibration_data
|
||||
|
||||
|
||||
def process_tilt_chain(control_unit_id: str, chain: str) -> int:
|
||||
"""
|
||||
Main function to process Tilt chain data.
|
||||
|
||||
Args:
|
||||
control_unit_id: Control unit identifier
|
||||
chain: Chain identifier
|
||||
|
||||
Returns:
|
||||
0 if successful, 1 if error
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
# Setup logger
|
||||
logger = setup_logger(control_unit_id, chain, "Tilt")
|
||||
|
||||
try:
|
||||
# Load database configuration
|
||||
db_config = DatabaseConfig()
|
||||
|
||||
# Connect to database
|
||||
with DatabaseConnection(db_config) as conn:
|
||||
logger.info("Database connection established")
|
||||
|
||||
# Get unit ID
|
||||
unit_id = get_unit_id(control_unit_id, conn)
|
||||
|
||||
# Load node configuration
|
||||
logger.info("Loading tilt sensor configuration")
|
||||
|
||||
# Query for tilt sensor types (TL, TLH, TLHR, BL, PL, etc.)
|
||||
query = """
|
||||
SELECT idTool, nodeID, nodeType, sensorModel
|
||||
FROM chain_nodes
|
||||
WHERE unitID = %s AND chain = %s
|
||||
AND nodeType IN ('TL', 'TLH', 'TLHR', 'TLHRH', 'BL', 'PL', 'RL', 'ThL', 'IPL', 'IPLHR', 'KL', 'KLHR', 'PT100')
|
||||
ORDER BY nodeOrder
|
||||
"""
|
||||
results = conn.execute_query(query, (unit_id, chain))
|
||||
|
||||
if not results:
|
||||
logger.warning("No tilt sensors found for this chain")
|
||||
return 0
|
||||
|
||||
id_tool = results[0]['idTool']
|
||||
|
||||
# Organize sensors by type
|
||||
tilt_sensors = {}
|
||||
for row in results:
|
||||
sensor_type = row['nodeType']
|
||||
if sensor_type not in tilt_sensors:
|
||||
tilt_sensors[sensor_type] = []
|
||||
tilt_sensors[sensor_type].append(row['nodeID'])
|
||||
|
||||
logger.info(f"Found tilt sensors: {', '.join([f'{k}:{len(v)}' for k, v in tilt_sensors.items()])}")
|
||||
|
||||
# Load installation parameters
|
||||
params = load_installation_parameters(id_tool, conn)
|
||||
|
||||
# Process each sensor type
|
||||
# TL - Tilt Link (basic biaxial inclinometer)
|
||||
if 'TL' in tilt_sensors:
|
||||
logger.info(f"Processing {len(tilt_sensors['TL'])} TL sensors")
|
||||
# Load, convert, average, elaborate, write
|
||||
# Implementation would follow RSN pattern
|
||||
|
||||
# TLHR - Tilt Link High Resolution
|
||||
if 'TLHR' in tilt_sensors:
|
||||
logger.info(f"Processing {len(tilt_sensors['TLHR'])} TLHR sensors")
|
||||
# Similar processing
|
||||
|
||||
# BL - Biaxial Link
|
||||
if 'BL' in tilt_sensors:
|
||||
logger.info(f"Processing {len(tilt_sensors['BL'])} BL sensors")
|
||||
|
||||
# PL - Pendulum Link
|
||||
if 'PL' in tilt_sensors:
|
||||
logger.info(f"Processing {len(tilt_sensors['PL'])} PL sensors")
|
||||
|
||||
# Additional sensor types...
|
||||
|
||||
logger.info("Tilt processing completed successfully")
|
||||
|
||||
# Log elapsed time
|
||||
elapsed = time.time() - start_time
|
||||
log_elapsed_time(logger, elapsed)
|
||||
|
||||
return 0
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error processing Tilt chain: {e}", exc_info=True)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
if len(sys.argv) < 3:
|
||||
print("Usage: python -m src.tilt.main <control_unit_id> <chain>")
|
||||
sys.exit(1)
|
||||
|
||||
control_unit_id = sys.argv[1]
|
||||
chain = sys.argv[2]
|
||||
|
||||
exit_code = process_tilt_chain(control_unit_id, chain)
|
||||
sys.exit(exit_code)
|
||||
0
src/tilt/sensors/__init__.py
Normal file
0
src/tilt/sensors/__init__.py
Normal file
Reference in New Issue
Block a user