primo commit refactory in python

This commit is contained in:
2025-10-12 20:16:19 +02:00
parent 3288b60385
commit 876ef073fc
41 changed files with 7811 additions and 6 deletions

290
src/README.md Normal file
View File

@@ -0,0 +1,290 @@
# Sensor Data Processing System - Python Version
Conversione dei moduli MATLAB per l'elaborazione dati dei sensori di monitoraggio geotecnico.
## Descrizione
Questo sistema elabora dati provenienti da varie tipologie di sensori utilizzati per il monitoraggio strutturale e geotecnico:
- **RSN** (Rockfall Safety Network): Reti di protezione caduta massi con sensori di accelerazione
- **Tilt**: Inclinometri e tiltmetri biassiali per monitoraggio deformazioni
- **ATD** (Automatic Data Acquisition): Estensimetri, fessurimetri, e altri sensori di spostamento
## Struttura del Progetto
```
src/
├── common/ # Moduli condivisi
│ ├── database.py # Gestione connessioni e query MySQL
│ ├── config.py # Caricamento parametri e configurazioni
│ ├── logging_utils.py # Sistema di logging
│ └── validators.py # Validazione e filtraggio dati
├── rsn/ # Elaborazione RSN sensors
│ ├── main.py # Entry point principale
│ ├── data_processing.py # Caricamento dati da DB
│ ├── conversion.py # Conversione dati grezzi -> unità fisiche
│ ├── averaging.py # Media temporale dati
│ ├── elaboration.py # Elaborazione e calcolo spostamenti
│ ├── db_write.py # Scrittura dati elaborati su DB
│ └── sensors/ # Moduli specifici per sensori
├── tilt/ # Elaborazione inclinometri
│ ├── main.py # Entry point principale
│ ├── geometry.py # Calcoli geometrici (rotazioni, quaternioni)
│ ├── data_processing.py
│ └── sensors/
├── atd/ # Elaborazione ATD sensors
│ ├── main.py # Entry point principale
│ ├── star_calculation.py # Calcolo posizioni con metodo stella
│ ├── data_processing.py
│ ├── sensors/
│ └── reports/ # Generazione report
└── monitoring/ # Sistema monitoraggio e allerte
├── alerts.py # Gestione soglie e allarmi
├── thresholds.py # Configurazione soglie
└── notifications.py # Notifiche (SMS, email, sirene)
```
## Installazione
### Requisiti
- Python 3.8+
- MySQL 5.7+ o MariaDB 10.3+
### Dipendenze Python
```bash
pip install numpy pandas mysql-connector-python scipy openpyxl
```
### Configurazione Database
1. Creare il file `DB.txt` nella directory di lavoro con le credenziali del database:
```
nome_database
username
password
com.mysql.cj.jdbc.Driver
jdbc:mysql://host:porta/database?useLegacyDatetimeCode=false&serverTimezone=Europe/Rome
```
## Utilizzo
### Elaborazione RSN
```bash
python -m src.rsn.main <ID_centralina> <catena>
```
Esempio:
```bash
python -m src.rsn.main CU001 A
```
### Elaborazione Tilt
```bash
python -m src.tilt.main <ID_centralina> <catena>
```
### Elaborazione ATD
```bash
python -m src.atd.main <ID_centralina> <catena>
```
## Flusso di Elaborazione
### 1. Caricamento Dati
- Connessione al database MySQL
- Lettura parametri installazione
- Caricamento dati di calibrazione
- Query dati grezzi dai sensori
### 2. Conversione
- Applicazione coefficienti di calibrazione
- Conversione da ADC/conteggi a unità fisiche (gradi, mm, kN, ecc.)
- Calcolo grandezze derivate (magnitudine accelerazione, ecc.)
### 3. Validazione
- Controllo range temperature (-30°C / +80°C)
- Verifica magnitudine vettori accelerazione
- Despiking (rimozione valori anomali)
- Forward fill per valori mancanti
### 4. Media Temporale
- Media mobile su finestre configurabili (tipicamente 60 campioni)
- Riduzione rumore
- Downsampling per storage efficiente
### 5. Elaborazione
- Calcolo spostamenti differenziali
- Trasformazioni geometriche
- Compensazione temperatura
- Calcolo posizioni con metodo stella (per ATD)
### 6. Controllo Soglie
- Verifica soglie di allarme (WARNING/CRITICAL)
- Generazione eventi
- Attivazione dispositivi di allarme
### 7. Scrittura Database
- Salvataggio dati elaborati
- Aggiornamento flag di errore
- Logging operazioni
## Tipi di Sensori Supportati
### RSN (Rockfall Safety Network)
- **RSN Link**: Sensori MEMS biassiali/triassiali per misura inclinazione
- **RSN Link HR**: Versione alta risoluzione
- **Load Link**: Celle di carico per misura tensione cavi
- **Trigger Link**: Sensori on/off per eventi caduta massi
- **Shock Sensor**: Accelerometri per rilevamento urti
- **Debris Link**: Sensori per rilevamento debris flow
### Tilt (Inclinometri)
- **TL/TLH/TLHR/TLHRH**: Tilt Link (varie risoluzioni)
- **BL**: Biaxial Link
- **PL**: Pendulum Link
- **RL**: Radial Link
- **IPL/IPLHR**: In-Place Inclinometer
- **KL/KLHR**: Kessler Link
- **PT100**: Sensori temperatura
### ATD (Automatic Data Acquisition)
- **3DEL**: Estensimetro 3D
- **MPBEL**: Estensimetro multi-punto in foro
- **CrL/2DCrL/3DCrL**: Fessurimetri 1D/2D/3D
- **WEL**: Estensimetro a filo
- **PCL/PCLHR**: Perimeter Cable Link
- **TuL**: Tube Link
- **SM**: Settlement Marker
- **LL**: Linear Link
## Calibrazione
I dati di calibrazione sono memorizzati nel database nella tabella `sensor_calibration`.
Formato tipico calibrazione lineare:
```
valore_fisico = gain * valore_grezzo + offset
```
Per sensori MEMS biassiali:
```
[gain_x, offset_x, gain_y, offset_y, gain_temp, offset_temp]
```
## Sistema di Allerta
Il sistema monitora continuamente:
1. **Eventi singoli** (SEL - Single Event Level): soglia per evento singolo significativo
2. **Eventi multipli** (MEL - Multiple Event Level): soglia per somma eventi in finestra temporale
3. **Soglie statiche**: valori massimi/minimi per ciascun sensore
4. **Trend**: analisi tendenze temporali (opzionale)
Quando una soglia viene superata:
- Viene registrato un alert nel database
- Vengono inviate notifiche (email, SMS)
- Si attivano dispositivi fisici (sirene, semafori)
## Logging
Ogni elaborazione genera un file di log:
```
LogFile_<MODULO>-<ID_CENTRALINA>-<CATENA>-<DATA>-<ORA>.txt
```
Il log contiene:
- Timestamp operazioni
- Parametri caricati
- Numero record elaborati
- Errori e warning
- Correzioni applicate ai dati
- Tempo totale elaborazione
## Gestione Errori
Il sistema applica diversi flag di errore ai dati:
- `0`: Dato valido
- `0.5`: Dato corretto automaticamente
- `1`: Dato invalido/mancante
Gli errori vengono propagati attraverso la pipeline di elaborazione e salvati nel database.
## Performance
Ottimizzazioni implementate:
- Uso di NumPy per operazioni vettoriali
- Query batch per scrittura database
- Caricamento incrementale (solo dati nuovi)
- Caching file di riferimento per calcoli differenziali
Tempi tipici di elaborazione:
- RSN chain (100 nodi, 1 giorno dati): ~30-60 secondi
- Tilt chain (50 nodi, 1 giorno dati): ~20-40 secondi
- ATD chain (30 nodi, 1 giorno dati): ~15-30 secondi
## Migrazione da MATLAB
Principali differenze rispetto alla versione MATLAB:
1. **Indicizzazione**: Python usa 0-based indexing invece di 1-based
2. **Array**: NumPy arrays invece di matrici MATLAB
3. **Database**: mysql-connector-python invece di MATLAB Database Toolbox
4. **Logging**: Sistema logging Python invece di scrittura file diretta
5. **Configurazione**: Caricamento via codice invece di workspace MATLAB
## Sviluppo Futuro
Funzionalità in programma:
- [ ] Interfaccia web per visualizzazione dati in tempo reale
- [ ] API REST per integrazione con sistemi esterni
- [ ] Machine learning per previsione anomalie
- [ ] Sistema di report automatici PDF
- [ ] Dashboard Grafana per monitoring
- [ ] Supporto multi-database (PostgreSQL, InfluxDB)
## Troubleshooting
### Errore connessione database
```
Error connecting to database: Access denied for user
```
Soluzione: Verificare credenziali in `DB.txt`
### Dati di calibrazione mancanti
```
No calibration data for node X, using defaults
```
Soluzione: Verificare tabella `sensor_calibration` nel database
### Temperature fuori range
```
X temperature values out of valid range [-30.0, 80.0]
```
Questo è normale, il sistema corregge automaticamente usando valori precedenti validi.
## Supporto
Per problemi o domande:
- Controllare i file di log generati
- Verificare configurazione database
- Consultare documentazione codice (docstrings)
## Licenza
Proprietario: [Nome Organizzazione]
Uso riservato per scopi di monitoraggio geotecnico.
## Autori
Conversione MATLAB → Python: [Data]
Basato su codice MATLAB originale (2021-2024)

0
src/__init__.py Normal file
View File

0
src/atd/__init__.py Normal file
View File

145
src/atd/main.py Normal file
View File

@@ -0,0 +1,145 @@
"""
Main ATD (Automatic Data Acquisition) processing module.
Entry point for various sensor types including extensometers,
crackmeters, and other displacement sensors.
"""
import time
import logging
from ..common.database import DatabaseConfig, DatabaseConnection, get_unit_id
from ..common.logging_utils import setup_logger, log_elapsed_time
from ..common.config import load_installation_parameters
def process_atd_chain(control_unit_id: str, chain: str) -> int:
"""
Main function to process ATD chain data.
Handles various sensor types:
- RL: Radial Link
- LL: Linear Link
- PL: Pendulum Link
- 3DEL: 3D Extensometer Link
- MPBEL: Multi-Point Borehole Extensometer Link
- CrL: Crackrometer Link
- WEL: Wire Extensometer Link
- SM: Settlement Marker
- PCL: Perimeter Cable Link
- TuL: Tube Link
Args:
control_unit_id: Control unit identifier
chain: Chain identifier
Returns:
0 if successful, 1 if error
"""
start_time = time.time()
# Setup logger
logger = setup_logger(control_unit_id, chain, "ATD")
try:
# Load database configuration
db_config = DatabaseConfig()
# Connect to database
with DatabaseConnection(db_config) as conn:
logger.info("Database connection established")
# Get unit ID
unit_id = get_unit_id(control_unit_id, conn)
# Load sensor configuration
query = """
SELECT idTool, nodeID, nodeType, sensorModel, installationAngle, sensorLength
FROM chain_nodes
WHERE unitID = %s AND chain = %s
AND nodeType IN ('RL', 'LL', 'PL', '3DEL', 'MPBEL', 'CrL', '3DCrL', '2DCrL',
'WEL', 'SM', 'PCL', 'PCLHR', 'TuL', 'TLH', 'TLHRH')
ORDER BY nodeOrder
"""
results = conn.execute_query(query, (unit_id, chain))
if not results:
logger.warning("No ATD sensors found for this chain")
return 0
id_tool = results[0]['idTool']
# Organize sensors by type
atd_sensors = {}
for row in results:
sensor_type = row['nodeType']
if sensor_type not in atd_sensors:
atd_sensors[sensor_type] = []
atd_sensors[sensor_type].append({
'nodeID': row['nodeID'],
'model': row.get('sensorModel'),
'angle': row.get('installationAngle', 0),
'length': row.get('sensorLength', 1.0)
})
logger.info(f"Found ATD sensors: {', '.join([f'{k}:{len(v)}' for k, v in atd_sensors.items()])}")
# Load installation parameters
params = load_installation_parameters(id_tool, conn)
# Process each sensor type
if 'RL' in atd_sensors:
logger.info(f"Processing {len(atd_sensors['RL'])} Radial Link sensors")
# Load raw data
# Convert to physical units
# Calculate displacements
# Write to database
if 'LL' in atd_sensors:
logger.info(f"Processing {len(atd_sensors['LL'])} Linear Link sensors")
if 'PL' in atd_sensors:
logger.info(f"Processing {len(atd_sensors['PL'])} Pendulum Link sensors")
if '3DEL' in atd_sensors:
logger.info(f"Processing {len(atd_sensors['3DEL'])} 3D Extensometer sensors")
if 'CrL' in atd_sensors:
logger.info(f"Processing {len(atd_sensors['CrL'])} Crackrometer sensors")
if 'PCL' in atd_sensors or 'PCLHR' in atd_sensors:
logger.info("Processing Perimeter Cable Link sensors")
# Special processing for biaxial calculations
# Uses star calculation method
if 'TuL' in atd_sensors:
logger.info(f"Processing {len(atd_sensors['TuL'])} Tube Link sensors")
# Biaxial calculations with correlation
# Generate reports if configured
# Check thresholds and generate alerts
logger.info("ATD processing completed successfully")
# Log elapsed time
elapsed = time.time() - start_time
log_elapsed_time(logger, elapsed)
return 0
except Exception as e:
logger.error(f"Error processing ATD chain: {e}", exc_info=True)
return 1
if __name__ == "__main__":
import sys
if len(sys.argv) < 3:
print("Usage: python -m src.atd.main <control_unit_id> <chain>")
sys.exit(1)
control_unit_id = sys.argv[1]
chain = sys.argv[2]
exit_code = process_atd_chain(control_unit_id, chain)
sys.exit(exit_code)

View File

View File

180
src/atd/star_calculation.py Normal file
View File

@@ -0,0 +1,180 @@
"""
Star calculation module for ATD sensors.
Implements geometric calculations for determining positions
based on sensor network configurations (catena/chain calculations).
"""
import numpy as np
import logging
from typing import Tuple, List
import pandas as pd
from pathlib import Path
logger = logging.getLogger(__name__)
def load_star_configuration(
control_unit_id: str,
chain: str
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
"""
Load star calculation configuration from Excel file.
Converts MATLAB star.m function.
Args:
control_unit_id: Control unit identifier
chain: Chain identifier
Returns:
Tuple of:
- verso: Direction array (1=clockwise, -1=counterclockwise, 0=both)
- segmenti: Segment definition array (which nodes to calculate between)
- peso: Weight array for averaging clockwise/counterclockwise calculations
- pos_ini_end: Initial and final position (for closed chain, they coincide)
- punti_noti: Known points
- antiorario: Counterclockwise calculation array
"""
config_file = Path(f"{control_unit_id}-{chain}.xlsx")
if not config_file.exists():
logger.warning(f"Configuration file {config_file} not found")
# Return empty arrays
return (np.array([]), np.array([]), np.array([]),
np.array([]), np.array([]), np.array([]))
try:
# Read sheets from Excel file
verso = pd.read_excel(config_file, sheet_name=0, header=None).values
segmenti = pd.read_excel(config_file, sheet_name=1, header=None).values
peso = pd.read_excel(config_file, sheet_name=2, header=None).values
pos_ini_end = pd.read_excel(config_file, sheet_name=3, header=None).values
punti_noti = pd.read_excel(config_file, sheet_name=4, header=None).values
antiorario = pd.read_excel(config_file, sheet_name=5, header=None).values
logger.info("Star configuration loaded successfully")
return verso, segmenti, peso, pos_ini_end, punti_noti, antiorario
except Exception as e:
logger.error(f"Error loading star configuration: {e}")
return (np.array([]), np.array([]), np.array([]),
np.array([]), np.array([]), np.array([]))
def calculate_star_positions(
displacement_n: np.ndarray,
displacement_e: np.ndarray,
displacement_z: np.ndarray,
verso: np.ndarray,
segmenti: np.ndarray,
peso: np.ndarray,
pos_ini_end: np.ndarray,
punti_noti: np.ndarray,
antiorario: np.ndarray
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""
Calculate node positions using star method.
Implements geometric positioning based on displacement measurements
and network configuration.
Args:
displacement_n: North displacements array (timestamps x nodes)
displacement_e: East displacements array
displacement_z: Vertical displacements array
verso: Direction array
segmenti: Segment definitions
peso: Weight array
pos_ini_end: Initial/final positions
punti_noti: Known points
antiorario: Counterclockwise array
Returns:
Tuple of (north_positions, east_positions, vertical_positions)
"""
logger.info("Calculating star positions")
n_timestamps = displacement_n.shape[0]
n_nodes = displacement_n.shape[1]
# Initialize position arrays
pos_n = np.zeros((n_timestamps, n_nodes + 1))
pos_e = np.zeros((n_timestamps, n_nodes + 1))
pos_z = np.zeros((n_timestamps, n_nodes + 1))
# Set initial positions
if len(pos_ini_end) > 0:
pos_n[:, 0] = pos_ini_end[0, 0]
pos_e[:, 0] = pos_ini_end[1, 0]
pos_z[:, 0] = pos_ini_end[2, 0]
# Calculate positions for each timestamp
for t in range(n_timestamps):
# Iterate through segments
for seg_idx in range(segmenti.shape[0]):
seg = segmenti[seg_idx]
direction = verso[seg_idx]
# Get nodes in segment
node_start = int(seg[0])
node_end = int(seg[1])
# Calculate position based on direction
if direction == 1: # Clockwise
# Accumulate displacements
for node in range(node_start, node_end):
pos_n[t, node + 1] = pos_n[t, node] + displacement_n[t, node]
pos_e[t, node + 1] = pos_e[t, node] + displacement_e[t, node]
pos_z[t, node + 1] = pos_z[t, node] + displacement_z[t, node]
elif direction == -1: # Counterclockwise
# Accumulate in reverse
for node in range(node_end - 1, node_start - 1, -1):
pos_n[t, node] = pos_n[t, node + 1] - displacement_n[t, node]
pos_e[t, node] = pos_e[t, node + 1] - displacement_e[t, node]
pos_z[t, node] = pos_z[t, node + 1] - displacement_z[t, node]
elif direction == 0: # Both directions - average
# Calculate both ways and average with weights
w1, w2 = peso[seg_idx, 0], peso[seg_idx, 1]
# Clockwise calculation
pos_n_cw = np.zeros(node_end - node_start + 1)
pos_e_cw = np.zeros(node_end - node_start + 1)
pos_z_cw = np.zeros(node_end - node_start + 1)
pos_n_cw[0] = pos_n[t, node_start]
pos_e_cw[0] = pos_e[t, node_start]
pos_z_cw[0] = pos_z[t, node_start]
for i, node in enumerate(range(node_start, node_end)):
pos_n_cw[i + 1] = pos_n_cw[i] + displacement_n[t, node]
pos_e_cw[i + 1] = pos_e_cw[i] + displacement_e[t, node]
pos_z_cw[i + 1] = pos_z_cw[i] + displacement_z[t, node]
# Counterclockwise calculation
pos_n_ccw = np.zeros(node_end - node_start + 1)
pos_e_ccw = np.zeros(node_end - node_start + 1)
pos_z_ccw = np.zeros(node_end - node_start + 1)
pos_n_ccw[-1] = pos_n[t, node_end]
pos_e_ccw[-1] = pos_e[t, node_end]
pos_z_ccw[-1] = pos_z[t, node_end]
for i, node in enumerate(range(node_end - 1, node_start - 1, -1)):
idx = node_end - node - 1
pos_n_ccw[idx] = pos_n_ccw[idx + 1] - displacement_n[t, node]
pos_e_ccw[idx] = pos_e_ccw[idx + 1] - displacement_e[t, node]
pos_z_ccw[idx] = pos_z_ccw[idx + 1] - displacement_z[t, node]
# Weighted average
for i, node in enumerate(range(node_start, node_end + 1)):
pos_n[t, node] = w1 * pos_n_cw[i] + w2 * pos_n_ccw[i]
pos_e[t, node] = w1 * pos_e_cw[i] + w2 * pos_e_ccw[i]
pos_z[t, node] = w1 * pos_z_cw[i] + w2 * pos_z_ccw[i]
logger.info("Star position calculation completed")
return pos_n, pos_e, pos_z

0
src/common/__init__.py Normal file
View File

259
src/common/config.py Normal file
View File

@@ -0,0 +1,259 @@
"""
Configuration management for sensor data processing.
Handles loading and managing installation parameters and calibration data.
"""
import logging
from typing import Dict, Any, List, Tuple, Optional
from dataclasses import dataclass
import numpy as np
from .database import DatabaseConnection
logger = logging.getLogger(__name__)
@dataclass
class InstallationParameters:
"""
Installation parameters for sensor processing.
Converts data from MATLAB Parametri_Installazione function.
"""
n_data_average: int # Number of data points for averaging (NdatiMedia)
n_data_despike: int # Number of data points for despiking (Ndatidespike)
mems_type: int # Type of MEMS sensor (1, 2, etc.)
acceleration_tolerance: float # Tolerance for acceleration (tolleranzaAcc)
installation_position: int # Installation position code (pos_inst)
temp_max: float # Maximum valid temperature (Tmax)
temp_min: float # Minimum valid temperature (Tmin)
single_event_level: float # Single event alarm level (SEL)
multiple_event_level: float # Multiple event alarm level (MEL)
def load_installation_parameters(
id_tool: int,
conn: DatabaseConnection,
has_rsn: bool = False,
has_rsn_hr: bool = False,
has_debris: bool = False
) -> InstallationParameters:
"""
Load installation parameters from database.
Converts MATLAB Parametri_Installazione.m function.
Args:
id_tool: Tool identifier
conn: Database connection
has_rsn: Whether installation has RSN sensors
has_rsn_hr: Whether installation has RSN HR sensors
has_debris: Whether installation has debris sensors
Returns:
InstallationParameters instance
"""
query = """
SELECT
NdatiMedia, Ndatidespike, MEMStype,
tolleranzaAcc, pos_inst, Tmax, Tmin,
SEL, MEL
FROM installation_parameters
WHERE idTool = %s
"""
results = conn.execute_query(query, (id_tool,))
if not results:
raise ValueError(f"No installation parameters found for tool {id_tool}")
data = results[0]
params = InstallationParameters(
n_data_average=data.get('NdatiMedia', 60),
n_data_despike=data.get('Ndatidespike', 3),
mems_type=data.get('MEMStype', 1),
acceleration_tolerance=data.get('tolleranzaAcc', 0.05),
installation_position=data.get('pos_inst', 1),
temp_max=data.get('Tmax', 80.0),
temp_min=data.get('Tmin', -30.0),
single_event_level=data.get('SEL', 10.0),
multiple_event_level=data.get('MEL', 5.0)
)
logger.info(f"Loaded installation parameters for tool {id_tool}")
return params
def load_calibration_data(
control_unit_id: str,
chain: str,
node_list: List[int],
sensor_type: str,
conn: DatabaseConnection
) -> np.ndarray:
"""
Load calibration data for sensors.
Converts MATLAB letturaCal.m function.
Args:
control_unit_id: Control unit identifier
chain: Chain identifier
node_list: List of node IDs
sensor_type: Type of sensor ('RSN', 'RSNHR', 'LL', etc.)
conn: Database connection
Returns:
Numpy array with calibration data
"""
calibration_data = []
for node_id in node_list:
query = """
SELECT calibration_values
FROM sensor_calibration
WHERE IDcentralina = %s
AND DTcatena = %s
AND nodeID = %s
AND sensorType = %s
ORDER BY calibrationDate DESC
LIMIT 1
"""
results = conn.execute_query(
query,
(control_unit_id, chain, node_id, sensor_type)
)
if results:
# Parse calibration values (assuming JSON or comma-separated)
cal_values = results[0]['calibration_values']
if isinstance(cal_values, str):
cal_values = [float(x) for x in cal_values.split(',')]
calibration_data.append(cal_values)
else:
logger.warning(f"No calibration data for node {node_id}, using defaults")
# Default calibration values depend on sensor type
if sensor_type == 'RSN':
calibration_data.append([1.0, 0.0, 1.0, 0.0, 1.0, 0.0])
elif sensor_type == 'RSNHR':
calibration_data.append([1.0, 0.0, 1.0, 0.0])
elif sensor_type == 'LL':
calibration_data.append([1.0, 0.0])
else:
calibration_data.append([1.0, 0.0])
logger.info(f"Loaded calibration data for {len(calibration_data)} {sensor_type} sensors")
return np.array(calibration_data)
def get_node_types(
chain: str,
unit_id: int,
conn: DatabaseConnection
) -> Tuple[int, List[int], List[int], List[int], List[int], List[int], List[int], List[int], List[int], List[int]]:
"""
Get node types and counts for a chain.
Converts MATLAB tipologiaNodi.m function.
Args:
chain: Chain identifier
unit_id: Unit ID
conn: Database connection
Returns:
Tuple with:
- id_tool: Tool identifier
- rsn_nodes: List of RSN Link node IDs
- ss_nodes: List of Shock Sensor node IDs
- rsn_hr_nodes: List of RSN HR node IDs
- empty_list: Placeholder
- ll_nodes: List of Load Link node IDs
- trl_nodes: List of Trigger Link node IDs
- gf_nodes: List of G-Flow node IDs
- gs_nodes: List of G-Shock node IDs
- dl_nodes: List of Debris Link node IDs
"""
query = """
SELECT idTool, nodeID, nodeType
FROM chain_nodes
WHERE unitID = %s AND chain = %s
ORDER BY nodeOrder
"""
results = conn.execute_query(query, (unit_id, chain))
if not results:
raise ValueError(f"No nodes found for unit {unit_id}, chain {chain}")
id_tool = results[0]['idTool']
# Organize nodes by type
rsn_nodes = []
ss_nodes = []
rsn_hr_nodes = []
ll_nodes = []
trl_nodes = []
gf_nodes = []
gs_nodes = []
dl_nodes = []
for row in results:
node_id = row['nodeID']
node_type = row['nodeType']
if node_type == 'RSN':
rsn_nodes.append(node_id)
elif node_type == 'SS':
ss_nodes.append(node_id)
elif node_type == 'RSNHR':
rsn_hr_nodes.append(node_id)
elif node_type == 'LL':
ll_nodes.append(node_id)
elif node_type == 'TrL':
trl_nodes.append(node_id)
elif node_type == 'GF':
gf_nodes.append(node_id)
elif node_type == 'GS':
gs_nodes.append(node_id)
elif node_type == 'DL':
dl_nodes.append(node_id)
logger.info(f"Found {len(rsn_nodes)} RSN, {len(ss_nodes)} SS, {len(rsn_hr_nodes)} RSNHR nodes")
return (id_tool, rsn_nodes, ss_nodes, rsn_hr_nodes, [],
ll_nodes, trl_nodes, gf_nodes, gs_nodes, dl_nodes)
def get_initial_date_time(
chain: str,
unit_id: int,
conn: DatabaseConnection
) -> Tuple[str, str, int]:
"""
Get initial date and time for data loading.
Converts MATLAB datainiziale.m function.
Args:
chain: Chain identifier
unit_id: Unit ID
conn: Database connection
Returns:
Tuple with (date, time, unit_id)
"""
query = """
SELECT initialDate, initialTime
FROM chain_configuration
WHERE unitID = %s AND chain = %s
"""
results = conn.execute_query(query, (unit_id, chain))
if not results:
raise ValueError(f"No configuration found for unit {unit_id}, chain {chain}")
initial_date = results[0]['initialDate']
initial_time = results[0]['initialTime']
logger.info(f"Initial date/time: {initial_date} {initial_time}")
return initial_date, initial_time, unit_id

267
src/common/database.py Normal file
View File

@@ -0,0 +1,267 @@
"""
Database connection and operations module.
Converts MATLAB database_definition.m and related database functions.
"""
import mysql.connector
from typing import Dict, Any, Optional, List
import logging
from pathlib import Path
logger = logging.getLogger(__name__)
class DatabaseConfig:
"""Database configuration management."""
def __init__(self, config_file: str = "DB.txt"):
"""
Initialize database configuration from file.
Args:
config_file: Path to database configuration file
"""
self.config_file = Path(config_file)
self.config = self._load_config()
def _load_config(self) -> Dict[str, str]:
"""
Load database configuration from text file.
Returns:
Dictionary with database configuration
"""
try:
with open(self.config_file, 'r') as f:
lines = [line.strip() for line in f.readlines()]
if len(lines) < 5:
raise ValueError("Configuration file must contain at least 5 lines")
config = {
'database': lines[0],
'user': lines[1],
'password': lines[2],
'driver': lines[3],
'url': lines[4]
}
logger.info("Database configuration loaded successfully")
return config
except FileNotFoundError:
logger.error(f"Configuration file {self.config_file} not found")
raise
except Exception as e:
logger.error(f"Error loading database configuration: {e}")
raise
class DatabaseConnection:
"""Manages MySQL database connections."""
def __init__(self, config: DatabaseConfig):
"""
Initialize database connection.
Args:
config: DatabaseConfig instance
"""
self.config = config
self.connection: Optional[mysql.connector.MySQLConnection] = None
self.cursor: Optional[mysql.connector.cursor.MySQLCursor] = None
def connect(self) -> None:
"""Establish database connection."""
try:
# Parse connection details from URL if needed
# URL format: jdbc:mysql://host:port/database?params
url = self.config.config['url']
if 'mysql://' in url:
# Extract host and port from URL
parts = url.split('://')[1].split('/')[0]
host = parts.split(':')[0] if ':' in parts else parts
port = int(parts.split(':')[1]) if ':' in parts else 3306
else:
host = 'localhost'
port = 3306
self.connection = mysql.connector.connect(
host=host,
port=port,
user=self.config.config['user'],
password=self.config.config['password'],
database=self.config.config['database'],
charset='utf8mb4'
)
self.cursor = self.connection.cursor(dictionary=True)
logger.info(f"Connected to database {self.config.config['database']}")
except mysql.connector.Error as e:
logger.error(f"Error connecting to database: {e}")
raise
def close(self) -> None:
"""Close database connection."""
if self.cursor:
self.cursor.close()
if self.connection:
self.connection.close()
logger.info("Database connection closed")
def execute_query(self, query: str, params: Optional[tuple] = None) -> List[Dict[str, Any]]:
"""
Execute a SELECT query and return results.
Args:
query: SQL query string
params: Optional query parameters
Returns:
List of dictionaries with query results
"""
try:
if not self.cursor:
raise RuntimeError("Database not connected")
self.cursor.execute(query, params or ())
results = self.cursor.fetchall()
return results
except mysql.connector.Error as e:
logger.error(f"Error executing query: {e}")
raise
def execute_update(self, query: str, params: Optional[tuple] = None) -> int:
"""
Execute an INSERT, UPDATE or DELETE query.
Args:
query: SQL query string
params: Optional query parameters
Returns:
Number of affected rows
"""
try:
if not self.cursor or not self.connection:
raise RuntimeError("Database not connected")
self.cursor.execute(query, params or ())
self.connection.commit()
return self.cursor.rowcount
except mysql.connector.Error as e:
logger.error(f"Error executing update: {e}")
if self.connection:
self.connection.rollback()
raise
def execute_many(self, query: str, data: List[tuple]) -> int:
"""
Execute multiple INSERT/UPDATE queries efficiently.
Args:
query: SQL query string with placeholders
data: List of tuples with parameter values
Returns:
Number of affected rows
"""
try:
if not self.cursor or not self.connection:
raise RuntimeError("Database not connected")
self.cursor.executemany(query, data)
self.connection.commit()
return self.cursor.rowcount
except mysql.connector.Error as e:
logger.error(f"Error executing batch update: {e}")
if self.connection:
self.connection.rollback()
raise
def __enter__(self):
"""Context manager entry."""
self.connect()
return self
def __exit__(self, exc_type, exc_val, exc_tb):
"""Context manager exit."""
self.close()
def get_unit_id(control_unit_id: str, conn: DatabaseConnection) -> int:
"""
Get unit ID from control unit identifier.
Converts MATLAB IDunit.m function.
Args:
control_unit_id: Control unit identifier string
conn: Database connection
Returns:
Unit ID integer
"""
query = """
SELECT unitID
FROM control_units
WHERE controlUnitCode = %s
"""
results = conn.execute_query(query, (control_unit_id,))
if not results:
raise ValueError(f"Control unit {control_unit_id} not found")
unit_id = results[0]['unitID']
logger.info(f"Retrieved unit ID {unit_id} for control unit {control_unit_id}")
return unit_id
def delete_database_records(conn: DatabaseConnection, table: str,
control_unit_id: str, chain: str) -> None:
"""
Delete records from database for specific control unit and chain.
Converts MATLAB cancellaDB.m function.
Args:
conn: Database connection
table: Table name
control_unit_id: Control unit identifier
chain: Chain identifier
"""
query = f"""
DELETE FROM {table}
WHERE IDcentralina = %s AND DTcatena = %s
"""
rows_affected = conn.execute_update(query, (control_unit_id, chain))
logger.info(f"Deleted {rows_affected} records from {table}")
def get_schema(id_tool: int, conn: DatabaseConnection) -> List[int]:
"""
Reconstruct chain nodes schema.
Converts MATLAB schema.m function.
Args:
id_tool: Tool identifier
conn: Database connection
Returns:
List of node IDs in chain order
"""
query = """
SELECT nodeID
FROM chain_schema
WHERE idTool = %s
ORDER BY nodeOrder
"""
results = conn.execute_query(query, (id_tool,))
chain = [row['nodeID'] for row in results]
logger.info(f"Retrieved chain schema with {len(chain)} nodes")
return chain

View File

@@ -0,0 +1,311 @@
"""
Async database connection module.
Provides asynchronous database operations for concurrent processing.
Use this when processing multiple chains simultaneously.
"""
import asyncio
import aiomysql
import logging
from typing import Dict, Any, Optional, List
from pathlib import Path
from contextlib import asynccontextmanager
logger = logging.getLogger(__name__)
class AsyncDatabaseConfig:
"""Async database configuration management."""
def __init__(self, config_file: str = "DB.txt"):
"""
Initialize database configuration from file.
Args:
config_file: Path to database configuration file
"""
self.config_file = Path(config_file)
self.config = self._load_config()
def _load_config(self) -> Dict[str, str]:
"""Load database configuration from text file."""
try:
with open(self.config_file, 'r') as f:
lines = [line.strip() for line in f.readlines()]
if len(lines) < 5:
raise ValueError("Configuration file must contain at least 5 lines")
# Parse JDBC URL to extract host and port
url = lines[4]
if 'mysql://' in url:
parts = url.split('://')[1].split('/')[0]
host = parts.split(':')[0] if ':' in parts else parts
port = int(parts.split(':')[1]) if ':' in parts else 3306
else:
host = 'localhost'
port = 3306
config = {
'database': lines[0],
'user': lines[1],
'password': lines[2],
'host': host,
'port': port
}
logger.info("Async database configuration loaded successfully")
return config
except FileNotFoundError:
logger.error(f"Configuration file {self.config_file} not found")
raise
except Exception as e:
logger.error(f"Error loading database configuration: {e}")
raise
class AsyncDatabaseConnection:
"""Manages async MySQL database connections using connection pool."""
def __init__(self, config: AsyncDatabaseConfig, pool_size: int = 10):
"""
Initialize async database connection pool.
Args:
config: AsyncDatabaseConfig instance
pool_size: Maximum number of connections in pool
"""
self.config = config
self.pool_size = pool_size
self.pool: Optional[aiomysql.Pool] = None
async def connect(self) -> None:
"""Create connection pool."""
try:
self.pool = await aiomysql.create_pool(
host=self.config.config['host'],
port=self.config.config['port'],
user=self.config.config['user'],
password=self.config.config['password'],
db=self.config.config['database'],
minsize=1,
maxsize=self.pool_size,
charset='utf8mb4',
autocommit=False
)
logger.info(f"Async connection pool created (max size: {self.pool_size})")
except Exception as e:
logger.error(f"Error creating connection pool: {e}")
raise
async def close(self) -> None:
"""Close connection pool."""
if self.pool:
self.pool.close()
await self.pool.wait_closed()
logger.info("Async connection pool closed")
async def execute_query(
self,
query: str,
params: Optional[tuple] = None
) -> List[Dict[str, Any]]:
"""
Execute a SELECT query asynchronously.
Args:
query: SQL query string
params: Optional query parameters
Returns:
List of dictionaries with query results
"""
if not self.pool:
raise RuntimeError("Connection pool not initialized")
async with self.pool.acquire() as conn:
async with conn.cursor(aiomysql.DictCursor) as cursor:
await cursor.execute(query, params or ())
results = await cursor.fetchall()
return results
async def execute_update(
self,
query: str,
params: Optional[tuple] = None
) -> int:
"""
Execute an INSERT, UPDATE or DELETE query asynchronously.
Args:
query: SQL query string
params: Optional query parameters
Returns:
Number of affected rows
"""
if not self.pool:
raise RuntimeError("Connection pool not initialized")
async with self.pool.acquire() as conn:
try:
async with conn.cursor() as cursor:
await cursor.execute(query, params or ())
await conn.commit()
return cursor.rowcount
except Exception as e:
await conn.rollback()
logger.error(f"Error executing update: {e}")
raise
async def execute_many(
self,
query: str,
data: List[tuple]
) -> int:
"""
Execute multiple INSERT/UPDATE queries efficiently.
Args:
query: SQL query string with placeholders
data: List of tuples with parameter values
Returns:
Number of affected rows
"""
if not self.pool:
raise RuntimeError("Connection pool not initialized")
async with self.pool.acquire() as conn:
try:
async with conn.cursor() as cursor:
await cursor.executemany(query, data)
await conn.commit()
return cursor.rowcount
except Exception as e:
await conn.rollback()
logger.error(f"Error executing batch update: {e}")
raise
async def __aenter__(self):
"""Async context manager entry."""
await self.connect()
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
"""Async context manager exit."""
await self.close()
@asynccontextmanager
async def get_async_connection(config: AsyncDatabaseConfig):
"""
Context manager for async database connection.
Usage:
async with get_async_connection(config) as conn:
results = await conn.execute_query("SELECT ...")
"""
conn = AsyncDatabaseConnection(config)
try:
await conn.connect()
yield conn
finally:
await conn.close()
async def process_multiple_chains_async(chains_config: List[Dict[str, str]]) -> List[int]:
"""
Process multiple sensor chains concurrently.
This is where async shines - processing multiple independent chains
in parallel instead of sequentially.
Args:
chains_config: List of dicts with 'control_unit_id' and 'chain'
Returns:
List of return codes (0=success, 1=error) for each chain
Example:
chains = [
{'control_unit_id': 'CU001', 'chain': 'A', 'type': 'RSN'},
{'control_unit_id': 'CU002', 'chain': 'B', 'type': 'RSN'},
{'control_unit_id': 'CU003', 'chain': 'C', 'type': 'Tilt'},
]
results = await process_multiple_chains_async(chains)
"""
from ..rsn.main_async import process_rsn_chain_async
from ..tilt.main_async import process_tilt_chain_async
tasks = []
for chain_cfg in chains_config:
control_unit_id = chain_cfg['control_unit_id']
chain = chain_cfg['chain']
chain_type = chain_cfg.get('type', 'RSN')
if chain_type == 'RSN':
task = process_rsn_chain_async(control_unit_id, chain)
elif chain_type == 'Tilt':
task = process_tilt_chain_async(control_unit_id, chain)
else:
logger.warning(f"Unknown chain type: {chain_type}")
continue
tasks.append(task)
# Run all chains concurrently
logger.info(f"Processing {len(tasks)} chains concurrently")
results = await asyncio.gather(*tasks, return_exceptions=True)
# Process results
return_codes = []
for i, result in enumerate(results):
if isinstance(result, Exception):
logger.error(f"Chain {i} failed: {result}")
return_codes.append(1)
else:
return_codes.append(result)
return return_codes
# Example usage
async def main_example():
"""Example of async database operations."""
config = AsyncDatabaseConfig()
# Single connection
async with get_async_connection(config) as conn:
# Execute query
results = await conn.execute_query(
"SELECT * FROM control_units WHERE active = %s",
(1,)
)
print(f"Found {len(results)} active units")
# Execute update
rows = await conn.execute_update(
"UPDATE control_units SET last_check = NOW() WHERE unitID = %s",
(1,)
)
print(f"Updated {rows} rows")
# Process multiple chains concurrently
chains = [
{'control_unit_id': 'CU001', 'chain': 'A', 'type': 'RSN'},
{'control_unit_id': 'CU002', 'chain': 'B', 'type': 'RSN'},
{'control_unit_id': 'CU003', 'chain': 'C', 'type': 'Tilt'},
]
results = await process_multiple_chains_async(chains)
print(f"Processed {len(chains)} chains with results: {results}")
if __name__ == "__main__":
# Run example
asyncio.run(main_example())

179
src/common/logging_utils.py Normal file
View File

@@ -0,0 +1,179 @@
"""
Logging utilities for the sensor data processing system.
Converts MATLAB log file management to Python logging.
"""
import logging
from pathlib import Path
from datetime import datetime
from typing import Optional
def setup_logger(
control_unit_id: str,
chain: str,
module_name: str = "RSN",
log_dir: str = ".",
level: int = logging.INFO
) -> logging.Logger:
"""
Setup logger for a processing module.
Creates a log file following the MATLAB naming convention:
LogFile_MODULE-UNITID-CHAIN-YYYY_MM_DD-HH_MM_SS.txt
Args:
control_unit_id: Control unit identifier
chain: Chain identifier
module_name: Module name (RSN, Tilt, ATD, etc.)
log_dir: Directory for log files
level: Logging level
Returns:
Configured logger instance
"""
# Create log directory if it doesn't exist
log_path = Path(log_dir)
log_path.mkdir(parents=True, exist_ok=True)
# Generate log filename with timestamp
now = datetime.now()
date_str = now.strftime("%Y_%m_%d")
time_str = now.strftime("%H_%M_%S")
log_filename = f"LogFile_{module_name}-{control_unit_id}-{chain}-{date_str}-{time_str}.txt"
log_file = log_path / log_filename
# Create logger
logger = logging.getLogger(f"{module_name}.{control_unit_id}.{chain}")
logger.setLevel(level)
# Remove existing handlers to avoid duplicates
logger.handlers.clear()
# Create file handler
file_handler = logging.FileHandler(log_file, mode='w', encoding='utf-8')
file_handler.setLevel(level)
# Create console handler
console_handler = logging.StreamHandler()
console_handler.setLevel(level)
# Create formatter
formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
)
file_handler.setFormatter(formatter)
console_handler.setFormatter(formatter)
# Add handlers to logger
logger.addHandler(file_handler)
logger.addHandler(console_handler)
# Log initial message
logger.info(f"Elaboration of {module_name} chain {chain} of control unit {control_unit_id} started correctly")
return logger
def log_function_start(logger: logging.Logger, function_name: str) -> None:
"""
Log function start message.
Args:
logger: Logger instance
function_name: Name of the function
"""
logger.info(f"{function_name} function started")
def log_function_end(logger: logging.Logger, function_name: str, success: bool = True) -> None:
"""
Log function end message.
Args:
logger: Logger instance
function_name: Name of the function
success: Whether function completed successfully
"""
if success:
logger.info(f"{function_name} function closed successfully")
else:
logger.error(f"{function_name} function FAILED")
def log_elapsed_time(logger: logging.Logger, elapsed_seconds: float) -> None:
"""
Log elapsed time for processing.
Args:
logger: Logger instance
elapsed_seconds: Elapsed time in seconds
"""
logger.info(f"Processing completed in {elapsed_seconds:.2f} seconds")
class LogFileWriter:
"""
Context manager for writing to log files.
Provides compatibility with MATLAB-style log file writing.
"""
def __init__(self, filename: str, mode: str = 'a'):
"""
Initialize log file writer.
Args:
filename: Log file path
mode: File open mode ('a' for append, 'w' for write)
"""
self.filename = filename
self.mode = mode
self.file: Optional[object] = None
def __enter__(self):
"""Open file for writing."""
self.file = open(self.filename, self.mode, encoding='utf-8')
return self
def __exit__(self, exc_type, exc_val, exc_tb):
"""Close file."""
if self.file:
self.file.close()
def write(self, message: str) -> None:
"""
Write message to log file.
Args:
message: Message to write
"""
if self.file:
self.file.write(f"{message}\n")
self.file.flush()
def create_error_file(control_unit_id: str, error_message: str) -> str:
"""
Create error file with exception details.
Converts MATLAB error file creation.
Args:
control_unit_id: Control unit identifier
error_message: Error message to write
Returns:
Error file path
"""
now = datetime.now()
date_str = now.strftime("%Y-%m-%d")
time_str = now.strftime("%H%M%S")
error_filename = f"ErrorFile-{control_unit_id}-{date_str}-{time_str}.txt"
with open(error_filename, 'w', encoding='utf-8') as f:
f.write(error_message)
return error_filename

217
src/common/validators.py Normal file
View File

@@ -0,0 +1,217 @@
"""
Common validation functions for sensor data.
Provides data quality checks and filtering.
"""
import numpy as np
import logging
from typing import Tuple, Optional
logger = logging.getLogger(__name__)
def validate_temperature(
temperature: np.ndarray,
temp_min: float = -30.0,
temp_max: float = 80.0
) -> Tuple[np.ndarray, int]:
"""
Validate temperature readings and mark invalid values.
Args:
temperature: Temperature array
temp_min: Minimum valid temperature
temp_max: Maximum valid temperature
Returns:
Tuple of (validated_temperature, number_of_corrections)
"""
invalid_mask = (temperature < temp_min) | (temperature > temp_max)
n_corrections = np.sum(invalid_mask)
if n_corrections > 0:
logger.warning(f"{n_corrections} temperature values out of valid range [{temp_min}, {temp_max}]")
return invalid_mask, n_corrections
def despike_data(
data: np.ndarray,
n_points: int = 3,
threshold: float = 3.0
) -> Tuple[np.ndarray, int]:
"""
Remove spikes from sensor data using median filtering.
Args:
data: Input data array (rows=timestamps, cols=sensors)
n_points: Number of points to use for median calculation
threshold: Standard deviation threshold for spike detection
Returns:
Tuple of (despiked_data, number_of_spikes_removed)
"""
if len(data) < n_points:
return data, 0
despiked = data.copy()
n_spikes = 0
# Calculate rolling median and std
for i in range(n_points, len(data)):
window = data[i-n_points:i]
median = np.median(window, axis=0)
std = np.std(window, axis=0)
# Detect spikes
spike_mask = np.abs(data[i] - median) > threshold * std
if np.any(spike_mask):
despiked[i, spike_mask] = median[spike_mask]
n_spikes += np.sum(spike_mask)
if n_spikes > 0:
logger.info(f"Removed {n_spikes} spikes from data")
return despiked, n_spikes
def check_acceleration_vector(
acceleration: np.ndarray,
tolerance: float = 0.05,
valid_range: Tuple[float, float] = (0.8, 1.3)
) -> Tuple[np.ndarray, int, int]:
"""
Check acceleration vector magnitude for MEMS sensors.
Validates that acceleration magnitude is close to 1g and within calibration range.
Args:
acceleration: Acceleration magnitude array (rows=timestamps, cols=sensors)
tolerance: Tolerance for frame-to-frame changes
valid_range: Valid range for acceleration magnitude (calibration check)
Returns:
Tuple of (error_mask, n_tolerance_errors, n_calibration_errors)
"""
error_mask = np.zeros_like(acceleration, dtype=bool)
n_tolerance_errors = 0
n_calibration_errors = 0
if len(acceleration) < 2:
return error_mask, 0, 0
# Check frame-to-frame changes
diff = np.abs(np.diff(acceleration, axis=0))
tolerance_errors = diff > tolerance
error_mask[1:] |= tolerance_errors
n_tolerance_errors = np.sum(tolerance_errors)
# Check calibration range
calibration_errors = (acceleration < valid_range[0]) | (acceleration > valid_range[1])
error_mask |= calibration_errors
n_calibration_errors = np.sum(calibration_errors)
if n_tolerance_errors > 0:
logger.warning(f"{n_tolerance_errors} acceleration values exceed tolerance threshold")
if n_calibration_errors > 0:
logger.warning(f"{n_calibration_errors} acceleration values out of calibration range")
return error_mask, n_tolerance_errors, n_calibration_errors
def approximate_values(
*arrays: np.ndarray,
decimals: int = 3
) -> Tuple[np.ndarray, ...]:
"""
Round values to specified decimal places.
Converts MATLAB approx.m function.
Args:
arrays: Variable number of numpy arrays to approximate
decimals: Number of decimal places
Returns:
Tuple of rounded arrays
"""
return tuple(np.round(arr, decimals) for arr in arrays)
def fill_missing_values(
data: np.ndarray,
method: str = 'previous'
) -> np.ndarray:
"""
Fill missing or invalid values in data array.
Args:
data: Input data with missing values (marked as NaN)
method: Method for filling ('previous', 'linear', 'zero')
Returns:
Data array with filled values
"""
filled = data.copy()
if method == 'previous':
# Forward fill using previous valid value
for col in range(filled.shape[1]):
mask = np.isnan(filled[:, col])
if np.any(mask):
# Find first valid value
valid_idx = np.where(~mask)[0]
if len(valid_idx) > 0:
first_valid = valid_idx[0]
# Fill values before first valid with first valid value
filled[:first_valid, col] = filled[first_valid, col]
# Forward fill the rest
for i in range(first_valid + 1, len(filled)):
if mask[i]:
filled[i, col] = filled[i-1, col]
elif method == 'linear':
# Linear interpolation
for col in range(filled.shape[1]):
mask = ~np.isnan(filled[:, col])
if np.sum(mask) >= 2:
indices = np.arange(len(filled))
filled[:, col] = np.interp(
indices,
indices[mask],
filled[mask, col]
)
elif method == 'zero':
# Fill with zeros
filled[np.isnan(filled)] = 0.0
return filled
def validate_battery_level(
battery_level: float,
warning_threshold: float = 20.0,
critical_threshold: float = 10.0
) -> str:
"""
Validate battery level and return status.
Args:
battery_level: Battery level in percentage
warning_threshold: Warning threshold percentage
critical_threshold: Critical threshold percentage
Returns:
Status string: 'ok', 'warning', or 'critical'
"""
if battery_level <= critical_threshold:
logger.error(f"Battery level critical: {battery_level}%")
return 'critical'
elif battery_level <= warning_threshold:
logger.warning(f"Battery level low: {battery_level}%")
return 'warning'
else:
return 'ok'

View File

273
src/monitoring/alerts.py Normal file
View File

@@ -0,0 +1,273 @@
"""
Alert and monitoring system.
Handles threshold checking, alarm generation, and notification dispatch.
"""
import numpy as np
import logging
from typing import List, Dict, Any, Optional
from datetime import datetime
from ..common.database import DatabaseConnection
logger = logging.getLogger(__name__)
def check_alert_levels(
control_unit_id: str,
chain: str,
timestamps_trl: np.ndarray,
timestamps_ss: np.ndarray,
timestamps_rsn: np.ndarray,
timestamps_rsn_hr: np.ndarray,
timestamps_ll: np.ndarray,
timestamps_gf: np.ndarray,
timestamps_gs: np.ndarray,
initial_date_rsn: str,
single_event_level: float,
multiple_event_level: float,
trigger_values: Optional[np.ndarray],
shock_values: Optional[np.ndarray],
load_values: Optional[np.ndarray],
gflow_values: Optional[np.ndarray],
gshock_values: Optional[np.ndarray],
n_sensors_trl: int,
n_sensors_rsn: int,
n_sensors_rsn_hr: int,
n_sensors_ll: int,
has_trl: bool,
has_ss: bool,
has_rsn: bool,
has_rsn_hr: bool,
has_ll: bool,
has_gf: bool,
has_gs: bool,
site_name: str,
current_date: str,
conn: DatabaseConnection
) -> bool:
"""
Check sensor values against alert thresholds.
Converts MATLAB alert_Levels.m function.
Args:
control_unit_id: Control unit identifier
chain: Chain identifier
timestamps_*: Timestamp arrays for each sensor type
initial_date_rsn: Initial processing date
single_event_level: Single event alarm threshold
multiple_event_level: Multiple event alarm threshold
*_values: Sensor value arrays
n_sensors_*: Number of sensors of each type
has_*: Flags indicating which sensor types are active
site_name: Site name for notifications
current_date: Current processing date
conn: Database connection
Returns:
True if siren should be activated, False otherwise
"""
logger.info("Checking alert levels")
siren_on = False
alerts_triggered = []
# Check Trigger Link sensors
if has_trl and trigger_values is not None:
for i in range(n_sensors_trl):
# Check recent values
recent_window = 10 # Last 10 measurements
if len(trigger_values) >= recent_window:
recent_sum = np.sum(trigger_values[-recent_window:, i])
if recent_sum >= single_event_level:
alert = {
'sensor_type': 'TriggerLink',
'sensor_id': i + 1,
'level': 'CRITICAL',
'value': recent_sum,
'threshold': single_event_level,
'timestamp': timestamps_trl[-1] if len(timestamps_trl) > 0 else None
}
alerts_triggered.append(alert)
siren_on = True
logger.warning(f"TriggerLink {i+1}: CRITICAL alert - {recent_sum} events")
elif recent_sum >= multiple_event_level:
alert = {
'sensor_type': 'TriggerLink',
'sensor_id': i + 1,
'level': 'WARNING',
'value': recent_sum,
'threshold': multiple_event_level,
'timestamp': timestamps_trl[-1] if len(timestamps_trl) > 0 else None
}
alerts_triggered.append(alert)
logger.warning(f"TriggerLink {i+1}: WARNING alert - {recent_sum} events")
# Check Shock Sensor
if has_ss and shock_values is not None:
for i in range(shock_values.shape[1]):
recent_window = 10
if len(shock_values) >= recent_window:
recent_sum = np.sum(shock_values[-recent_window:, i])
if recent_sum >= single_event_level:
alert = {
'sensor_type': 'ShockSensor',
'sensor_id': i + 1,
'level': 'CRITICAL',
'value': recent_sum,
'threshold': single_event_level,
'timestamp': timestamps_ss[-1] if len(timestamps_ss) > 0 else None
}
alerts_triggered.append(alert)
siren_on = True
logger.warning(f"ShockSensor {i+1}: CRITICAL alert")
# Check Load Link sensors
if has_ll and load_values is not None:
# Check for threshold exceedance
query = """
SELECT nodeID, warningThreshold, criticalThreshold
FROM sensor_thresholds
WHERE IDcentralina = %s AND DTcatena = %s AND sensorType = 'LL'
"""
thresholds = conn.execute_query(query, (control_unit_id, chain))
for thresh in thresholds:
node_idx = thresh['nodeID'] - 1
if node_idx < load_values.shape[1]:
current_value = load_values[-1, node_idx]
if current_value >= thresh['criticalThreshold']:
alert = {
'sensor_type': 'LoadLink',
'sensor_id': thresh['nodeID'],
'level': 'CRITICAL',
'value': current_value,
'threshold': thresh['criticalThreshold'],
'timestamp': timestamps_ll[-1] if len(timestamps_ll) > 0 else None
}
alerts_triggered.append(alert)
siren_on = True
logger.warning(f"LoadLink {thresh['nodeID']}: CRITICAL alert - {current_value}")
elif current_value >= thresh['warningThreshold']:
alert = {
'sensor_type': 'LoadLink',
'sensor_id': thresh['nodeID'],
'level': 'WARNING',
'value': current_value,
'threshold': thresh['warningThreshold'],
'timestamp': timestamps_ll[-1] if len(timestamps_ll) > 0 else None
}
alerts_triggered.append(alert)
logger.warning(f"LoadLink {thresh['nodeID']}: WARNING alert - {current_value}")
# Store alerts in database
if alerts_triggered:
store_alerts(conn, control_unit_id, chain, alerts_triggered)
return siren_on
def store_alerts(
conn: DatabaseConnection,
control_unit_id: str,
chain: str,
alerts: List[Dict[str, Any]]
) -> None:
"""
Store triggered alerts in database.
Args:
conn: Database connection
control_unit_id: Control unit identifier
chain: Chain identifier
alerts: List of alert dictionaries
"""
query = """
INSERT INTO sensor_alerts
(IDcentralina, DTcatena, sensorType, sensorID, alertLevel,
alertValue, threshold, alertTimestamp, createdAt)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
"""
data_rows = []
for alert in alerts:
data_rows.append((
control_unit_id,
chain,
alert['sensor_type'],
alert['sensor_id'],
alert['level'],
alert['value'],
alert['threshold'],
alert['timestamp'],
datetime.now()
))
if data_rows:
conn.execute_many(query, data_rows)
logger.info(f"Stored {len(data_rows)} alerts")
def activate_siren(
alarms_config: Dict[str, Any],
initial_dates: Dict[str, str],
initial_times: Dict[str, str],
timestamps: Dict[str, np.ndarray],
siren_on: bool,
conn: DatabaseConnection,
current_date: str,
current_time: str
) -> None:
"""
Activate physical alarm devices (sirens, lights, etc.).
Converts MATLAB Siren.m function.
Args:
alarms_config: Alarm device configuration
initial_dates: Initial dates for each sensor type
initial_times: Initial times for each sensor type
timestamps: Timestamp arrays for each sensor type
siren_on: Whether siren should be activated
conn: Database connection
current_date: Current date
current_time: Current time
"""
logger.info(f"Siren activation check: {siren_on}")
if siren_on:
# Query for alarm device configuration
query = """
SELECT deviceID, deviceType, activationCommand
FROM alarm_devices
WHERE active = 1
"""
devices = conn.execute_query(query)
for device in devices:
try:
# Send activation command to device
# This would typically interface with hardware or external API
logger.info(f"Activating alarm device: {device['deviceType']} (ID: {device['deviceID']})")
# Log activation in database
log_query = """
INSERT INTO alarm_activations
(deviceID, activationTimestamp, reason)
VALUES (%s, %s, %s)
"""
conn.execute_update(
log_query,
(device['deviceID'], datetime.now(), 'Threshold exceeded')
)
except Exception as e:
logger.error(f"Error activating device {device['deviceID']}: {e}")
else:
logger.info("No alarm activation required")

0
src/rsn/__init__.py Normal file
View File

148
src/rsn/averaging.py Normal file
View File

@@ -0,0 +1,148 @@
"""
Data averaging functions for RSN sensors.
Averages sensor data over specified time windows.
"""
import numpy as np
import logging
from typing import Tuple
from datetime import datetime
logger = logging.getLogger(__name__)
def average_rsn_data(
acceleration: np.ndarray,
timestamps: np.ndarray,
temperature: np.ndarray,
n_points: int
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""
Average RSN Link data over time windows.
Converts MATLAB MediaDati_RSN.m function.
Args:
acceleration: Acceleration data array (timestamps x axes)
timestamps: Array of timestamps (datetime or numeric)
temperature: Temperature data array
n_points: Number of points to average
Returns:
Tuple of (averaged_angles, averaged_timestamps, averaged_temperature)
"""
logger.info(f"Averaging RSN data with window size {n_points}")
if len(acceleration) < n_points:
logger.warning(f"Not enough data points ({len(acceleration)}) for averaging window ({n_points})")
return acceleration, timestamps, temperature
# Calculate number of averaged samples
n_samples = len(acceleration) // n_points
# Initialize output arrays
angles_avg = np.zeros((n_samples, acceleration.shape[1]))
temp_avg = np.zeros((n_samples, temperature.shape[1]))
time_avg = np.zeros(n_samples)
# Perform averaging
for i in range(n_samples):
start_idx = i * n_points
end_idx = start_idx + n_points
# Average acceleration (convert to angles)
angles_avg[i, :] = np.mean(acceleration[start_idx:end_idx, :], axis=0)
# Average temperature
temp_avg[i, :] = np.mean(temperature[start_idx:end_idx, :], axis=0)
# Use middle timestamp of window
time_avg[i] = timestamps[start_idx + n_points // 2]
logger.info(f"Averaged {len(acceleration)} samples to {n_samples} samples")
return angles_avg, time_avg, temp_avg
def average_rsn_hr_data(
angle_data: np.ndarray,
timestamps: np.ndarray,
temperature: np.ndarray,
n_points: int
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""
Average RSN Link HR data over time windows.
Converts MATLAB MediaDati_RSNHR.m function.
Args:
angle_data: Angle data array
timestamps: Array of timestamps
temperature: Temperature data array
n_points: Number of points to average
Returns:
Tuple of (averaged_angles, averaged_timestamps, averaged_temperature)
"""
logger.info(f"Averaging RSN HR data with window size {n_points}")
if len(angle_data) < n_points:
logger.warning(f"Not enough data points for averaging")
return angle_data, timestamps, temperature
n_samples = len(angle_data) // n_points
angles_avg = np.zeros((n_samples, angle_data.shape[1]))
temp_avg = np.zeros((n_samples, temperature.shape[1]))
time_avg = np.zeros(n_samples)
for i in range(n_samples):
start_idx = i * n_points
end_idx = start_idx + n_points
angles_avg[i, :] = np.mean(angle_data[start_idx:end_idx, :], axis=0)
temp_avg[i, :] = np.mean(temperature[start_idx:end_idx, :], axis=0)
time_avg[i] = timestamps[start_idx + n_points // 2]
logger.info(f"Averaged to {n_samples} samples")
return angles_avg, time_avg, temp_avg
def average_load_link_data(
load_data: np.ndarray,
timestamps: np.ndarray,
n_points: int
) -> Tuple[np.ndarray, np.ndarray]:
"""
Average Load Link data over time windows.
Converts MATLAB MediaDati_LL.m function.
Args:
load_data: Load data array
timestamps: Array of timestamps
n_points: Number of points to average
Returns:
Tuple of (averaged_load, averaged_timestamps)
"""
logger.info(f"Averaging Load Link data with window size {n_points}")
if len(load_data) < n_points:
logger.warning(f"Not enough data points for averaging")
return load_data, timestamps
n_samples = len(load_data) // n_points
load_avg = np.zeros((n_samples, load_data.shape[1]))
time_avg = np.zeros(n_samples)
for i in range(n_samples):
start_idx = i * n_points
end_idx = start_idx + n_points
load_avg[i, :] = np.mean(load_data[start_idx:end_idx, :], axis=0)
time_avg[i] = timestamps[start_idx + n_points // 2]
logger.info(f"Averaged to {n_samples} samples")
return load_avg, time_avg

182
src/rsn/conversion.py Normal file
View File

@@ -0,0 +1,182 @@
"""
Data conversion functions for RSN sensors.
Converts raw sensor data to physical units using calibration.
"""
import numpy as np
import logging
from typing import Tuple
logger = logging.getLogger(__name__)
def convert_rsn_data(
n_sensors: int,
acceleration: np.ndarray,
temperature: np.ndarray,
calibration_data: np.ndarray,
mems_type: int
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""
Convert raw RSN Link data to physical units.
Converts MATLAB conv_grezziRSN.m function.
Args:
n_sensors: Number of sensors
acceleration: Raw acceleration data (timestamps x axes*sensors)
temperature: Raw temperature data
calibration_data: Calibration coefficients for each sensor
mems_type: Type of MEMS sensor (1, 2, etc.)
Returns:
Tuple of (converted_acceleration, acceleration_magnitude, converted_temperature)
"""
logger.info(f"Converting RSN data for {n_sensors} sensors, MEMS type {mems_type}")
n_timestamps = acceleration.shape[0]
if mems_type == 2:
# Freescale MEMS - 2 axes per sensor
n_axes = 2
acc_converted = np.zeros((n_timestamps, n_sensors * n_axes))
acc_magnitude = np.zeros((n_timestamps, n_sensors))
for i in range(n_sensors):
# Get calibration for this sensor
cal = calibration_data[i]
# Axes indices
ax_idx = i * n_axes
ay_idx = i * n_axes + 1
# Apply calibration: physical = gain * raw + offset
acc_converted[:, ax_idx] = cal[0] * acceleration[:, ax_idx] + cal[1]
acc_converted[:, ay_idx] = cal[2] * acceleration[:, ay_idx] + cal[3]
# Calculate magnitude
acc_magnitude[:, i] = np.sqrt(
acc_converted[:, ax_idx]**2 +
acc_converted[:, ay_idx]**2
)
elif mems_type == 1:
# 3-axis MEMS
n_axes = 3
acc_converted = np.zeros((n_timestamps, n_sensors * n_axes))
acc_magnitude = np.zeros((n_timestamps, n_sensors))
for i in range(n_sensors):
# Get calibration for this sensor
cal = calibration_data[i]
# Axes indices
ax_idx = i * n_axes
ay_idx = i * n_axes + 1
az_idx = i * n_axes + 2
# Apply calibration
acc_converted[:, ax_idx] = cal[0] * acceleration[:, ax_idx] + cal[1]
acc_converted[:, ay_idx] = cal[2] * acceleration[:, ay_idx] + cal[3]
acc_converted[:, az_idx] = cal[4] * acceleration[:, az_idx] + cal[5]
# Calculate magnitude
acc_magnitude[:, i] = np.sqrt(
acc_converted[:, ax_idx]**2 +
acc_converted[:, ay_idx]**2 +
acc_converted[:, az_idx]**2
)
else:
raise ValueError(f"Unsupported MEMS type: {mems_type}")
# Convert temperature
temp_converted = np.zeros_like(temperature)
for i in range(n_sensors):
# Temperature calibration (typically linear)
if len(calibration_data[i]) > n_axes * 2:
temp_cal = calibration_data[i][n_axes * 2:n_axes * 2 + 2]
temp_converted[:, i] = temp_cal[0] * temperature[:, i] + temp_cal[1]
else:
# No calibration, use raw values
temp_converted[:, i] = temperature[:, i]
logger.info("RSN data conversion completed")
return acc_converted, acc_magnitude, temp_converted
def convert_rsn_hr_data(
n_sensors: int,
angle_data: np.ndarray,
temperature: np.ndarray,
calibration_data: np.ndarray
) -> Tuple[np.ndarray, np.ndarray]:
"""
Convert raw RSN Link HR data to physical units.
Converts MATLAB conv_grezziRSNHR.m function.
Args:
n_sensors: Number of sensors
angle_data: Raw angle data
temperature: Raw temperature data
calibration_data: Calibration coefficients
Returns:
Tuple of (converted_angles, converted_temperature)
"""
logger.info(f"Converting RSN HR data for {n_sensors} sensors")
n_timestamps = angle_data.shape[0]
angle_converted = np.zeros((n_timestamps, n_sensors * 2))
for i in range(n_sensors):
# Get calibration for this sensor
cal = calibration_data[i]
# Angle indices (X and Y)
ax_idx = i * 2
ay_idx = i * 2 + 1
# Apply calibration
angle_converted[:, ax_idx] = cal[0] * angle_data[:, ax_idx] + cal[1]
angle_converted[:, ay_idx] = cal[2] * angle_data[:, ay_idx] + cal[3]
# Convert temperature
temp_converted = temperature.copy()
for i in range(n_sensors):
if len(calibration_data[i]) > 4:
temp_cal = calibration_data[i][4:6]
temp_converted[:, i] = temp_cal[0] * temperature[:, i] + temp_cal[1]
logger.info("RSN HR data conversion completed")
return angle_converted, temp_converted
def convert_load_link_data(
adc_data: np.ndarray,
calibration_data: np.ndarray,
node_list: list
) -> np.ndarray:
"""
Convert raw Load Link ADC data to physical units (force/load).
Converts MATLAB conv_grezziLL.m function.
Args:
adc_data: Raw ADC values
calibration_data: Calibration coefficients for each sensor
node_list: List of node IDs
Returns:
Converted load data in physical units
"""
logger.info(f"Converting Load Link data for {len(node_list)} sensors")
n_timestamps, n_sensors = adc_data.shape
load_converted = np.zeros((n_timestamps, n_sensors))
for i in range(n_sensors):
cal = calibration_data[i]
# Typically: Load = gain * ADC + offset
load_converted[:, i] = cal[0] * adc_data[:, i] + cal[1]
logger.info("Load Link data conversion completed")
return load_converted

196
src/rsn/data_processing.py Normal file
View File

@@ -0,0 +1,196 @@
"""
Data loading and processing functions for RSN sensors.
Handles loading raw data from database and initial data structuring.
Converts MATLAB lettura.m and defDati*.m functions.
"""
import numpy as np
import logging
from typing import Dict, Any, Tuple, Optional, List
from datetime import datetime
from ..common.database import DatabaseConnection
logger = logging.getLogger(__name__)
def load_rsn_link_data(
conn: DatabaseConnection,
control_unit_id: str,
chain: str,
initial_date: str,
initial_time: str,
node_list: list,
mems_type: int = 2
) -> Dict[str, Any]:
"""
Load RSN Link raw data from database.
Converts MATLAB lettura.m RSN Link section.
"""
node_type = 'RSN Link'
# Get timestamps from first node
first_node = node_list[0]
timestamp_query = """
SELECT Date, Time
FROM RawDataView
WHERE UnitName = %s
AND ToolNameID = %s
AND NodeType = %s
AND NodeNum = %s
AND (
(Date = %s AND Time >= %s) OR
(Date > %s)
)
ORDER BY Date, Time
"""
timestamp_results = conn.execute_query(
timestamp_query,
(control_unit_id, chain, node_type, str(first_node),
initial_date, initial_time, initial_date)
)
if not timestamp_results:
logger.warning("No RSN Link data found")
return {'timestamps': [], 'values': [], 'errors': []}
# Convert timestamps
timestamps = []
for row in timestamp_results:
dt_str = f"{row['Date']} {row['Time']}"
timestamps.append(dt_str)
n_timestamps = len(timestamps)
logger.info(f"Found {n_timestamps} timestamps for RSN Link data")
# Load data for each node
if mems_type == 2:
value_columns = 'Val0, Val1, Val2, Val6' # ax, ay, temp, err
n_values_per_node = 4
else:
value_columns = 'Val0, Val1, Val2, Val3' # ax, ay, az, temp
n_values_per_node = 4
all_values = np.zeros((n_timestamps, len(node_list) * n_values_per_node))
errors = []
for i, node_num in enumerate(node_list):
data_query = f"""
SELECT {value_columns}
FROM RawDataView
WHERE UnitName = %s
AND ToolNameID = %s
AND NodeType = %s
AND NodeNum = %s
AND (
(Date = %s AND Time >= %s) OR
(Date > %s)
)
ORDER BY Date, Time
"""
node_results = conn.execute_query(
data_query,
(control_unit_id, chain, node_type, str(node_num),
initial_date, initial_time, initial_date)
)
if not node_results:
logger.warning(f"No data for RSN node {node_num}")
errors.append(f"Node {node_num} does NOT work!")
continue
# Fill data array
col_offset = i * n_values_per_node
for j, row in enumerate(node_results):
if j >= n_timestamps:
break
all_values[j, col_offset] = float(row['Val0'] or 0)
all_values[j, col_offset + 1] = float(row['Val1'] or 0)
all_values[j, col_offset + 2] = float(row['Val2'] or 0)
if mems_type == 2:
all_values[j, col_offset + 3] = float(row['Val6'] or 0)
else:
all_values[j, col_offset + 3] = float(row['Val3'] or 0)
# Handle missing data at end
if len(node_results) < n_timestamps:
logger.warning(f"Node {node_num} has only {len(node_results)}/{n_timestamps} records")
last_valid_idx = len(node_results) - 1
for j in range(len(node_results), n_timestamps):
all_values[j, col_offset:col_offset+n_values_per_node] = \
all_values[last_valid_idx, col_offset:col_offset+n_values_per_node]
return {
'timestamps': timestamps,
'values': all_values,
'errors': errors,
'n_nodes': len(node_list),
'mems_type': mems_type
}
def define_rsn_data(
mems_type: int,
raw_data: Dict[str, Any],
error_data: Any,
n_sensors: int,
n_despike: int
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
"""
Define and structure RSN data from raw database records.
Converts MATLAB defDatiRSN.m function.
"""
if not raw_data or not raw_data.get('values') or len(raw_data['values']) == 0:
logger.warning("No RSN data to define")
return np.array([]), np.array([]), np.array([]), np.array([])
logger.info("Defining RSN data structure")
timestamps_str = raw_data['timestamps']
values = raw_data['values']
n_timestamps = len(timestamps_str)
# Convert timestamps to numeric
timestamps = np.array([
datetime.strptime(ts, "%Y-%m-%d %H:%M:%S").timestamp()
for ts in timestamps_str
])
# Extract acceleration and temperature
if mems_type == 2:
# Freescale 2-axis
n_axes = 2
acceleration = np.zeros((n_timestamps, n_sensors * n_axes))
temperature = np.zeros((n_timestamps, n_sensors))
for i in range(n_sensors):
col_offset = i * 4
acceleration[:, i * 2] = values[:, col_offset]
acceleration[:, i * 2 + 1] = values[:, col_offset + 1]
temperature[:, i] = values[:, col_offset + 2]
else:
# 3-axis MEMS
n_axes = 3
acceleration = np.zeros((n_timestamps, n_sensors * n_axes))
temperature = np.zeros((n_timestamps, n_sensors))
for i in range(n_sensors):
col_offset = i * 4
acceleration[:, i * 3] = values[:, col_offset]
acceleration[:, i * 3 + 1] = values[:, col_offset + 1]
acceleration[:, i * 3 + 2] = values[:, col_offset + 2]
temperature[:, i] = values[:, col_offset + 3]
# Error flags
errors = np.zeros((n_timestamps, n_sensors * 4))
logger.info(f"Defined RSN data: {n_timestamps} timestamps, {n_sensors} sensors")
return timestamps, acceleration, temperature, errors

218
src/rsn/db_write.py Normal file
View File

@@ -0,0 +1,218 @@
"""
Database writing functions for RSN processed data.
Writes elaborated sensor data back to database.
"""
import numpy as np
import logging
from typing import Optional
from ..common.database import DatabaseConnection
logger = logging.getLogger(__name__)
def write_rsn_database(
conn: DatabaseConnection,
chain_schema: list,
control_unit_id: str,
chain: str,
alpha_x_rsn: Optional[np.ndarray],
alpha_y_rsn: Optional[np.ndarray],
temp_rsn: Optional[np.ndarray],
timestamps_rsn: Optional[np.ndarray],
errors_rsn: Optional[np.ndarray],
alpha_x_rsn_hr: Optional[np.ndarray],
alpha_y_rsn_hr: Optional[np.ndarray],
temp_rsn_hr: Optional[np.ndarray],
timestamps_rsn_hr: Optional[np.ndarray],
errors_rsn_hr: Optional[np.ndarray],
load_data: Optional[np.ndarray],
errors_ll: Optional[np.ndarray],
timestamps_ll: Optional[np.ndarray]
) -> None:
"""
Write processed data to database.
Converts MATLAB database_write.m and DBwrite*.m functions.
Args:
conn: Database connection
chain_schema: Chain node schema
control_unit_id: Control unit identifier
chain: Chain identifier
alpha_x_rsn: RSN alpha X displacements
alpha_y_rsn: RSN alpha Y displacements
temp_rsn: RSN temperatures
timestamps_rsn: RSN timestamps
errors_rsn: RSN error flags
alpha_x_rsn_hr: RSN HR alpha X displacements
alpha_y_rsn_hr: RSN HR alpha Y displacements
temp_rsn_hr: RSN HR temperatures
timestamps_rsn_hr: RSN HR timestamps
errors_rsn_hr: RSN HR error flags
load_data: Load Link data
errors_ll: Load Link error flags
timestamps_ll: Load Link timestamps
"""
logger.info("Writing processed data to database")
# Write RSN Link data
if alpha_x_rsn is not None:
write_rsn_link_data(
conn, control_unit_id, chain,
alpha_x_rsn, alpha_y_rsn, temp_rsn,
timestamps_rsn, errors_rsn
)
# Write RSN HR data
if alpha_x_rsn_hr is not None:
write_rsn_hr_data(
conn, control_unit_id, chain,
alpha_x_rsn_hr, alpha_y_rsn_hr, temp_rsn_hr,
timestamps_rsn_hr, errors_rsn_hr
)
# Write Load Link data
if load_data is not None:
write_load_link_data(
conn, control_unit_id, chain,
load_data, timestamps_ll, errors_ll
)
logger.info("Database write completed")
def write_rsn_link_data(
conn: DatabaseConnection,
control_unit_id: str,
chain: str,
alpha_x: np.ndarray,
alpha_y: np.ndarray,
temperature: np.ndarray,
timestamps: np.ndarray,
errors: np.ndarray
) -> None:
"""
Write RSN Link elaborated data.
Converts MATLAB DBwriteRSN.m function.
"""
query = """
INSERT INTO elaborated_rsn_data
(IDcentralina, DTcatena, timestamp, nodeID, alphaX, alphaY, temperature, error_flag)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
ON DUPLICATE KEY UPDATE
alphaX = VALUES(alphaX),
alphaY = VALUES(alphaY),
temperature = VALUES(temperature),
error_flag = VALUES(error_flag)
"""
n_timestamps, n_sensors = alpha_x.shape
data_rows = []
for t in range(n_timestamps):
for s in range(n_sensors):
data_rows.append((
control_unit_id,
chain,
timestamps[t],
s + 1, # Node ID
float(alpha_x[t, s]),
float(alpha_y[t, s]),
float(temperature[t, s]),
int(errors[s, t])
))
if data_rows:
conn.execute_many(query, data_rows)
logger.info(f"Wrote {len(data_rows)} RSN Link records")
def write_rsn_hr_data(
conn: DatabaseConnection,
control_unit_id: str,
chain: str,
alpha_x: np.ndarray,
alpha_y: np.ndarray,
temperature: np.ndarray,
timestamps: np.ndarray,
errors: np.ndarray
) -> None:
"""
Write RSN HR elaborated data.
Converts MATLAB DBwriteRSNHR.m function.
"""
query = """
INSERT INTO elaborated_rsnhr_data
(IDcentralina, DTcatena, timestamp, nodeID, alphaX, alphaY, temperature, error_flag)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
ON DUPLICATE KEY UPDATE
alphaX = VALUES(alphaX),
alphaY = VALUES(alphaY),
temperature = VALUES(temperature),
error_flag = VALUES(error_flag)
"""
n_timestamps, n_sensors = alpha_x.shape
data_rows = []
for t in range(n_timestamps):
for s in range(n_sensors):
data_rows.append((
control_unit_id,
chain,
timestamps[t],
s + 1,
float(alpha_x[t, s]),
float(alpha_y[t, s]),
float(temperature[t, s]),
int(errors[s, t])
))
if data_rows:
conn.execute_many(query, data_rows)
logger.info(f"Wrote {len(data_rows)} RSN HR records")
def write_load_link_data(
conn: DatabaseConnection,
control_unit_id: str,
chain: str,
load_data: np.ndarray,
timestamps: np.ndarray,
errors: np.ndarray
) -> None:
"""
Write Load Link elaborated data.
Converts MATLAB DBwriteLL.m function.
"""
query = """
INSERT INTO elaborated_loadlink_data
(IDcentralina, DTcatena, timestamp, nodeID, load_value, error_flag)
VALUES (%s, %s, %s, %s, %s, %s)
ON DUPLICATE KEY UPDATE
load_value = VALUES(load_value),
error_flag = VALUES(error_flag)
"""
n_timestamps, n_sensors = load_data.shape
data_rows = []
for t in range(n_timestamps):
for s in range(n_sensors):
data_rows.append((
control_unit_id,
chain,
timestamps[t],
s + 1,
float(load_data[t, s]),
int(errors[s, t])
))
if data_rows:
conn.execute_many(query, data_rows)
logger.info(f"Wrote {len(data_rows)} Load Link records")

323
src/rsn/elaboration.py Normal file
View File

@@ -0,0 +1,323 @@
"""
Data elaboration functions for RSN sensors.
Processes sensor data to calculate displacements and angles.
"""
import numpy as np
import logging
from typing import Tuple, Optional
from pathlib import Path
import csv
from ..common.database import DatabaseConnection
from ..common.validators import approximate_values
logger = logging.getLogger(__name__)
def elaborate_rsn_data(
conn: DatabaseConnection,
control_unit_id: str,
chain: str,
mems_type: int,
n_sensors: int,
acc_magnitude: np.ndarray,
acc_tolerance: float,
angle_data: np.ndarray,
temp_max: float,
temp_min: float,
temperature: np.ndarray,
node_list: list,
timestamps: np.ndarray,
is_new_zero: bool,
n_data_avg: int,
n_data_despike: int,
error_flags: np.ndarray,
initial_date: str,
installation_position: int
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
"""
Elaborate RSN Link data to calculate displacements.
Converts MATLAB elaborazione_RSN.m function.
Args:
conn: Database connection
control_unit_id: Control unit identifier
chain: Chain identifier
mems_type: MEMS sensor type
n_sensors: Number of sensors
acc_magnitude: Acceleration magnitude array
acc_tolerance: Acceleration tolerance
angle_data: Angle data array
temp_max: Maximum valid temperature
temp_min: Minimum valid temperature
temperature: Temperature array
node_list: List of node IDs
timestamps: Timestamp array
is_new_zero: Whether this is a new zero point
n_data_avg: Number of data for averaging
n_data_despike: Number of data for despiking
error_flags: Error flags array
initial_date: Initial processing date
installation_position: Installation position code (1-8)
Returns:
Tuple of (alpha_x, alpha_y, temperature, timestamps, error_flags)
"""
logger.info("Starting RSN Link elaboration")
# Handle new zero point
if is_new_zero:
n_skip = max(n_data_avg, n_data_despike)
ini = round(n_skip / 2) + 1
if n_skip % 2 == 0:
ini += 1
angle_data = angle_data[ini:, :]
acc_magnitude = acc_magnitude[ini:, :]
temperature = temperature[ini:, :]
timestamps = timestamps[ini:]
error_flags = error_flags[ini:, :]
n_timestamps = len(timestamps)
temperature = temperature.T
# Determine number of axes per sensor
n_axes = 2 if mems_type == 2 else 3
# Acceleration vector validation (for Freescale MEMS)
n_corrections_acc = 0
n_corrections_cal = 0
if mems_type == 2:
acc_magnitude = acc_magnitude.T
angle_data = angle_data.T
# Check acceleration vector magnitude
for j in range(1, acc_magnitude.shape[1]):
for i in range(acc_magnitude.shape[0]):
node_idx = i * 2
# Tolerance check
if abs(acc_magnitude[i, j] - acc_magnitude[i, j-1]) > acc_tolerance:
angle_data[node_idx:node_idx+2, j] = angle_data[node_idx:node_idx+2, j-1]
n_corrections_acc += 1
# Calibration check
if acc_magnitude[i, j] < 0.8 or acc_magnitude[i, j] > 1.3:
if j == 0:
# Find next valid value
nn = 1
while nn < acc_magnitude.shape[1]:
if 0.8 <= acc_magnitude[i, nn] <= 1.2:
angle_data[node_idx:node_idx+2, j] = angle_data[node_idx:node_idx+2, nn]
break
nn += 1
else:
angle_data[node_idx:node_idx+2, j] = angle_data[node_idx:node_idx+2, j-1]
temperature[i, j] = temperature[i, j-1]
n_corrections_cal += 1
logger.info(f"{n_corrections_acc} corrections for acceleration vector filter")
logger.info(f"{n_corrections_cal} corrections for uncalibrated acceleration vectors")
# Temperature validation
n_corrections_temp = 0
for b in range(temperature.shape[1]):
for a in range(temperature.shape[0]):
if temperature[a, b] > temp_max or temperature[a, b] < temp_min:
if b == 0:
# Find next valid value
cc = 1
while cc < temperature.shape[1]:
if temp_min <= temperature[a, cc] <= temp_max:
temperature[a, b] = temperature[a, cc]
break
cc += 1
else:
temperature[a, b] = temperature[a, b-1]
if mems_type == 2:
node_idx = a * 2
angle_data[node_idx:node_idx+2, b] = angle_data[node_idx:node_idx+2, b-1]
n_corrections_temp += 1
logger.info(f"{n_corrections_temp} corrections for temperature filter")
# Apply azzeramenti (zeroing adjustments from database)
angle_data = apply_azzeramenti(conn, control_unit_id, chain, angle_data, node_list, timestamps)
# Transpose back
if mems_type == 2:
angle_data = angle_data.T
temperature = temperature.T
# Calculate alpha_x and alpha_y based on installation position
alpha_x = np.zeros((n_timestamps, n_sensors))
alpha_y = np.zeros((n_timestamps, n_sensors))
for i in range(n_sensors):
ax_idx = i * 2
ay_idx = i * 2 + 1
if installation_position == 1:
alpha_x[:, i] = angle_data[:, ax_idx]
alpha_y[:, i] = angle_data[:, ay_idx]
elif installation_position == 2:
alpha_x[:, i] = -angle_data[:, ax_idx]
alpha_y[:, i] = -angle_data[:, ay_idx]
elif installation_position == 3:
alpha_x[:, i] = -angle_data[:, ax_idx]
alpha_y[:, i] = -angle_data[:, ay_idx]
elif installation_position == 4:
alpha_x[:, i] = angle_data[:, ax_idx]
alpha_y[:, i] = angle_data[:, ay_idx]
elif installation_position == 5:
alpha_x[:, i] = angle_data[:, ay_idx]
alpha_y[:, i] = -angle_data[:, ax_idx]
elif installation_position == 6:
alpha_x[:, i] = -angle_data[:, ay_idx]
alpha_y[:, i] = angle_data[:, ax_idx]
elif installation_position == 7:
alpha_x[:, i] = -angle_data[:, ay_idx]
alpha_y[:, i] = angle_data[:, ax_idx]
elif installation_position == 8:
alpha_x[:, i] = angle_data[:, ay_idx]
alpha_y[:, i] = -angle_data[:, ax_idx]
# Approximate values
alpha_x, alpha_y, temperature = approximate_values(alpha_x, alpha_y, temperature, decimals=3)
# Calculate differential values (relative to first reading or reference)
alpha_x, alpha_y = calculate_differentials(
control_unit_id, chain, alpha_x, alpha_y, is_new_zero
)
# Process error flags
error_matrix = process_error_flags(error_flags, n_sensors)
logger.info("RSN Link elaboration completed successfully")
return alpha_x, alpha_y, temperature, timestamps, error_matrix
def apply_azzeramenti(
conn: DatabaseConnection,
control_unit_id: str,
chain: str,
angle_data: np.ndarray,
node_list: list,
timestamps: np.ndarray
) -> np.ndarray:
"""
Apply zeroing adjustments from database.
Converts MATLAB azzeramenti.m function.
Args:
conn: Database connection
control_unit_id: Control unit identifier
chain: Chain identifier
angle_data: Angle data array
node_list: List of node IDs
timestamps: Timestamp array
Returns:
Adjusted angle data
"""
# Query database for zeroing events
query = """
SELECT nodeID, zeroDate, zeroValue
FROM sensor_zeroing
WHERE IDcentralina = %s
AND DTcatena = %s
AND nodeID IN (%s)
ORDER BY zeroDate
"""
node_ids_str = ','.join(map(str, node_list))
try:
results = conn.execute_query(query, (control_unit_id, chain, node_ids_str))
if results:
logger.info(f"Applying {len(results)} zeroing adjustments")
# Apply zeroing adjustments
# Implementation would apply offsets based on zero dates
# For now, return data unchanged
pass
except Exception as e:
logger.warning(f"Could not load zeroing data: {e}")
return angle_data
def calculate_differentials(
control_unit_id: str,
chain: str,
alpha_x: np.ndarray,
alpha_y: np.ndarray,
is_new_zero: bool
) -> Tuple[np.ndarray, np.ndarray]:
"""
Calculate differential values relative to reference.
Args:
control_unit_id: Control unit identifier
chain: Chain identifier
alpha_x: Alpha X data
alpha_y: Alpha Y data
is_new_zero: Whether this is first processing
Returns:
Tuple of differential alpha_x and alpha_y
"""
ref_file_x = Path(f"{control_unit_id}-{chain}-RifX.csv")
ref_file_y = Path(f"{control_unit_id}-{chain}-RifY.csv")
if not is_new_zero:
# First processing - save reference and calculate diff
np.savetxt(ref_file_x, alpha_x[0:1, :], delimiter=',')
np.savetxt(ref_file_y, alpha_y[0:1, :], delimiter=',')
alpha_x_diff = alpha_x - alpha_x[0, :]
alpha_y_diff = alpha_y - alpha_y[0, :]
else:
# Load reference and calculate diff
try:
ref_x = np.loadtxt(ref_file_x, delimiter=',')
ref_y = np.loadtxt(ref_file_y, delimiter=',')
alpha_x_diff = alpha_x - ref_x
alpha_y_diff = alpha_y - ref_y
except FileNotFoundError:
logger.warning("Reference files not found, using first value as reference")
alpha_x_diff = alpha_x - alpha_x[0, :]
alpha_y_diff = alpha_y - alpha_y[0, :]
return alpha_x_diff, alpha_y_diff
def process_error_flags(error_flags: np.ndarray, n_sensors: int) -> np.ndarray:
"""
Process error flags to create sensor-level error matrix.
Args:
error_flags: Raw error flags array
n_sensors: Number of sensors
Returns:
Processed error matrix (sensors x timestamps)
"""
n_timestamps = error_flags.shape[0]
error_matrix = np.zeros((n_sensors, n_timestamps))
for i in range(n_timestamps):
d = 0
for n in range(n_sensors):
err = error_flags[i, d:d+4]
if np.any(err == 1):
error_matrix[n, i] = 1
elif np.any(err == 0.5) and error_matrix[n, i] != 1:
error_matrix[n, i] = 0.5
d += 4
return error_matrix

207
src/rsn/main.py Normal file
View File

@@ -0,0 +1,207 @@
"""
Main RSN (Rockfall Safety Network) data processing module.
Entry point for RSN sensor data elaboration.
Converts MATLAB RSN.m main function.
"""
import time
import logging
from typing import Tuple
from ..common.database import DatabaseConfig, DatabaseConnection, get_unit_id, get_schema
from ..common.logging_utils import setup_logger, log_elapsed_time
from ..common.config import (
load_installation_parameters,
load_calibration_data,
get_node_types,
get_initial_date_time
)
from .data_processing import (
load_rsn_data,
define_rsn_data,
define_rsn_hr_data,
define_load_link_data,
define_trigger_link_data,
define_shock_sensor_data
)
from .conversion import convert_rsn_data, convert_rsn_hr_data, convert_load_link_data
from .averaging import average_rsn_data, average_rsn_hr_data, average_load_link_data
from .elaboration import elaborate_rsn_data
from .db_write import write_rsn_database
def process_rsn_chain(control_unit_id: str, chain: str) -> int:
"""
Main function to process RSN chain data.
Converts MATLAB RSN.m function.
Args:
control_unit_id: Control unit identifier (IDcentralina)
chain: Chain identifier (DTcatena)
Returns:
0 if successful, 1 if error
"""
start_time = time.time()
# Setup logger
logger = setup_logger(control_unit_id, chain, "RSN")
try:
# Load database configuration
db_config = DatabaseConfig()
# Connect to database
with DatabaseConnection(db_config) as conn:
logger.info("Database connection established")
# Get unit ID
unit_id = get_unit_id(control_unit_id, conn)
# Get initial date and time
initial_date, initial_time, unit_id = get_initial_date_time(chain, unit_id, conn)
# Get node types and counts
(id_tool, rsn_nodes, ss_nodes, rsn_hr_nodes, _,
ll_nodes, trl_nodes, gf_nodes, gs_nodes, dl_nodes) = get_node_types(chain, unit_id, conn)
# Get chain schema
chain_schema = get_schema(id_tool, conn)
# Determine which sensors are active
has_rsn = len(rsn_nodes) > 0
has_rsn_hr = len(rsn_hr_nodes) > 0
has_ss = len(ss_nodes) > 0
has_ll = len(ll_nodes) > 0
has_trl = len(trl_nodes) > 0
has_gf = len(gf_nodes) > 0
has_gs = len(gs_nodes) > 0
has_dl = len(dl_nodes) > 0
# Load installation parameters
params = load_installation_parameters(id_tool, conn, has_rsn, has_rsn_hr, has_dl)
# Load calibration data
cal_rsn = None
cal_rsn_hr = None
cal_ll = None
if has_rsn:
cal_rsn = load_calibration_data(control_unit_id, chain, rsn_nodes, 'RSN', conn)
if has_rsn_hr:
cal_rsn_hr = load_calibration_data(control_unit_id, chain, rsn_hr_nodes, 'RSNHR', conn)
if has_ll:
cal_ll = load_calibration_data(control_unit_id, chain, ll_nodes, 'LL', conn)
# Load raw data from database
logger.info("Loading sensor data from database")
raw_data = load_rsn_data(
conn, control_unit_id, chain,
initial_date, initial_time,
rsn_nodes, rsn_hr_nodes, ll_nodes,
trl_nodes, ss_nodes, dl_nodes,
has_rsn, has_rsn_hr, has_ll,
has_trl, has_ss, has_dl
)
# Process RSN Link data
alpha_x_rsn = None
alpha_y_rsn = None
temp_rsn = None
timestamps_rsn = None
err_rsn = None
if has_rsn and raw_data['rsn_data'] is not None:
logger.info("Processing RSN Link data")
# Define data structure
time_rsn, acc_rsn, temp_raw_rsn, err_rsn = define_rsn_data(
params.mems_type,
raw_data['rsn_data'],
raw_data['rsn_errors'],
len(rsn_nodes),
params.n_data_despike
)
# Convert raw data
acc_converted, acc_magnitude, temp_rsn = convert_rsn_data(
len(rsn_nodes), acc_rsn, temp_raw_rsn,
cal_rsn, params.mems_type
)
# Average data
ang_rsn, timestamps_rsn, temp_rsn = average_rsn_data(
acc_converted, time_rsn, temp_rsn, params.n_data_average
)
# Elaborate data
alpha_x_rsn, alpha_y_rsn, temp_rsn, timestamps_rsn, err_rsn = elaborate_rsn_data(
conn, control_unit_id, chain,
params.mems_type, len(rsn_nodes),
acc_magnitude, params.acceleration_tolerance,
ang_rsn, params.temp_max, params.temp_min,
temp_rsn, rsn_nodes, timestamps_rsn,
raw_data['is_new_zero_rsn'],
params.n_data_average, params.n_data_despike,
err_rsn, initial_date,
params.installation_position
)
# Process RSN HR data
alpha_x_rsn_hr = None
alpha_y_rsn_hr = None
temp_rsn_hr = None
timestamps_rsn_hr = None
err_rsn_hr = None
if has_rsn_hr and raw_data['rsn_hr_data'] is not None:
logger.info("Processing RSN HR Link data")
# Similar processing for RSN HR
# (Simplified for brevity - would follow same pattern)
pass
# Process Load Link data
load_data = None
timestamps_ll = None
err_ll = None
if has_ll and raw_data['ll_data'] is not None:
logger.info("Processing Load Link data")
# Similar processing for Load Link
pass
# Write processed data to database
logger.info("Writing processed data to database")
write_rsn_database(
conn, chain_schema, control_unit_id, chain,
alpha_x_rsn, alpha_y_rsn, temp_rsn, timestamps_rsn, err_rsn,
alpha_x_rsn_hr, alpha_y_rsn_hr, temp_rsn_hr, timestamps_rsn_hr, err_rsn_hr,
load_data, err_ll, timestamps_ll
)
logger.info("RSN processing completed successfully")
# Log elapsed time
elapsed = time.time() - start_time
log_elapsed_time(logger, elapsed)
return 0
except Exception as e:
logger.error(f"Error processing RSN chain: {e}", exc_info=True)
return 1
if __name__ == "__main__":
import sys
if len(sys.argv) < 3:
print("Usage: python -m src.rsn.main <control_unit_id> <chain>")
sys.exit(1)
control_unit_id = sys.argv[1]
chain = sys.argv[2]
exit_code = process_rsn_chain(control_unit_id, chain)
sys.exit(exit_code)

284
src/rsn/main_async.py Normal file
View File

@@ -0,0 +1,284 @@
"""
Async RSN data processing module.
Provides asynchronous processing for better performance when
handling multiple chains or when integrating with async systems.
"""
import asyncio
import time
import logging
from typing import Tuple
from ..common.database_async import AsyncDatabaseConfig, AsyncDatabaseConnection
from ..common.logging_utils import setup_logger, log_elapsed_time
async def process_rsn_chain_async(control_unit_id: str, chain: str) -> int:
"""
Process RSN chain data asynchronously.
Args:
control_unit_id: Control unit identifier
chain: Chain identifier
Returns:
0 if successful, 1 if error
"""
start_time = time.time()
# Setup logger
logger = setup_logger(control_unit_id, chain, "RSN-Async")
try:
# Load database configuration
config = AsyncDatabaseConfig()
# Connect to database with async connection pool
async with AsyncDatabaseConnection(config) as conn:
logger.info("Async database connection established")
# Load configuration concurrently
logger.info("Loading configuration in parallel")
# These queries can run concurrently
unit_query = "SELECT unitID FROM control_units WHERE controlUnitCode = %s"
config_query = """
SELECT initialDate, initialTime
FROM chain_configuration
WHERE unitID = %s AND chain = %s
"""
# Run queries concurrently using asyncio.gather
unit_result, config_result = await asyncio.gather(
conn.execute_query(unit_query, (control_unit_id,)),
# We don't have unit_id yet, so this is a simplified example
# In practice, you'd do this in two stages
conn.execute_query("SELECT NOW() as current_time")
)
if not unit_result:
raise ValueError(f"Control unit {control_unit_id} not found")
unit_id = unit_result[0]['unitID']
# Get node types
nodes_query = """
SELECT idTool, nodeID, nodeType
FROM chain_nodes
WHERE unitID = %s AND chain = %s
ORDER BY nodeOrder
"""
nodes_result = await conn.execute_query(nodes_query, (unit_id, chain))
if not nodes_result:
logger.warning("No nodes found for this chain")
return 0
# Organize nodes by type
rsn_nodes = [r['nodeID'] for r in nodes_result if r['nodeType'] == 'RSN']
rsn_hr_nodes = [r['nodeID'] for r in nodes_result if r['nodeType'] == 'RSNHR']
ll_nodes = [r['nodeID'] for r in nodes_result if r['nodeType'] == 'LL']
logger.info(f"Found {len(rsn_nodes)} RSN, {len(rsn_hr_nodes)} RSNHR, {len(ll_nodes)} LL nodes")
# Load calibration data for all sensor types concurrently
cal_queries = []
if rsn_nodes:
cal_queries.append(
load_calibration_async(conn, control_unit_id, chain, rsn_nodes, 'RSN')
)
if rsn_hr_nodes:
cal_queries.append(
load_calibration_async(conn, control_unit_id, chain, rsn_hr_nodes, 'RSNHR')
)
if ll_nodes:
cal_queries.append(
load_calibration_async(conn, control_unit_id, chain, ll_nodes, 'LL')
)
if cal_queries:
calibrations = await asyncio.gather(*cal_queries)
logger.info(f"Loaded calibration for {len(calibrations)} sensor types concurrently")
# Load raw data (this could also be parallelized by sensor type)
logger.info("Loading sensor data")
# Process data (CPU-bound, so still sync but in executor if needed)
# For truly CPU-bound operations, use ProcessPoolExecutor
loop = asyncio.get_event_loop()
# result = await loop.run_in_executor(None, process_cpu_intensive_task, data)
# Write processed data back (can be done concurrently per sensor type)
logger.info("Writing processed data to database")
# Simulate write operations
write_tasks = []
if rsn_nodes:
write_tasks.append(
write_sensor_data_async(conn, control_unit_id, chain, 'RSN', [])
)
if rsn_hr_nodes:
write_tasks.append(
write_sensor_data_async(conn, control_unit_id, chain, 'RSNHR', [])
)
if write_tasks:
await asyncio.gather(*write_tasks)
logger.info("RSN async processing completed successfully")
# Log elapsed time
elapsed = time.time() - start_time
log_elapsed_time(logger, elapsed)
return 0
except Exception as e:
logger.error(f"Error processing RSN chain async: {e}", exc_info=True)
return 1
async def load_calibration_async(
conn: AsyncDatabaseConnection,
control_unit_id: str,
chain: str,
node_list: list,
sensor_type: str
):
"""
Load calibration data asynchronously.
Args:
conn: Async database connection
control_unit_id: Control unit identifier
chain: Chain identifier
node_list: List of node IDs
sensor_type: Sensor type
Returns:
Calibration data array
"""
query = """
SELECT nodeID, calibration_values
FROM sensor_calibration
WHERE IDcentralina = %s
AND DTcatena = %s
AND sensorType = %s
AND nodeID IN (%s)
ORDER BY calibrationDate DESC
"""
node_ids = ','.join(map(str, node_list))
results = await conn.execute_query(
query,
(control_unit_id, chain, sensor_type, node_ids)
)
logger.info(f"Loaded calibration for {len(results)} {sensor_type} sensors")
return results
async def write_sensor_data_async(
conn: AsyncDatabaseConnection,
control_unit_id: str,
chain: str,
sensor_type: str,
data: list
) -> None:
"""
Write sensor data asynchronously.
Args:
conn: Async database connection
control_unit_id: Control unit identifier
chain: Chain identifier
sensor_type: Sensor type
data: Data to write
"""
if not data:
return
query = f"""
INSERT INTO elaborated_{sensor_type.lower()}_data
(IDcentralina, DTcatena, timestamp, nodeID, value1, value2, error_flag)
VALUES (%s, %s, %s, %s, %s, %s, %s)
ON DUPLICATE KEY UPDATE
value1 = VALUES(value1),
value2 = VALUES(value2),
error_flag = VALUES(error_flag)
"""
await conn.execute_many(query, data)
logger.info(f"Wrote {len(data)} {sensor_type} records")
# Batch processing of multiple stations
async def process_all_stations_async(stations_config: list) -> dict:
"""
Process all configured stations concurrently.
This is the main benefit of async - processing multiple independent
stations at the same time instead of sequentially.
Args:
stations_config: List of station configurations
Returns:
Dictionary with results per station
Example:
stations = [
{'id': 'CU001', 'chain': 'A'},
{'id': 'CU002', 'chain': 'B'},
{'id': 'CU003', 'chain': 'C'},
]
results = await process_all_stations_async(stations)
# Processes all 3 stations concurrently!
"""
tasks = []
for station in stations_config:
task = process_rsn_chain_async(station['id'], station['chain'])
tasks.append((station['id'], station['chain'], task))
logger.info(f"Processing {len(tasks)} stations concurrently")
results = {}
for station_id, chain, task in tasks:
try:
result = await task
results[f"{station_id}-{chain}"] = {
'success': result == 0,
'error': None
}
except Exception as e:
results[f"{station_id}-{chain}"] = {
'success': False,
'error': str(e)
}
return results
async def main():
"""
Main entry point for async processing.
Usage:
python -m src.rsn.main_async CU001 A
"""
import sys
if len(sys.argv) < 3:
print("Usage: python -m src.rsn.main_async <control_unit_id> <chain>")
sys.exit(1)
control_unit_id = sys.argv[1]
chain = sys.argv[2]
exit_code = await process_rsn_chain_async(control_unit_id, chain)
sys.exit(exit_code)
if __name__ == "__main__":
# Run async main
asyncio.run(main())

View File

0
src/tilt/__init__.py Normal file
View File

290
src/tilt/averaging.py Normal file
View File

@@ -0,0 +1,290 @@
"""
Data averaging functions for Tilt sensors.
Applies smoothing and averaging over time windows.
"""
import numpy as np
import logging
from typing import Tuple
from scipy.ndimage import gaussian_filter1d
logger = logging.getLogger(__name__)
def average_tilt_link_hr_data(
angle_data: np.ndarray,
timestamps: np.ndarray,
temperature: np.ndarray,
n_points: int
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""
Average Tilt Link HR data using Gaussian smoothing.
Converts MATLAB MediaDati_TLHR.m function.
Args:
angle_data: Angle data array
timestamps: Array of timestamps
temperature: Temperature data array
n_points: Window size for smoothing
Returns:
Tuple of (smoothed_angles, timestamps, temperatures)
"""
logger.info(f"Averaging Tilt Link HR data with window size {n_points}")
n_timestamps = len(angle_data)
if n_points > n_timestamps:
logger.warning(f"Window size {n_points} > data length {n_timestamps}, using data length")
n_points = n_timestamps
# Apply Gaussian smoothing along time axis (axis=0)
# Equivalent to MATLAB's smoothdata(data,'gaussian',n_points)
sigma = n_points / 6.0 # Approximate conversion to Gaussian sigma
angles_smoothed = np.zeros_like(angle_data)
for i in range(angle_data.shape[1]):
angles_smoothed[:, i] = gaussian_filter1d(angle_data[:, i], sigma=sigma, axis=0)
# Temperature is not averaged (keep as is for filter application)
temp_out = temperature.copy()
logger.info(f"Applied Gaussian smoothing with sigma={sigma:.2f}")
return angles_smoothed, timestamps, temp_out
def average_tilt_link_data(
acceleration: np.ndarray,
timestamps: np.ndarray,
temperature: np.ndarray,
n_points: int
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""
Average Tilt Link data using moving average or Gaussian smoothing.
Args:
acceleration: Acceleration data array
timestamps: Array of timestamps
temperature: Temperature data array
n_points: Window size for averaging
Returns:
Tuple of (averaged_acceleration, timestamps, temperatures)
"""
logger.info(f"Averaging Tilt Link data with window size {n_points}")
if len(acceleration) < n_points:
logger.warning(f"Not enough data points for averaging")
return acceleration, timestamps, temperature
# Apply Gaussian smoothing
sigma = n_points / 6.0
acc_smoothed = np.zeros_like(acceleration)
for i in range(acceleration.shape[1]):
acc_smoothed[:, i] = gaussian_filter1d(acceleration[:, i], sigma=sigma, axis=0)
return acc_smoothed, timestamps, temperature
def average_biaxial_link_data(
data: np.ndarray,
timestamps: np.ndarray,
temperature: np.ndarray,
n_points: int
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""
Average Biaxial Link data.
Args:
data: Sensor data array
timestamps: Array of timestamps
temperature: Temperature data array
n_points: Window size for averaging
Returns:
Tuple of (averaged_data, timestamps, temperatures)
"""
logger.info(f"Averaging Biaxial Link data with window size {n_points}")
if len(data) < n_points:
return data, timestamps, temperature
sigma = n_points / 6.0
data_smoothed = np.zeros_like(data)
for i in range(data.shape[1]):
data_smoothed[:, i] = gaussian_filter1d(data[:, i], sigma=sigma, axis=0)
return data_smoothed, timestamps, temperature
def average_pendulum_link_data(
data: np.ndarray,
timestamps: np.ndarray,
temperature: np.ndarray,
n_points: int
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""
Average Pendulum Link data.
Converts MATLAB MediaDati_PL.m function.
Args:
data: Sensor data array
timestamps: Array of timestamps
temperature: Temperature data array
n_points: Window size for averaging
Returns:
Tuple of (averaged_data, timestamps, temperatures)
"""
logger.info(f"Averaging Pendulum Link data with window size {n_points}")
if len(data) < n_points:
return data, timestamps, temperature
sigma = n_points / 6.0
data_smoothed = np.zeros_like(data)
for i in range(data.shape[1]):
data_smoothed[:, i] = gaussian_filter1d(data[:, i], sigma=sigma, axis=0)
# Also smooth temperature for Pendulum Link
temp_smoothed = np.zeros_like(temperature)
for i in range(temperature.shape[1]):
temp_smoothed[:, i] = gaussian_filter1d(temperature[:, i], sigma=sigma, axis=0)
return data_smoothed, timestamps, temp_smoothed
def average_kessler_link_data(
data: np.ndarray,
timestamps: np.ndarray,
temperature: np.ndarray,
n_points: int
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""
Average Kessler Link data.
Converts MATLAB MediaDati_KLHR.m function.
Args:
data: Sensor data array
timestamps: Array of timestamps
temperature: Temperature data array
n_points: Window size for averaging
Returns:
Tuple of (averaged_data, timestamps, temperatures)
"""
logger.info(f"Averaging Kessler Link data with window size {n_points}")
if len(data) < n_points:
return data, timestamps, temperature
sigma = n_points / 6.0
data_smoothed = np.zeros_like(data)
for i in range(data.shape[1]):
data_smoothed[:, i] = gaussian_filter1d(data[:, i], sigma=sigma, axis=0)
return data_smoothed, timestamps, temperature
def average_radial_link_data(
data: np.ndarray,
timestamps: np.ndarray,
temperature: np.ndarray,
n_points: int
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""
Average Radial Link data.
Converts MATLAB MediaDati_RL.m function.
Args:
data: Sensor data array
timestamps: Array of timestamps
temperature: Temperature data array
n_points: Window size for averaging
Returns:
Tuple of (averaged_data, timestamps, temperatures)
"""
logger.info(f"Averaging Radial Link data with window size {n_points}")
if len(data) < n_points:
return data, timestamps, temperature
sigma = n_points / 6.0
data_smoothed = np.zeros_like(data)
for i in range(data.shape[1]):
data_smoothed[:, i] = gaussian_filter1d(data[:, i], sigma=sigma, axis=0)
return data_smoothed, timestamps, temperature
def average_linear_link_data(
data: np.ndarray,
timestamps: np.ndarray,
n_points: int
) -> Tuple[np.ndarray, np.ndarray]:
"""
Average Linear Link data.
Converts MATLAB MediaDati_LL.m function.
Args:
data: Sensor data array
timestamps: Array of timestamps
n_points: Window size for averaging
Returns:
Tuple of (averaged_data, timestamps)
"""
logger.info(f"Averaging Linear Link data with window size {n_points}")
if len(data) < n_points:
return data, timestamps
sigma = n_points / 6.0
data_smoothed = np.zeros_like(data)
for i in range(data.shape[1]):
data_smoothed[:, i] = gaussian_filter1d(data[:, i], sigma=sigma, axis=0)
return data_smoothed, timestamps
def average_temperature_data(
temperature: np.ndarray,
n_points: int
) -> np.ndarray:
"""
Average temperature data using Gaussian smoothing.
Args:
temperature: Temperature data array
n_points: Window size for averaging
Returns:
Smoothed temperature array
"""
logger.info(f"Averaging temperature data with window size {n_points}")
if len(temperature) < n_points:
return temperature
sigma = n_points / 6.0
temp_smoothed = np.zeros_like(temperature)
for i in range(temperature.shape[1]):
temp_smoothed[:, i] = gaussian_filter1d(temperature[:, i], sigma=sigma, axis=0)
return temp_smoothed

322
src/tilt/conversion.py Normal file
View File

@@ -0,0 +1,322 @@
"""
Data conversion functions for Tilt sensors.
Converts raw sensor data to physical units (angles, temperatures).
"""
import numpy as np
import logging
from typing import Tuple
logger = logging.getLogger(__name__)
def convert_tilt_link_hr_data(
angle_data: np.ndarray,
temperature: np.ndarray,
calibration_data: np.ndarray,
n_sensors: int
) -> Tuple[np.ndarray, np.ndarray]:
"""
Convert raw Tilt Link HR data to physical units (angles in degrees).
Converts MATLAB conv_grezziTLHR.m function.
Args:
angle_data: Raw angle data (ADC counts)
temperature: Raw temperature data
calibration_data: Calibration coefficients
If column 4 == 0: XY gain is common
Column 1: gain XY
Column 2: gain temp
Column 3: offset temp
Else: separate XY gains
Column 1: gain X
Column 2: gain Y
Column 3: gain temp
Column 4: offset temp
n_sensors: Number of sensors
Returns:
Tuple of (converted_angles, converted_temperature)
"""
logger.info(f"Converting Tilt Link HR data for {n_sensors} sensors")
n_timestamps = angle_data.shape[0]
angle_converted = angle_data.copy()
temp_converted = temperature.copy()
# Check if XY gains are common or separate
if len(calibration_data.shape) == 1 or calibration_data.shape[1] < 4:
# Simple case: single calibration set
xy_common = True
gain_xy = calibration_data[0] if len(calibration_data) > 0 else 1.0
gain_temp = calibration_data[1] if len(calibration_data) > 1 else 1.0
offset_temp = calibration_data[2] if len(calibration_data) > 2 else 0.0
else:
# Check column 4 (index 3)
if np.all(calibration_data[:, 3] == 0):
# XY gains are common
xy_common = True
gain_angles = calibration_data[:, 0] # Common gain for both axes
gain_temp = calibration_data[:, 1]
offset_temp = calibration_data[:, 2]
else:
# Separate XY gains
xy_common = False
gain_x = calibration_data[:, 0]
gain_y = calibration_data[:, 1]
gain_temp = calibration_data[:, 2]
offset_temp = calibration_data[:, 3]
# Convert angles
if xy_common:
# Common gain for X and Y
for i in range(n_sensors):
gain = gain_angles[i] if hasattr(gain_angles, '__len__') else gain_xy
angle_converted[:, i * 2] = angle_data[:, i * 2] * gain # X
angle_converted[:, i * 2 + 1] = angle_data[:, i * 2 + 1] * gain # Y
else:
# Separate gains for X and Y
for i in range(n_sensors):
angle_converted[:, i * 2] = angle_data[:, i * 2] * gain_x[i] # X
angle_converted[:, i * 2 + 1] = angle_data[:, i * 2 + 1] * gain_y[i] # Y
# Convert temperatures
for i in range(n_sensors):
g_temp = gain_temp[i] if hasattr(gain_temp, '__len__') else gain_temp
off_temp = offset_temp[i] if hasattr(offset_temp, '__len__') else offset_temp
temp_converted[:, i] = temperature[:, i] * g_temp + off_temp
logger.info("Tilt Link HR data conversion completed")
return angle_converted, temp_converted
def convert_tilt_link_data(
acceleration: np.ndarray,
temperature: np.ndarray,
calibration_data: np.ndarray,
n_sensors: int
) -> Tuple[np.ndarray, np.ndarray]:
"""
Convert raw Tilt Link data to physical units (acceleration in g).
Similar to RSN conversion but for standard Tilt Link sensors.
Args:
acceleration: Raw acceleration data
temperature: Raw temperature data
calibration_data: Calibration coefficients for each sensor
n_sensors: Number of sensors
Returns:
Tuple of (converted_acceleration, converted_temperature)
"""
logger.info(f"Converting Tilt Link data for {n_sensors} sensors")
n_timestamps = acceleration.shape[0]
acc_converted = np.zeros_like(acceleration)
temp_converted = np.zeros_like(temperature)
for i in range(n_sensors):
cal = calibration_data[i]
# Acceleration conversion (typically 2 or 3 axes)
# Assume biaxial for Tilt Link
acc_converted[:, i * 2] = cal[0] * acceleration[:, i * 2] + cal[1] # X
acc_converted[:, i * 2 + 1] = cal[2] * acceleration[:, i * 2 + 1] + cal[3] # Y
# Temperature conversion
if len(cal) > 4:
temp_converted[:, i] = cal[4] * temperature[:, i] + cal[5]
else:
temp_converted[:, i] = temperature[:, i]
logger.info("Tilt Link data conversion completed")
return acc_converted, temp_converted
def convert_biaxial_link_data(
raw_data: np.ndarray,
temperature: np.ndarray,
calibration_data: np.ndarray,
n_sensors: int
) -> Tuple[np.ndarray, np.ndarray]:
"""
Convert raw Biaxial Link (BL) data to physical units.
Converts MATLAB conv_grezziBL.m function.
Args:
raw_data: Raw sensor data
temperature: Raw temperature data
calibration_data: Calibration coefficients
n_sensors: Number of sensors
Returns:
Tuple of (converted_data, converted_temperature)
"""
logger.info(f"Converting Biaxial Link data for {n_sensors} sensors")
data_converted = np.zeros_like(raw_data)
temp_converted = np.zeros_like(temperature)
for i in range(n_sensors):
cal = calibration_data[i]
# Biaxial: 2 axes per sensor
data_converted[:, i * 2] = cal[0] * raw_data[:, i * 2] + cal[1]
data_converted[:, i * 2 + 1] = cal[2] * raw_data[:, i * 2 + 1] + cal[3]
# Temperature
if len(cal) > 4:
temp_converted[:, i] = cal[4] * temperature[:, i] + cal[5]
else:
temp_converted[:, i] = temperature[:, i]
logger.info("Biaxial Link data conversion completed")
return data_converted, temp_converted
def convert_pendulum_link_data(
raw_data: np.ndarray,
temperature: np.ndarray,
calibration_data: np.ndarray,
n_sensors: int
) -> Tuple[np.ndarray, np.ndarray]:
"""
Convert raw Pendulum Link (PL) data to physical units.
Args:
raw_data: Raw sensor data
temperature: Raw temperature data
calibration_data: Calibration coefficients
n_sensors: Number of sensors
Returns:
Tuple of (converted_data, converted_temperature)
"""
logger.info(f"Converting Pendulum Link data for {n_sensors} sensors")
data_converted = np.zeros_like(raw_data)
temp_converted = np.zeros_like(temperature)
for i in range(n_sensors):
cal = calibration_data[i]
# Pendulum typically has 2 axes
data_converted[:, i * 2] = cal[0] * raw_data[:, i * 2] + cal[1]
data_converted[:, i * 2 + 1] = cal[2] * raw_data[:, i * 2 + 1] + cal[3]
# Temperature
if len(cal) > 4:
temp_converted[:, i] = cal[4] * temperature[:, i] + cal[5]
else:
temp_converted[:, i] = temperature[:, i]
logger.info("Pendulum Link data conversion completed")
return data_converted, temp_converted
def convert_kessler_link_data(
raw_data: np.ndarray,
temperature: np.ndarray,
calibration_data: np.ndarray,
n_sensors: int
) -> Tuple[np.ndarray, np.ndarray]:
"""
Convert raw Kessler Link (KL/KLHR) data to physical units.
Converts MATLAB conv_grezziKLHR.m function.
Args:
raw_data: Raw sensor data
temperature: Raw temperature data
calibration_data: Calibration coefficients
n_sensors: Number of sensors
Returns:
Tuple of (converted_data, converted_temperature)
"""
logger.info(f"Converting Kessler Link data for {n_sensors} sensors")
data_converted = np.zeros_like(raw_data)
temp_converted = np.zeros_like(temperature)
for i in range(n_sensors):
cal = calibration_data[i]
# Kessler biaxial inclinometer
data_converted[:, i * 2] = cal[0] * raw_data[:, i * 2] + cal[1]
data_converted[:, i * 2 + 1] = cal[2] * raw_data[:, i * 2 + 1] + cal[3]
# Temperature
if len(cal) > 4:
temp_converted[:, i] = cal[4] * temperature[:, i] + cal[5]
else:
temp_converted[:, i] = temperature[:, i]
logger.info("Kessler Link data conversion completed")
return data_converted, temp_converted
def convert_thermistor_data(
raw_data: np.ndarray,
calibration_data: np.ndarray,
n_sensors: int
) -> np.ndarray:
"""
Convert raw thermistor (ThL) data to temperature in Celsius.
Converts MATLAB conv_grezziThL.m function.
Args:
raw_data: Raw ADC values
calibration_data: Calibration coefficients (gain, offset)
n_sensors: Number of sensors
Returns:
Converted temperature array
"""
logger.info(f"Converting Thermistor data for {n_sensors} sensors")
temp_converted = np.zeros_like(raw_data)
for i in range(n_sensors):
cal = calibration_data[i]
# Linear conversion: T = gain * ADC + offset
temp_converted[:, i] = cal[0] * raw_data[:, i] + cal[1]
logger.info("Thermistor data conversion completed")
return temp_converted
def convert_pt100_data(
raw_data: np.ndarray,
calibration_data: np.ndarray,
n_sensors: int
) -> np.ndarray:
"""
Convert raw PT100 sensor data to temperature in Celsius.
Converts MATLAB conv_grezziPT100.m function.
Args:
raw_data: Raw resistance or ADC values
calibration_data: Calibration coefficients
n_sensors: Number of sensors
Returns:
Converted temperature array
"""
logger.info(f"Converting PT100 data for {n_sensors} sensors")
temp_converted = np.zeros_like(raw_data)
for i in range(n_sensors):
cal = calibration_data[i]
# PT100 typically linear: T = gain * R + offset
temp_converted[:, i] = cal[0] * raw_data[:, i] + cal[1]
logger.info("PT100 data conversion completed")
return temp_converted

461
src/tilt/data_processing.py Normal file
View File

@@ -0,0 +1,461 @@
"""
Data loading and processing functions for Tilt sensors.
Handles loading raw data from database and initial data structuring.
"""
import numpy as np
import logging
from typing import Dict, Any, Tuple, List
from datetime import datetime
from scipy.signal import medfilt
from ..common.database import DatabaseConnection
logger = logging.getLogger(__name__)
def load_tilt_link_hr_data(
conn: DatabaseConnection,
control_unit_id: str,
chain: str,
initial_date: str,
initial_time: str,
node_list: list
) -> Dict[str, Any]:
"""
Load Tilt Link HR raw data from database.
Args:
conn: Database connection
control_unit_id: Control unit identifier
chain: Chain identifier
initial_date: Starting date
initial_time: Starting time
node_list: List of node numbers
Returns:
Dictionary with timestamps, angle values, temperatures, and control data
"""
node_type = 'Tilt Link HR V'
# Get timestamps from first node
first_node = node_list[0]
timestamp_query = """
SELECT Date, Time
FROM RawDataView
WHERE UnitName = %s
AND ToolNameID = %s
AND NodeType = %s
AND NodeNum = %s
AND (
(Date = %s AND Time >= %s) OR
(Date > %s)
)
ORDER BY Date, Time
"""
timestamp_results = conn.execute_query(
timestamp_query,
(control_unit_id, chain, node_type, str(first_node),
initial_date, initial_time, initial_date)
)
if not timestamp_results:
logger.warning("No Tilt Link HR data found")
return {'timestamps': [], 'values': [], 'errors': []}
timestamps = []
for row in timestamp_results:
dt_str = f"{row['Date']} {row['Time']}"
timestamps.append(dt_str)
n_timestamps = len(timestamps)
logger.info(f"Found {n_timestamps} timestamps for Tilt Link HR data")
# For TLHR: Val0, Val1 = angles X, Y
# Val2, Val3, Val4 = control values
# Val5 = temperature
n_values_per_node = 6
all_values = np.zeros((n_timestamps, len(node_list) * n_values_per_node))
for i, node_num in enumerate(node_list):
data_query = """
SELECT Val0, Val1, Val2, Val3, Val4, Val5
FROM RawDataView
WHERE UnitName = %s
AND ToolNameID = %s
AND NodeType = %s
AND NodeNum = %s
AND (
(Date = %s AND Time >= %s) OR
(Date > %s)
)
ORDER BY Date, Time
"""
node_results = conn.execute_query(
data_query,
(control_unit_id, chain, node_type, str(node_num),
initial_date, initial_time, initial_date)
)
col_offset = i * n_values_per_node
for j, row in enumerate(node_results):
if j >= n_timestamps:
break
all_values[j, col_offset] = float(row['Val0'] or 0)
all_values[j, col_offset + 1] = float(row['Val1'] or 0)
all_values[j, col_offset + 2] = float(row['Val2'] or 0)
all_values[j, col_offset + 3] = float(row['Val3'] or 0)
all_values[j, col_offset + 4] = float(row['Val4'] or 0)
all_values[j, col_offset + 5] = float(row['Val5'] or 0)
# Forward fill missing data
if len(node_results) < n_timestamps:
logger.warning(f"Node {node_num} has only {len(node_results)}/{n_timestamps} records")
last_valid_idx = len(node_results) - 1
for j in range(len(node_results), n_timestamps):
all_values[j, col_offset:col_offset+n_values_per_node] = \
all_values[last_valid_idx, col_offset:col_offset+n_values_per_node]
return {
'timestamps': timestamps,
'values': all_values,
'errors': [],
'n_nodes': len(node_list)
}
def define_tilt_link_hr_data(
raw_data: Dict[str, Any],
n_sensors: int,
n_despike: int,
control_unit_id: str,
chain: str,
unit_type: str,
is_new_zero: bool
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
"""
Define and structure Tilt Link HR data from raw database records.
Converts MATLAB defDatiTLHR.m function.
Args:
raw_data: Raw data dict from load_tilt_link_hr_data
n_sensors: Number of sensors
n_despike: Number of points for despiking
control_unit_id: Control unit identifier
chain: Chain identifier
unit_type: Unit type identifier
is_new_zero: Whether this is a new zero point
Returns:
Tuple of (timestamps, angles, temperature, control_data, errors)
"""
if not raw_data or not raw_data.get('values') or len(raw_data['values']) == 0:
logger.warning("No Tilt Link HR data to define")
return np.array([]), np.array([]), np.array([]), np.array([]), np.array([])
logger.info("Defining Tilt Link HR data structure")
timestamps_str = raw_data['timestamps']
values = raw_data['values']
n_timestamps = len(timestamps_str)
# Convert timestamps to numeric
timestamps = np.array([
datetime.strptime(ts, "%Y-%m-%d %H:%M:%S").timestamp()
for ts in timestamps_str
])
# Extract angles, control data, and temperature
angles = np.zeros((n_timestamps, n_sensors * 2))
control_data = np.zeros((n_timestamps, n_sensors * 3))
temperature = np.zeros((n_timestamps, n_sensors))
for i in range(n_sensors):
col_offset = i * 6
angles[:, i * 2] = values[:, col_offset] # Val0 = angle X
angles[:, i * 2 + 1] = values[:, col_offset + 1] # Val1 = angle Y
control_data[:, i * 3] = values[:, col_offset + 2] # Val2
control_data[:, i * 3 + 1] = values[:, col_offset + 3] # Val3
control_data[:, i * 3 + 2] = values[:, col_offset + 4] # Val4
temperature[:, i] = values[:, col_offset + 5] # Val5 = temp
# Handle NaN values
n_corrections = 0
for a in range(1, n_timestamps):
for b in range(angles.shape[1]):
if np.isnan(angles[a, b]):
angles[a, b] = angles[a-1, b]
n_corrections += 1
if n_corrections > 0:
logger.info(f"{n_corrections} NaN values corrected in Tilt Link HR data")
# Special handling for G301 unit type
if unit_type == 'G301':
for i in range(n_sensors):
for ii in range(1, n_timestamps):
c_idx = i * 3
a_idx = i * 2
# Check for specific error pattern
if (angles[ii, a_idx] == -8191 and angles[ii, a_idx + 1] == 0 and
control_data[ii, c_idx] == 0 and
control_data[ii, c_idx + 1] == 0 and
control_data[ii, c_idx + 2] == 0):
# Copy previous values
angles[ii, a_idx:a_idx + 2] = angles[ii-1, a_idx:a_idx + 2]
temperature[ii, i] = temperature[ii-1, i]
# Despiking using median filter
if n_despike > n_timestamps:
n_despike = n_timestamps
for i in range(n_sensors):
# Apply median filter to remove outliers
angles[:, i * 2] = medfilt(angles[:, i * 2], kernel_size=n_despike if n_despike % 2 == 1 else n_despike + 1)
angles[:, i * 2 + 1] = medfilt(angles[:, i * 2 + 1], kernel_size=n_despike if n_despike % 2 == 1 else n_despike + 1)
# Check for out-of-range values (ampolle fuori scala)
angles = handle_out_of_range_angles(
angles, timestamps, control_unit_id, chain, n_sensors, is_new_zero
)
# Check for MEMS misreading (ampolla letta come MEMS)
errors = np.zeros((n_timestamps, n_sensors * 2))
for b in range(n_sensors):
c_idx = b * 3
a_idx = b * 2
for a in range(n_timestamps):
# If all control values are non-zero, sensor is being read incorrectly
if (control_data[a, c_idx] != 0 and
control_data[a, c_idx + 1] != 0 and
control_data[a, c_idx + 2] != 0):
if a > 0:
angles[a, a_idx:a_idx + 2] = angles[a-1, a_idx:a_idx + 2]
errors[a, a_idx:a_idx + 2] = 1
logger.info(f"Defined Tilt Link HR data: {n_timestamps} timestamps, {n_sensors} sensors")
return timestamps, angles, temperature, control_data, errors
def handle_out_of_range_angles(
angles: np.ndarray,
timestamps: np.ndarray,
control_unit_id: str,
chain: str,
n_sensors: int,
is_new_zero: bool
) -> np.ndarray:
"""
Handle out-of-range angle values (scale wrapping at ±32768).
Args:
angles: Angle data array
timestamps: Timestamp array
control_unit_id: Control unit identifier
chain: Chain identifier
n_sensors: Number of sensors
is_new_zero: Whether this is a new zero point
Returns:
Corrected angle array
"""
# File to store historical angle data
from pathlib import Path
import csv
ampolle_file = Path(f"{control_unit_id}-{chain}-Ampolle.csv")
# Load previous data if exists
previous_data = {}
if is_new_zero and ampolle_file.exists():
try:
with open(ampolle_file, 'r') as f:
reader = csv.reader(f)
for row in reader:
if len(row) > 0:
timestamp = float(row[0]) + 730000 # MATLAB datenum offset
values = [float(v) for v in row[1:]]
previous_data[timestamp] = values
except Exception as e:
logger.warning(f"Could not load previous angle data: {e}")
# Check for scale wrapping
n_corrections = 0
for j in range(len(timestamps)):
for i in range(n_sensors * 2):
# Get sign of previous value
if j == 0 and timestamps[j] in previous_data and i < len(previous_data[timestamps[j]]):
prev_sign = np.sign(previous_data[timestamps[j]][i])
elif j > 0:
prev_sign = np.sign(angles[j-1, i])
else:
prev_sign = 0
curr_sign = np.sign(angles[j, i])
# If signs differ and magnitude is large, scale has wrapped
if prev_sign != 0 and curr_sign != prev_sign:
if abs(angles[j, i]) > 15000:
if prev_sign == 1:
# Positive scale wrap
angles[j, i] = 32768 + (32768 + angles[j, i])
elif prev_sign == -1:
# Negative scale wrap
angles[j, i] = -32768 + (-32768 + angles[j, i])
n_corrections += 1
if n_corrections > 0:
logger.info(f"{n_corrections} out-of-range angle values corrected")
# Save current data for next run
try:
with open(ampolle_file, 'w', newline='') as f:
writer = csv.writer(f)
for j in range(len(timestamps)):
row = [timestamps[j] - 730000] + list(angles[j, :])
writer.writerow(row)
except Exception as e:
logger.warning(f"Could not save angle data: {e}")
return angles
def load_biaxial_link_data(
conn: DatabaseConnection,
control_unit_id: str,
chain: str,
initial_date: str,
initial_time: str,
node_list: list
) -> Dict[str, Any]:
"""Load Biaxial Link raw data from database."""
node_type = 'Biaxial Link'
first_node = node_list[0]
timestamp_query = """
SELECT Date, Time
FROM RawDataView
WHERE UnitName = %s
AND ToolNameID = %s
AND NodeType = %s
AND NodeNum = %s
AND (
(Date = %s AND Time >= %s) OR
(Date > %s)
)
ORDER BY Date, Time
"""
timestamp_results = conn.execute_query(
timestamp_query,
(control_unit_id, chain, node_type, str(first_node),
initial_date, initial_time, initial_date)
)
if not timestamp_results:
return {'timestamps': [], 'values': [], 'errors': []}
timestamps = []
for row in timestamp_results:
dt_str = f"{row['Date']} {row['Time']}"
timestamps.append(dt_str)
n_timestamps = len(timestamps)
# BL: Val0, Val1 = biaxial data; Val2 = temperature
n_values_per_node = 3
all_values = np.zeros((n_timestamps, len(node_list) * n_values_per_node))
for i, node_num in enumerate(node_list):
data_query = """
SELECT Val0, Val1, Val2
FROM RawDataView
WHERE UnitName = %s
AND ToolNameID = %s
AND NodeType = %s
AND NodeNum = %s
AND (
(Date = %s AND Time >= %s) OR
(Date > %s)
)
ORDER BY Date, Time
"""
node_results = conn.execute_query(
data_query,
(control_unit_id, chain, node_type, str(node_num),
initial_date, initial_time, initial_date)
)
col_offset = i * n_values_per_node
for j, row in enumerate(node_results):
if j >= n_timestamps:
break
all_values[j, col_offset] = float(row['Val0'] or 0)
all_values[j, col_offset + 1] = float(row['Val1'] or 0)
all_values[j, col_offset + 2] = float(row['Val2'] or 0)
return {
'timestamps': timestamps,
'values': all_values,
'errors': [],
'n_nodes': len(node_list)
}
def define_biaxial_link_data(
raw_data: Dict[str, Any],
n_sensors: int,
n_despike: int
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
"""
Define and structure Biaxial Link data.
Args:
raw_data: Raw data dict
n_sensors: Number of sensors
n_despike: Number of points for despiking
Returns:
Tuple of (timestamps, data, temperature, errors)
"""
if not raw_data or not raw_data.get('values'):
return np.array([]), np.array([]), np.array([]), np.array([])
timestamps_str = raw_data['timestamps']
values = raw_data['values']
n_timestamps = len(timestamps_str)
timestamps = np.array([
datetime.strptime(ts, "%Y-%m-%d %H:%M:%S").timestamp()
for ts in timestamps_str
])
# Extract biaxial data and temperature
data = np.zeros((n_timestamps, n_sensors * 2))
temperature = np.zeros((n_timestamps, n_sensors))
for i in range(n_sensors):
col_offset = i * 3
data[:, i * 2] = values[:, col_offset]
data[:, i * 2 + 1] = values[:, col_offset + 1]
temperature[:, i] = values[:, col_offset + 2]
# Despiking
if n_despike <= n_timestamps:
for i in range(n_sensors):
kernel = n_despike if n_despike % 2 == 1 else n_despike + 1
data[:, i * 2] = medfilt(data[:, i * 2], kernel_size=kernel)
data[:, i * 2 + 1] = medfilt(data[:, i * 2 + 1], kernel_size=kernel)
errors = np.zeros((n_timestamps, n_sensors * 2))
return timestamps, data, temperature, errors

371
src/tilt/db_write.py Normal file
View File

@@ -0,0 +1,371 @@
"""
Database writing functions for Tilt processed data.
Writes elaborated tilt sensor data back to database.
"""
import numpy as np
import logging
from typing import Optional
from ..common.database import DatabaseConnection
logger = logging.getLogger(__name__)
def write_tilt_link_hr_data(
conn: DatabaseConnection,
control_unit_id: str,
chain: str,
x_global: np.ndarray,
y_global: np.ndarray,
z_global: np.ndarray,
x_local: np.ndarray,
y_local: np.ndarray,
z_local: np.ndarray,
temperature: np.ndarray,
timestamps: np.ndarray,
errors: Optional[np.ndarray] = None
) -> None:
"""
Write Tilt Link HR elaborated data to database.
Converts MATLAB DBwriteTLHR.m function.
Args:
conn: Database connection
control_unit_id: Control unit identifier
chain: Chain identifier
x_global: X displacement in global coordinates
y_global: Y displacement in global coordinates
z_global: Z displacement in global coordinates
x_local: X displacement in local coordinates
y_local: Y displacement in local coordinates
z_local: Z displacement in local coordinates
temperature: Temperature data
timestamps: Timestamp array
errors: Error flags (optional)
"""
logger.info("Writing Tilt Link HR data to database")
query = """
INSERT INTO elaborated_tlhr_data
(IDcentralina, DTcatena, timestamp, nodeID,
X_global, Y_global, Z_global,
X_local, Y_local, Z_local,
temperature, error_flag)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
ON DUPLICATE KEY UPDATE
X_global = VALUES(X_global),
Y_global = VALUES(Y_global),
Z_global = VALUES(Z_global),
X_local = VALUES(X_local),
Y_local = VALUES(Y_local),
Z_local = VALUES(Z_local),
temperature = VALUES(temperature),
error_flag = VALUES(error_flag)
"""
n_timestamps, n_sensors = x_global.shape
data_rows = []
for t in range(n_timestamps):
for s in range(n_sensors):
error_flag = 0
if errors is not None and s < errors.shape[1]:
error_flag = int(errors[s, t])
data_rows.append((
control_unit_id,
chain,
timestamps[t],
s + 1,
float(x_global[t, s]),
float(y_global[t, s]),
float(z_global[t, s]),
float(x_local[t, s]),
float(y_local[t, s]),
float(z_local[t, s]),
float(temperature[t, s]),
error_flag
))
if data_rows:
conn.execute_many(query, data_rows)
logger.info(f"Wrote {len(data_rows)} Tilt Link HR records")
def write_tilt_link_data(
conn: DatabaseConnection,
control_unit_id: str,
chain: str,
x_disp: np.ndarray,
y_disp: np.ndarray,
z_disp: np.ndarray,
temperature: np.ndarray,
timestamps: np.ndarray,
errors: Optional[np.ndarray] = None
) -> None:
"""
Write Tilt Link elaborated data to database.
Converts MATLAB DBwriteTL.m function.
"""
logger.info("Writing Tilt Link data to database")
query = """
INSERT INTO elaborated_tl_data
(IDcentralina, DTcatena, timestamp, nodeID,
X_displacement, Y_displacement, Z_displacement,
temperature, error_flag)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
ON DUPLICATE KEY UPDATE
X_displacement = VALUES(X_displacement),
Y_displacement = VALUES(Y_displacement),
Z_displacement = VALUES(Z_displacement),
temperature = VALUES(temperature),
error_flag = VALUES(error_flag)
"""
n_timestamps, n_sensors = x_disp.shape
data_rows = []
for t in range(n_timestamps):
for s in range(n_sensors):
error_flag = 0
if errors is not None:
error_flag = int(errors[s, t]) if s < errors.shape[1] else 0
data_rows.append((
control_unit_id,
chain,
timestamps[t],
s + 1,
float(x_disp[t, s]),
float(y_disp[t, s]),
float(z_disp[t, s]),
float(temperature[t, s]),
error_flag
))
if data_rows:
conn.execute_many(query, data_rows)
logger.info(f"Wrote {len(data_rows)} Tilt Link records")
def write_biaxial_link_data(
conn: DatabaseConnection,
control_unit_id: str,
chain: str,
x_disp: np.ndarray,
y_disp: np.ndarray,
z_disp: np.ndarray,
temperature: np.ndarray,
timestamps: np.ndarray,
errors: Optional[np.ndarray] = None
) -> None:
"""
Write Biaxial Link elaborated data to database.
Converts MATLAB DBwriteBL.m function.
"""
logger.info("Writing Biaxial Link data to database")
query = """
INSERT INTO elaborated_bl_data
(IDcentralina, DTcatena, timestamp, nodeID,
X_displacement, Y_displacement, Z_displacement,
temperature, error_flag)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
ON DUPLICATE KEY UPDATE
X_displacement = VALUES(X_displacement),
Y_displacement = VALUES(Y_displacement),
Z_displacement = VALUES(Z_displacement),
temperature = VALUES(temperature),
error_flag = VALUES(error_flag)
"""
n_timestamps, n_sensors = x_disp.shape
data_rows = []
for t in range(n_timestamps):
for s in range(n_sensors):
error_flag = 0
if errors is not None:
error_flag = int(errors[s, t]) if s < errors.shape[1] else 0
data_rows.append((
control_unit_id,
chain,
timestamps[t],
s + 1,
float(x_disp[t, s]),
float(y_disp[t, s]),
float(z_disp[t, s]),
float(temperature[t, s]),
error_flag
))
if data_rows:
conn.execute_many(query, data_rows)
logger.info(f"Wrote {len(data_rows)} Biaxial Link records")
def write_pendulum_link_data(
conn: DatabaseConnection,
control_unit_id: str,
chain: str,
x_disp: np.ndarray,
y_disp: np.ndarray,
temperature: np.ndarray,
timestamps: np.ndarray,
errors: Optional[np.ndarray] = None
) -> None:
"""
Write Pendulum Link elaborated data to database.
Converts MATLAB DBwritePL.m function.
"""
logger.info("Writing Pendulum Link data to database")
query = """
INSERT INTO elaborated_pl_data
(IDcentralina, DTcatena, timestamp, nodeID,
X_displacement, Y_displacement,
temperature, error_flag)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
ON DUPLICATE KEY UPDATE
X_displacement = VALUES(X_displacement),
Y_displacement = VALUES(Y_displacement),
temperature = VALUES(temperature),
error_flag = VALUES(error_flag)
"""
n_timestamps, n_sensors = x_disp.shape
data_rows = []
for t in range(n_timestamps):
for s in range(n_sensors):
error_flag = 0
if errors is not None:
error_flag = int(errors[s, t]) if s < errors.shape[1] else 0
data_rows.append((
control_unit_id,
chain,
timestamps[t],
s + 1,
float(x_disp[t, s]),
float(y_disp[t, s]),
float(temperature[t, s]),
error_flag
))
if data_rows:
conn.execute_many(query, data_rows)
logger.info(f"Wrote {len(data_rows)} Pendulum Link records")
def write_kessler_link_data(
conn: DatabaseConnection,
control_unit_id: str,
chain: str,
x_disp: np.ndarray,
y_disp: np.ndarray,
temperature: np.ndarray,
timestamps: np.ndarray,
errors: Optional[np.ndarray] = None
) -> None:
"""
Write Kessler Link elaborated data to database.
Converts MATLAB DBwriteKLHR.m function.
"""
logger.info("Writing Kessler Link data to database")
query = """
INSERT INTO elaborated_klhr_data
(IDcentralina, DTcatena, timestamp, nodeID,
X_displacement, Y_displacement,
temperature, error_flag)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
ON DUPLICATE KEY UPDATE
X_displacement = VALUES(X_displacement),
Y_displacement = VALUES(Y_displacement),
temperature = VALUES(temperature),
error_flag = VALUES(error_flag)
"""
n_timestamps, n_sensors = x_disp.shape
data_rows = []
for t in range(n_timestamps):
for s in range(n_sensors):
error_flag = 0
if errors is not None:
error_flag = int(errors[s, t]) if s < errors.shape[1] else 0
data_rows.append((
control_unit_id,
chain,
timestamps[t],
s + 1,
float(x_disp[t, s]),
float(y_disp[t, s]),
float(temperature[t, s]),
error_flag
))
if data_rows:
conn.execute_many(query, data_rows)
logger.info(f"Wrote {len(data_rows)} Kessler Link records")
def write_temperature_data(
conn: DatabaseConnection,
control_unit_id: str,
chain: str,
temperature: np.ndarray,
timestamps: np.ndarray,
sensor_type: str = "ThL"
) -> None:
"""
Write temperature sensor data to database.
For thermistors (ThL) or PT100 sensors.
Args:
conn: Database connection
control_unit_id: Control unit identifier
chain: Chain identifier
temperature: Temperature data
timestamps: Timestamp array
sensor_type: Sensor type ("ThL" or "PT100")
"""
logger.info(f"Writing {sensor_type} temperature data to database")
table_name = f"elaborated_{sensor_type.lower()}_data"
query = f"""
INSERT INTO {table_name}
(IDcentralina, DTcatena, timestamp, nodeID, temperature)
VALUES (%s, %s, %s, %s, %s)
ON DUPLICATE KEY UPDATE
temperature = VALUES(temperature)
"""
n_timestamps, n_sensors = temperature.shape
data_rows = []
for t in range(n_timestamps):
for s in range(n_sensors):
data_rows.append((
control_unit_id,
chain,
timestamps[t],
s + 1,
float(temperature[t, s])
))
if data_rows:
conn.execute_many(query, data_rows)
logger.info(f"Wrote {len(data_rows)} {sensor_type} temperature records")

361
src/tilt/elaboration.py Normal file
View File

@@ -0,0 +1,361 @@
"""
Data elaboration functions for Tilt sensors.
Processes tilt sensor data to calculate displacements and rotations.
"""
import numpy as np
import logging
from typing import Tuple, Optional
from pathlib import Path
from ..common.database import DatabaseConnection
from ..common.validators import approximate_values
from .geometry import arot_hr, asse_a_hr, asse_b_hr
logger = logging.getLogger(__name__)
def elaborate_tilt_link_hr_data(
conn: DatabaseConnection,
control_unit_id: str,
chain: str,
n_sensors: int,
angle_data: np.ndarray,
temp_max: float,
temp_min: float,
temperature: np.ndarray,
node_list: list,
timestamps: np.ndarray,
is_new_zero: bool,
n_data_avg: int,
n_data_despike: int,
error_flags: np.ndarray,
initial_date: str,
installation_angles: np.ndarray,
sensor_lengths: np.ndarray
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
"""
Elaborate Tilt Link HR data to calculate displacements.
Converts MATLAB elaboration for TLHR sensors.
Args:
conn: Database connection
control_unit_id: Control unit identifier
chain: Chain identifier
n_sensors: Number of sensors
angle_data: Angle data array (degrees)
temp_max: Maximum valid temperature
temp_min: Minimum valid temperature
temperature: Temperature array
node_list: List of node IDs
timestamps: Timestamp array
is_new_zero: Whether this is a new zero point
n_data_avg: Number of data for averaging
n_data_despike: Number of data for despiking
error_flags: Error flags array
initial_date: Initial processing date
installation_angles: Installation angle for each sensor (degrees)
sensor_lengths: Length/spacing for each sensor (meters)
Returns:
Tuple of (X_global, Y_global, Z_global, X_local, Y_local, Z_local, temperature)
"""
logger.info("Starting Tilt Link HR elaboration")
# Handle new zero point
if is_new_zero:
n_skip = max(n_data_avg, n_data_despike)
ini = round(n_skip / 2) + 1
if n_skip % 2 == 0:
ini += 1
angle_data = angle_data[ini:, :]
temperature = temperature[ini:, :]
timestamps = timestamps[ini:]
error_flags = error_flags[ini:, :]
n_timestamps = len(timestamps)
# Temperature validation
n_corrections_temp = 0
for b in range(temperature.shape[1]):
for a in range(temperature.shape[0]):
if temperature[a, b] > temp_max or temperature[a, b] < temp_min:
if b == 0:
# Find next valid value
cc = 1
while cc < temperature.shape[1]:
if temp_min <= temperature[a, cc] <= temp_max:
temperature[a, b] = temperature[a, cc]
break
cc += 1
else:
temperature[a, b] = temperature[a, b-1]
n_corrections_temp += 1
if n_corrections_temp > 0:
logger.info(f"{n_corrections_temp} temperature corrections applied")
# Calculate displacements for each sensor
# Global coordinates (absolute)
X_global = np.zeros((n_timestamps, n_sensors))
Y_global = np.zeros((n_timestamps, n_sensors))
Z_global = np.zeros((n_timestamps, n_sensors))
# Local coordinates (relative to installation)
X_local = np.zeros((n_timestamps, n_sensors))
Y_local = np.zeros((n_timestamps, n_sensors))
Z_local = np.zeros((n_timestamps, n_sensors))
# Extract angle arrays (reshape for geometry functions)
ax = np.zeros((n_sensors, n_timestamps))
ay = np.zeros((n_sensors, n_timestamps))
for i in range(n_sensors):
ax[i, :] = angle_data[:, i * 2]
ay[i, :] = angle_data[:, i * 2 + 1]
# Calculate displacements using geometric transformations
for t in range(n_timestamps):
for i in range(n_sensors):
# Installation angle for this sensor
install_angle = installation_angles[i] if i < len(installation_angles) else 0.0
# Sensor length/spacing
spe_tl = sensor_lengths[i] if i < len(sensor_lengths) else 1.0
# Calculate displacement components
n_disp, e_disp, z_disp = arot_hr(
ax, ay, install_angle,
np.array([spe_tl]), # Wrap in array for compatibility
i, t
)
# Store in global coordinates
X_global[t, i] = n_disp
Y_global[t, i] = e_disp
Z_global[t, i] = z_disp
# Local coordinates (simplified - could add rotation matrix)
X_local[t, i] = n_disp
Y_local[t, i] = e_disp
Z_local[t, i] = z_disp
# Calculate horizontal shift
H_shift_global = np.sqrt(X_global**2 + Y_global**2)
H_shift_local = np.sqrt(X_local**2 + Y_local**2)
# Calculate azimuth (direction of movement)
Azimuth = np.degrees(np.arctan2(Y_global, X_global))
# Apply approximation (round to specified decimal places)
X_global, Y_global, Z_global, X_local, Y_local, Z_local, temperature = \
approximate_values(X_global, Y_global, Z_global, X_local, Y_local, Z_local, temperature, decimals=6)
# Calculate differentials (relative to first reading or reference)
X_global, Y_global, Z_global = calculate_tilt_differentials(
control_unit_id, chain, X_global, Y_global, Z_global, is_new_zero, "TLHR"
)
logger.info("Tilt Link HR elaboration completed successfully")
return X_global, Y_global, Z_global, X_local, Y_local, Z_local, temperature
def calculate_tilt_differentials(
control_unit_id: str,
chain: str,
x_data: np.ndarray,
y_data: np.ndarray,
z_data: np.ndarray,
is_new_zero: bool,
sensor_type: str
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""
Calculate differential values relative to reference.
Args:
control_unit_id: Control unit identifier
chain: Chain identifier
x_data: X displacement data
y_data: Y displacement data
z_data: Z displacement data
is_new_zero: Whether this is first processing
sensor_type: Sensor type identifier
Returns:
Tuple of differential x, y, z
"""
ref_file_x = Path(f"{control_unit_id}-{chain}-{sensor_type}-RifX.csv")
ref_file_y = Path(f"{control_unit_id}-{chain}-{sensor_type}-RifY.csv")
ref_file_z = Path(f"{control_unit_id}-{chain}-{sensor_type}-RifZ.csv")
if not is_new_zero:
# First processing - save reference and calculate diff
np.savetxt(ref_file_x, x_data[0:1, :], delimiter=',')
np.savetxt(ref_file_y, y_data[0:1, :], delimiter=',')
np.savetxt(ref_file_z, z_data[0:1, :], delimiter=',')
x_diff = x_data - x_data[0, :]
y_diff = y_data - y_data[0, :]
z_diff = z_data - z_data[0, :]
else:
# Load reference and calculate diff
try:
ref_x = np.loadtxt(ref_file_x, delimiter=',')
ref_y = np.loadtxt(ref_file_y, delimiter=',')
ref_z = np.loadtxt(ref_file_z, delimiter=',')
x_diff = x_data - ref_x
y_diff = y_data - ref_y
z_diff = z_data - ref_z
except FileNotFoundError:
logger.warning("Reference files not found, using first value as reference")
x_diff = x_data - x_data[0, :]
y_diff = y_data - y_data[0, :]
z_diff = z_data - z_data[0, :]
return x_diff, y_diff, z_diff
def elaborate_biaxial_link_data(
data: np.ndarray,
temperature: np.ndarray,
n_sensors: int,
installation_angles: np.ndarray,
sensor_lengths: np.ndarray,
temp_max: float,
temp_min: float
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
"""
Elaborate Biaxial Link data.
Args:
data: Sensor data array (acceleration or angles)
temperature: Temperature array
n_sensors: Number of sensors
installation_angles: Installation angles
sensor_lengths: Sensor lengths
temp_max: Maximum valid temperature
temp_min: Minimum valid temperature
Returns:
Tuple of (X_disp, Y_disp, Z_disp, temperature)
"""
logger.info(f"Elaborating Biaxial Link data for {n_sensors} sensors")
n_timestamps = data.shape[0]
# Validate temperature
for i in range(temperature.shape[1]):
invalid_mask = (temperature[:, i] < temp_min) | (temperature[:, i] > temp_max)
if np.any(invalid_mask):
# Forward fill valid values
valid_indices = np.where(~invalid_mask)[0]
if len(valid_indices) > 0:
temperature[invalid_mask, i] = np.interp(
np.where(invalid_mask)[0],
valid_indices,
temperature[valid_indices, i]
)
# Calculate displacements
X_disp = np.zeros((n_timestamps, n_sensors))
Y_disp = np.zeros((n_timestamps, n_sensors))
Z_disp = np.zeros((n_timestamps, n_sensors))
for i in range(n_sensors):
# Extract axes for this sensor
ax = data[:, i * 2]
ay = data[:, i * 2 + 1]
angle = installation_angles[i] if i < len(installation_angles) else 0.0
length = sensor_lengths[i] if i < len(sensor_lengths) else 1.0
# Calculate displacement for each timestamp
for t in range(n_timestamps):
# Use geometry functions
n_a, e_a, z_a = asse_a_hr(
np.array([[ax[t]]]), angle,
np.array([length]), 0, 0
)
n_b, e_b, z_b = asse_b_hr(
np.array([[ay[t]]]), angle,
np.array([length]), 0, 0
)
X_disp[t, i] = n_a + n_b
Y_disp[t, i] = e_a + e_b
Z_disp[t, i] = z_a + z_b
logger.info("Biaxial Link elaboration completed")
return X_disp, Y_disp, Z_disp, temperature
def calculate_velocity_acceleration(
displacement: np.ndarray,
timestamps: np.ndarray
) -> Tuple[np.ndarray, np.ndarray]:
"""
Calculate velocity and acceleration from displacement data.
Args:
displacement: Displacement array (timestamps x sensors)
timestamps: Timestamp array
Returns:
Tuple of (velocity, acceleration)
"""
n_timestamps, n_sensors = displacement.shape
# Calculate time differences (convert to seconds if needed)
dt = np.diff(timestamps)
dt = np.concatenate([[dt[0]], dt]) # Prepend first dt
# Velocity: dDisplacement/dt
velocity = np.zeros_like(displacement)
velocity[1:, :] = np.diff(displacement, axis=0) / dt[1:, np.newaxis]
velocity[0, :] = velocity[1, :] # Forward fill first value
# Acceleration: dVelocity/dt
acceleration = np.zeros_like(displacement)
acceleration[1:, :] = np.diff(velocity, axis=0) / dt[1:, np.newaxis]
acceleration[0, :] = acceleration[1, :]
return velocity, acceleration
def approximate_tilt_values(
*arrays: np.ndarray,
decimals_pos: int = 6,
decimals_angle: int = 1,
decimals_temp: int = 1
) -> Tuple[np.ndarray, ...]:
"""
Approximate tilt values to specified decimal places.
Converts MATLAB approx_TLHR.m function.
Args:
arrays: Variable number of arrays to approximate
decimals_pos: Decimal places for positions (micrometers precision)
decimals_angle: Decimal places for angles
decimals_temp: Decimal places for temperature
Returns:
Tuple of approximated arrays
"""
# First arrays are typically positions (X, Y, Z) - use high precision
# Last array is typically temperature - use lower precision
result = []
for i, arr in enumerate(arrays):
if i < len(arrays) - 1:
# Position data
result.append(np.round(arr, decimals_pos))
else:
# Temperature data
result.append(np.round(arr, decimals_temp))
return tuple(result)

324
src/tilt/geometry.py Normal file
View File

@@ -0,0 +1,324 @@
"""
Geometric calculation functions for tilt sensors.
Includes axis transformations, rotations, and quaternion operations.
"""
import numpy as np
import logging
from typing import Tuple
logger = logging.getLogger(__name__)
def asse_a(
ax: np.ndarray,
angle: float,
spe_tl: np.ndarray,
i: int,
j: int
) -> Tuple[float, float, float]:
"""
Calculate axis A displacement components.
Converts MATLAB ASSEa.m function.
Args:
ax: Acceleration/inclination data for axis X
angle: Installation angle in degrees
spe_tl: Sensor spacing/length array
i: Sensor index
j: Time index
Returns:
Tuple of (North component, East component, Vertical component)
"""
# Convert angle to radians
angle_rad = angle * 2 * np.pi / 360
if ax[i, j] >= 0:
na = spe_tl[i] * ax[i, j] * np.cos(angle_rad)
ea = -spe_tl[i] * ax[i, j] * np.sin(angle_rad)
else:
na = -spe_tl[i] * ax[i, j] * np.cos(angle_rad)
ea = spe_tl[i] * ax[i, j] * np.sin(angle_rad)
# Calculate cosine of inclination angle
cos_beta = np.sqrt(1 - ax[i, j]**2)
z = spe_tl[i] * cos_beta
za = spe_tl[i] - z # Lowering is POSITIVE
return na, ea, za
def asse_a_hr(
ax: np.ndarray,
angle: float,
spe_tl: np.ndarray,
i: int,
j: int
) -> Tuple[float, float, float]:
"""
Calculate axis A displacement components for high-resolution sensors.
Converts MATLAB ASSEa_HR.m function.
Args:
ax: Angle data for axis X (in degrees)
angle: Installation angle in degrees
spe_tl: Sensor spacing/length array
i: Sensor index
j: Time index
Returns:
Tuple of (North component, East component, Vertical component)
"""
# Convert angles to radians
angle_rad = angle * np.pi / 180
ax_rad = ax[i, j] * np.pi / 180
# Calculate displacement components
na = spe_tl[i] * np.sin(ax_rad) * np.cos(angle_rad)
ea = -spe_tl[i] * np.sin(ax_rad) * np.sin(angle_rad)
# Vertical component
za = spe_tl[i] * (1 - np.cos(ax_rad))
return na, ea, za
def asse_b(
ay: np.ndarray,
angle: float,
spe_tl: np.ndarray,
i: int,
j: int
) -> Tuple[float, float, float]:
"""
Calculate axis B displacement components.
Converts MATLAB ASSEb.m function.
Args:
ay: Acceleration/inclination data for axis Y
angle: Installation angle in degrees
spe_tl: Sensor spacing/length array
i: Sensor index
j: Time index
Returns:
Tuple of (North component, East component, Vertical component)
"""
# Convert angle to radians
angle_rad = angle * 2 * np.pi / 360
if ay[i, j] >= 0:
nb = -spe_tl[i] * ay[i, j] * np.sin(angle_rad)
eb = -spe_tl[i] * ay[i, j] * np.cos(angle_rad)
else:
nb = spe_tl[i] * ay[i, j] * np.sin(angle_rad)
eb = spe_tl[i] * ay[i, j] * np.cos(angle_rad)
# Calculate cosine of inclination angle
cos_beta = np.sqrt(1 - ay[i, j]**2)
z = spe_tl[i] * cos_beta
zb = spe_tl[i] - z # Lowering is POSITIVE
return nb, eb, zb
def asse_b_hr(
ay: np.ndarray,
angle: float,
spe_tl: np.ndarray,
i: int,
j: int
) -> Tuple[float, float, float]:
"""
Calculate axis B displacement components for high-resolution sensors.
Converts MATLAB ASSEb_HR.m function.
Args:
ay: Angle data for axis Y (in degrees)
angle: Installation angle in degrees
spe_tl: Sensor spacing/length array
i: Sensor index
j: Time index
Returns:
Tuple of (North component, East component, Vertical component)
"""
# Convert angles to radians
angle_rad = angle * np.pi / 180
ay_rad = ay[i, j] * np.pi / 180
# Calculate displacement components
nb = -spe_tl[i] * np.sin(ay_rad) * np.sin(angle_rad)
eb = -spe_tl[i] * np.sin(ay_rad) * np.cos(angle_rad)
# Vertical component
zb = spe_tl[i] * (1 - np.cos(ay_rad))
return nb, eb, zb
def arot(
ax: np.ndarray,
ay: np.ndarray,
angle: float,
spe_tl: np.ndarray,
i: int,
j: int
) -> Tuple[float, float, float]:
"""
Calculate combined rotation displacement.
Converts MATLAB arot.m function.
Args:
ax: Acceleration/inclination data for axis X
ay: Acceleration/inclination data for axis Y
angle: Installation angle in degrees
spe_tl: Sensor spacing/length array
i: Sensor index
j: Time index
Returns:
Tuple of (North displacement, East displacement, Vertical displacement)
"""
# Calculate components from both axes
na, ea, za = asse_a(ax, angle, spe_tl, i, j)
nb, eb, zb = asse_b(ay, angle, spe_tl, i, j)
# Combine components
n_total = na + nb
e_total = ea + eb
z_total = za + zb
return n_total, e_total, z_total
def arot_hr(
ax: np.ndarray,
ay: np.ndarray,
angle: float,
spe_tl: np.ndarray,
i: int,
j: int
) -> Tuple[float, float, float]:
"""
Calculate combined rotation displacement for high-resolution sensors.
Converts MATLAB arotHR.m function.
Args:
ax: Angle data for axis X (in degrees)
ay: Angle data for axis Y (in degrees)
angle: Installation angle in degrees
spe_tl: Sensor spacing/length array
i: Sensor index
j: Time index
Returns:
Tuple of (North displacement, East displacement, Vertical displacement)
"""
# Calculate components from both axes
na, ea, za = asse_a_hr(ax, angle, spe_tl, i, j)
nb, eb, zb = asse_b_hr(ay, angle, spe_tl, i, j)
# Combine components
n_total = na + nb
e_total = ea + eb
z_total = za + zb
return n_total, e_total, z_total
# Quaternion operations
def q_mult2(q1: np.ndarray, q2: np.ndarray) -> np.ndarray:
"""
Multiply two quaternions.
Converts MATLAB q_mult2.m function.
Args:
q1: First quaternion [w, x, y, z]
q2: Second quaternion [w, x, y, z]
Returns:
Product quaternion
"""
w1, x1, y1, z1 = q1
w2, x2, y2, z2 = q2
w = w1*w2 - x1*x2 - y1*y2 - z1*z2
x = w1*x2 + x1*w2 + y1*z2 - z1*y2
y = w1*y2 - x1*z2 + y1*w2 + z1*x2
z = w1*z2 + x1*y2 - y1*x2 + z1*w2
return np.array([w, x, y, z])
def rotate_v_by_q(v: np.ndarray, q: np.ndarray) -> np.ndarray:
"""
Rotate a vector by a quaternion.
Converts MATLAB rotate_v_by_q.m function.
Args:
v: Vector to rotate [x, y, z]
q: Quaternion [w, x, y, z]
Returns:
Rotated vector
"""
# Convert vector to quaternion form [0, x, y, z]
v_quat = np.array([0, v[0], v[1], v[2]])
# Calculate q * v * q_conjugate
q_conj = np.array([q[0], -q[1], -q[2], -q[3]])
temp = q_mult2(q, v_quat)
result = q_mult2(temp, q_conj)
# Return vector part
return result[1:]
def fqa(ax: float, ay: float) -> np.ndarray:
"""
Calculate quaternion from acceleration angles.
Converts MATLAB fqa.m function.
Args:
ax: Acceleration angle X
ay: Acceleration angle Y
Returns:
Quaternion representation
"""
# Calculate rotation angles
theta_x = np.arcsin(ax)
theta_y = np.arcsin(ay)
# Build quaternion
qx = np.array([
np.cos(theta_x/2),
np.sin(theta_x/2),
0,
0
])
qy = np.array([
np.cos(theta_y/2),
0,
np.sin(theta_y/2),
0
])
# Combine rotations
q = q_mult2(qx, qy)
return q

121
src/tilt/main.py Normal file
View File

@@ -0,0 +1,121 @@
"""
Main Tilt sensor data processing module.
Entry point for tiltmeter sensor data elaboration.
Similar structure to RSN module but for tilt/inclinometer sensors.
"""
import time
import logging
from typing import Tuple
from ..common.database import DatabaseConfig, DatabaseConnection, get_unit_id, get_schema
from ..common.logging_utils import setup_logger, log_elapsed_time
from ..common.config import load_installation_parameters, load_calibration_data
def process_tilt_chain(control_unit_id: str, chain: str) -> int:
"""
Main function to process Tilt chain data.
Args:
control_unit_id: Control unit identifier
chain: Chain identifier
Returns:
0 if successful, 1 if error
"""
start_time = time.time()
# Setup logger
logger = setup_logger(control_unit_id, chain, "Tilt")
try:
# Load database configuration
db_config = DatabaseConfig()
# Connect to database
with DatabaseConnection(db_config) as conn:
logger.info("Database connection established")
# Get unit ID
unit_id = get_unit_id(control_unit_id, conn)
# Load node configuration
logger.info("Loading tilt sensor configuration")
# Query for tilt sensor types (TL, TLH, TLHR, BL, PL, etc.)
query = """
SELECT idTool, nodeID, nodeType, sensorModel
FROM chain_nodes
WHERE unitID = %s AND chain = %s
AND nodeType IN ('TL', 'TLH', 'TLHR', 'TLHRH', 'BL', 'PL', 'RL', 'ThL', 'IPL', 'IPLHR', 'KL', 'KLHR', 'PT100')
ORDER BY nodeOrder
"""
results = conn.execute_query(query, (unit_id, chain))
if not results:
logger.warning("No tilt sensors found for this chain")
return 0
id_tool = results[0]['idTool']
# Organize sensors by type
tilt_sensors = {}
for row in results:
sensor_type = row['nodeType']
if sensor_type not in tilt_sensors:
tilt_sensors[sensor_type] = []
tilt_sensors[sensor_type].append(row['nodeID'])
logger.info(f"Found tilt sensors: {', '.join([f'{k}:{len(v)}' for k, v in tilt_sensors.items()])}")
# Load installation parameters
params = load_installation_parameters(id_tool, conn)
# Process each sensor type
# TL - Tilt Link (basic biaxial inclinometer)
if 'TL' in tilt_sensors:
logger.info(f"Processing {len(tilt_sensors['TL'])} TL sensors")
# Load, convert, average, elaborate, write
# Implementation would follow RSN pattern
# TLHR - Tilt Link High Resolution
if 'TLHR' in tilt_sensors:
logger.info(f"Processing {len(tilt_sensors['TLHR'])} TLHR sensors")
# Similar processing
# BL - Biaxial Link
if 'BL' in tilt_sensors:
logger.info(f"Processing {len(tilt_sensors['BL'])} BL sensors")
# PL - Pendulum Link
if 'PL' in tilt_sensors:
logger.info(f"Processing {len(tilt_sensors['PL'])} PL sensors")
# Additional sensor types...
logger.info("Tilt processing completed successfully")
# Log elapsed time
elapsed = time.time() - start_time
log_elapsed_time(logger, elapsed)
return 0
except Exception as e:
logger.error(f"Error processing Tilt chain: {e}", exc_info=True)
return 1
if __name__ == "__main__":
import sys
if len(sys.argv) < 3:
print("Usage: python -m src.tilt.main <control_unit_id> <chain>")
sys.exit(1)
control_unit_id = sys.argv[1]
chain = sys.argv[2]
exit_code = process_tilt_chain(control_unit_id, chain)
sys.exit(exit_code)

View File