Preprocessing Module#

This module provides functions for standardizing, cleaning, and resampling telemetry data before analysis.

Preprocessing utilities for telemetry data.

This module handles data cleaning, normalization, and other transformations before feature extraction.

telemetry_anomdet.preprocessing.preprocessing.clean(df: DataFrame, *, physical_bounds=None) → DataFrame[source]#

Remove non existant values, non numeric readings, and physically impossible sensor values.

Parameters:

  • df (pd.DataFrame) – Long form telemetry data with columns [‘timestamp’, ‘variable’, ‘value’].

  • physical_bounds (dict, optional) – Mapping of variable names or patterns to (min, max) valid ranges. Example: {‘Battery_Voltage’: (0, 20), ‘Battery_Temp’: (-40, 85)}.

Returns:

Cleaned dataset.

Return type:

pd.DataFrame

telemetry_anomdet.preprocessing.preprocessing.dedupe(df: DataFrame) → DataFrame[source]#

Remove duplicate or retransmitted rows.

Parameters:

df (pd.DataFrame) – Long form telemetry data with potential duplicates.

Returns:

DataFrame with duplicates (timestamp, variable) removed.

Return type:

pd.DataFrame

telemetry_anomdet.preprocessing.preprocessing.integrity_check(df: DataFrame, *, require_utc: bool = True, require_sorted: bool = True) → None[source]#

Verify timestamp format, timezone, and column consistency.

Parameters:

  • df (pd.DataFrame) – Long form telemetry data.

  • require_utc (bool) – If True, ensure timestamps are UTC.

  • require_sorted (bool) – If True, ensure timestamps are sorted ascending.

Raises:

ValueError – If schema or ordering fails validation.

telemetry_anomdet.preprocessing.preprocessing.resample(df: DataFrame, *, rule: str = '5s', agg: str = 'mean') → DataFrame[source]#

Resample irregularly spaced data to a uniform cadence.

Parameters:

  • df (pd.DataFrame) – Long form telemetry data.

  • rule (str) – Resample frequency (‘1s’, ‘5s’, ‘1min’).

  • agg (str) – Aggregation method (‘mean’, ‘median’, etc.) when multiple values exist per interval.

Returns:

Resampled dataset with regular time intervals.

Return type:

pd.DataFrame

telemetry_anomdet.preprocessing.preprocessing.interpolate_gaps(df: DataFrame, *, method: str = 'ffill', limit: int | None = 1) → DataFrame[source]#

Fill small missing gaps to ensure continuous time steps.

Parameters:

  • df (pd.DataFrame) – Resampled telemetry data in long form.

  • method (str) – Interpolation strategy (‘ffill’, ‘bfill’, ‘linear’, etc.).

  • limit (int, optional) – Maximum consecutive non existant value steps to fill. None means no limit.

Returns:

Gap filled dataset in long form.

Return type:

pd.DataFrame

telemetry_anomdet.preprocessing.preprocessing.normalize_fit(df: DataFrame, *, method: str = 'zscore') → dict[source]#

Compute normalization parameters for each variable.

Parameters:

  • df (pd.DataFrame) – Cleaned telemetry data (usually training subset).

  • method (str) – Normalization method (‘zscore’ or ‘minmax’).

Returns:

Mapping {variable: (center, scale)} where the pair is

(mean, std) for ‘zscore’ or (min, range) for ‘minmax’. A scale of 0 (constant variable) is stored as 1.0 so that applying the parameters never divides by zero.

Return type:

dict

telemetry_anomdet.preprocessing.preprocessing.pipeline(df: DataFrame, *, physical_bounds: dict | None = None, resample_rule: str | None = '5s', resample_agg: str = 'mean', interpolate_method: str = 'ffill', gap_limit: int | None = 1) → DataFrame[source]#

Execute minimal preprocessing pipeline for this dataset.

Steps: clean -> dedupe -> integrity_check -> resample -> interpolate_gaps.

Normalization is intentionally kept out of this pipeline. Following scikit-learn convention, fitting normalization stats is a separate step: call normalize_fit() on the returned (training) frame and reuse the resulting params at inference to prevent leakage.

Parameters:

  • df (pd.DataFrame) – Raw telemetry dataset.

  • physical_bounds (dict, optional) – Min/max physical limits per variable.

  • resample_rule (str) – Resampling frequency (default ‘5s’). None skips resampling.

  • resample_agg (str) – Aggregation method (default ‘mean’).

  • interpolate_method (str) – Gap-fill strategy (‘ffill’, ‘linear’, etc.).

  • gap_limit (int, optional) – Max consecutive steps to interpolate.

Returns:

Fully preprocessed dataset.

Return type:

pd.DataFrame