Evaluation Module

Point-adjusted evaluation metrics for time-series anomaly detection, the standard SMAP / MSL protocol used by telemanom and MEMTO.

Point-adjusted evaluation metrics for time-series anomaly detection.

Point adjustment is the standard SMAP / MSL protocol (Xu et al., 2018): if a detector flags any point inside a labeled anomaly segment, the whole segment counts as detected. It rewards catching an event even when not every point of it is flagged, and makes results comparable to the telemanom and MEMTO baselines that report point-adjusted F1.

telemetry_anomdet.evaluation.point_adjust(pred: Sequence[bool], truth: Sequence[bool]) → ndarray[source]

Apply point adjustment to point-level predictions.

For each contiguous anomaly segment in truth, if pred flags any point inside it, the entire segment is marked as predicted.

Parameters:

  • pred – Point-level boolean predictions.

  • truth – Point-level boolean ground truth, same length as pred.

Returns:

The adjusted boolean prediction array.

Return type:

np.ndarray

telemetry_anomdet.evaluation.prf(pred: Sequence[bool], truth: Sequence[bool]) → dict[source]

Precision, recall, and F1 for point-level boolean arrays.

Returns:

{‘precision’, ‘recall’, ‘f1’, ‘tp’, ‘fp’, ‘fn’}.

Return type:

dict

telemetry_anomdet.evaluation.point_adjusted_f1(pred: Sequence[bool], truth: Sequence[bool]) → dict[source]

Point-adjusted precision, recall, and F1 (the standard SMAP metric).

Equivalent to prf() computed on point_adjust() output.

telemetry_anomdet.evaluation.windows_to_points(window_flags: Sequence[bool], n_points: int, *, window_size: int, step: int) → ndarray[source]

Expand window-level flags to a point-level mask.

Point t is flagged if it falls inside any flagged window. Window i spans [i*step, i*step + window_size), matching windowify.

Parameters:

  • window_flags – Boolean flag per window.

  • n_points – Length of the point-level series the windows came from.

  • window_size – Samples per window.

  • step – Stride between windows.

Returns:

Boolean mask of length n_points.

Return type:

np.ndarray

telemetry_anomdet.evaluation.windows_to_point_scores(window_scores: Sequence[float], n_points: int, *, window_size: int, step: int) → ndarray[source]

Expand window-level scores to a point-level score array.

Each point takes the maximum score among the windows covering it (window i spans [i*step, i*step + window_size)). Points not covered by any window fall back to the minimum window score.

Parameters:

  • window_scores – Anomaly score per window (higher = more anomalous).

  • n_points – Length of the point-level series.

  • window_size – Samples per window.

  • step – Stride between windows.

Returns:

Float score array of length n_points.

Return type:

np.ndarray

telemetry_anomdet.evaluation.best_point_adjusted_f1(scores: Sequence[float], truth: Sequence[bool], *, n_thresholds: int = 200) → dict[source]

Best point-adjusted F1 over a threshold sweep on point-level scores.

Selects the threshold that maximizes point-adjusted F1. This is the standard SMAP / MSL “best F1” protocol used by telemanom and MEMTO, which makes those baselines comparable. Note that it selects the threshold using the labels, so it should be reported as an oracle-threshold upper bound, not a deployable operating point.

Parameters:

  • scores – Point-level anomaly scores (higher = more anomalous).

  • truth – Point-level boolean ground truth.

  • n_thresholds – Number of candidate thresholds sampled across the score range.

Returns:

The best {‘precision’, ‘recall’, ‘f1’, ‘tp’, ‘fp’, ‘fn’, ‘threshold’}.

Return type:

dict