Evaluation Module#

Point-adjusted evaluation metrics for time-series anomaly detection, the standard SMAP / MSL protocol used by telemanom and MEMTO.

Point-adjusted evaluation metrics for time-series anomaly detection.

Point adjustment is the standard SMAP / MSL protocol (Xu et al., 2018): if a detector flags any point inside a labeled anomaly segment, the whole segment counts as detected. It rewards catching an event even when not every point of it is flagged, and makes results comparable to the telemanom and MEMTO baselines that report point-adjusted F1.

telemetry_anomdet.evaluation.point_adjust(pred: Sequence[bool], truth: Sequence[bool]) → ndarray[source]#

Apply point adjustment to point-level predictions.

For each contiguous anomaly segment in truth, if pred flags any point inside it, the entire segment is marked as predicted.

Parameters:

  • pred – Point-level boolean predictions.

  • truth – Point-level boolean ground truth, same length as pred.

Returns:

The adjusted boolean prediction array.

Return type:

np.ndarray

telemetry_anomdet.evaluation.prf(pred: Sequence[bool], truth: Sequence[bool]) → dict[source]#

Precision, recall, and F1 for point-level boolean arrays.

Returns:

{‘precision’, ‘recall’, ‘f1’, ‘tp’, ‘fp’, ‘fn’}.

Return type:

dict

telemetry_anomdet.evaluation.point_adjusted_f1(pred: Sequence[bool], truth: Sequence[bool]) → dict[source]#

Point-adjusted precision, recall, and F1 (the standard SMAP metric).

Equivalent to prf() computed on point_adjust() output.

telemetry_anomdet.evaluation.windows_to_points(window_flags: Sequence[bool], n_points: int, *, window_size: int, step: int) → ndarray[source]#

Expand window-level flags to a point-level mask.

Point t is flagged if it falls inside any flagged window. Window i spans [i*step, i*step + window_size), matching windowify.

Parameters:

  • window_flags – Boolean flag per window.

  • n_points – Length of the point-level series the windows came from.

  • window_size – Samples per window.

  • step – Stride between windows.

Returns:

Boolean mask of length n_points.

Return type:

np.ndarray

telemetry_anomdet.evaluation.windows_to_point_scores(window_scores: Sequence[float], n_points: int, *, window_size: int, step: int) → ndarray[source]#

Expand window-level scores to a point-level score array.

Each point takes the maximum score among the windows covering it (window i spans [i*step, i*step + window_size)). Points not covered by any window fall back to the minimum window score.

Parameters:

  • window_scores – Anomaly score per window (higher = more anomalous).

  • n_points – Length of the point-level series.

  • window_size – Samples per window.

  • step – Stride between windows.

Returns:

Float score array of length n_points.

Return type:

np.ndarray

telemetry_anomdet.evaluation.best_point_adjusted_f1(scores: Sequence[float], truth: Sequence[bool], *, n_thresholds: int = 200) → dict[source]#

Best point-adjusted F1 over a threshold sweep on point-level scores.

Selects the threshold that maximizes point-adjusted F1. This is the standard SMAP / MSL “best F1” protocol used by telemanom and MEMTO, which makes those baselines comparable. Note that it selects the threshold using the labels, so it should be reported as an oracle-threshold upper bound, not a deployable operating point.

Parameters:

  • scores – Point-level anomaly scores (higher = more anomalous).

  • truth – Point-level boolean ground truth.

  • n_thresholds – Number of candidate thresholds sampled across the score range.

Returns:

The best {‘precision’, ‘recall’, ‘f1’, ‘tp’, ‘fp’, ‘fn’, ‘threshold’}.

Return type:

dict

telemetry_anomdet.evaluation.pr_auc(scores: Sequence[float], truth: Sequence[bool]) → float[source]#

Area under the precision-recall curve, as average precision.

Computed on raw point scores with no point adjustment, so a detector is credited for the points it actually flags. Two properties make this the honest companion to best_point_adjusted_f1():

  • It integrates over every operating point instead of reporting the single best one, so no threshold can be selected against the labels.

  • Its value for an uninformative detector is the positive base rate. Any score above that reflects real ranking ability, and the margin is interpretable. ROC AUC instead sits at 0.5 for random regardless of class balance, which flatters a detector when anomalies are rare.

Parameters:

  • scores – Point-level anomaly scores (higher = more anomalous).

  • truth – Point-level boolean ground truth, same length as scores.

Returns:

Average precision in [0, 1]; the base rate for random scores.

Return type:

float

telemetry_anomdet.evaluation.false_alarm_rate_at_recall(scores: Sequence[float], truth: Sequence[bool], target_recall: float = 0.8) → dict[source]#

Cost of reaching a recall target, as a false positive rate per point.

Answers the operational question a fixed threshold has to settle: to catch this fraction of anomalous points, how often does the detector fire on nominal data? Unlike a best-F1 figure this is reported at a stated recall, so two detectors are compared at the same sensitivity.

Parameters:

  • scores – Point-level anomaly scores (higher = more anomalous).

  • truth – Point-level boolean ground truth, same length as scores.

  • target_recall – Recall to reach, in (0, 1].

Returns:

{'threshold', 'recall', 'false_alarm_rate', 'precision'}. The false alarm rate is false positives divided by the number of nominal points. Returns a rate of 1.0 when the target recall is unreachable.

Return type:

dict

telemetry_anomdet.evaluation.evaluate_sequences(predicted: Sequence[tuple[int, int]], true_sequences: Sequence[tuple[int, int]]) → dict[source]#

Score predicted anomaly ranges against labelled ones, event by event.

This reproduces the scoring in Hundman et al.’s telemanom, so numbers are directly comparable with the results published for SMAP and MSL. It is the metric an operator experiences: how many real events were caught, and how many times the system cried wolf.

The two sides are counted over different things, which is deliberate and easy to get wrong. A true positive is a labelled sequence that some prediction overlapped, so true positives and false negatives partition the labelled sequences. A false positive is a predicted sequence that overlapped nothing, so it is counted over predictions instead.

When one prediction spans several labelled sequences, only the first is credited. A single alarm covering two events is one catch, not two.

Contrast point_adjusted_f1(), which counts points and credits an entire labelled segment to a single flagged sample. That inflates a detector raising many short false alarms beside a few long true ones, sometimes by a wide margin, so the two metrics can rank detectors differently.

Parameters:

  • predicted – Predicted (start, end) ranges, inclusive of both ends.

  • true_sequences – Labelled (start, end) ranges, inclusive.

Returns:

true_positives, false_positives, false_negatives, and the tp_sequences / fp_sequences that produced them.

Return type:

dict

telemetry_anomdet.evaluation.f_beta(precision: float, recall: float, beta: float = 1.0) → float[source]#

Weighted harmonic mean of precision and recall.

beta sets how much recall counts relative to precision: below 1 favours precision, above 1 favours recall. beta = 0.5 is what the telemanom results report, and it suits a trigger whose false alarms are expensive.

Note that when precision and recall are equal, every beta returns that same value, which is a useful check when reading published tables.

Parameters:

  • precision – Precision in [0, 1].

  • recall – Recall in [0, 1].

  • beta – Relative weight on recall.

Returns:

The F-beta score, or 0.0 when precision and recall are both zero.

Return type:

float

telemetry_anomdet.evaluation.sequence_prf(rows: Sequence[dict]) → dict[source]#

Aggregate per-channel evaluate_sequences() results, as telemanom does.

Counts are pooled across channels before precision and recall are computed, rather than averaging per-channel rates. Channels that detect nothing then contribute their misses without also contributing a precision of zero.

Both f1 and f_0.5 are returned. The latter weights precision more heavily and is the statistic the telemanom results headline.

Parameters:

rows – Per-channel dicts from evaluate_sequences().

Returns:

pooled true_positives, false_positives, false_negatives, and the derived precision, recall, f1 and f_half.

Return type:

dict