Skip to content

Evaluation API

Stateful anomaly evaluation preserves score-before-learn ordering and produces immutable records and results. See the evaluation guide for warm-up, ranking metrics, retention, learning policies, and reproducibility.

PrequentialEvaluator

Python
PrequentialEvaluator(model: ModelProtocol | DetectorConfig, *, warmup: int | None = None, readiness: Readiness | None = None, learn_filter: LearnFilter | None = None, learn_policy_name: str | None = None, metrics: bool = False, metric_window_size: int | None = 1000, higher_is_more_anomalous: bool | None = None, measure_time: bool = True)

Evaluate anomaly events against prior state, then optionally learn them.

Parameters:

Name Type Description Default
model ModelProtocol | DetectorConfig

An existing structural model, or a catalog configuration from which a fresh model is built. Existing models are never reset.

 required
warmup int | None

Number of successful learns excluded before scoring. For a configuration without transformers, None uses catalog event-based warm-up. Pipelines and other warm-up units require an explicit value or readiness callback. Existing model instances default to 0.

 None
readiness Readiness | None

Optional predicate evaluated against current model state before scoring, after warm-up. False excludes the candidate from scoring while still permitting learning. It must not mutate state.

 None
learn_filter LearnFilter | None

Optional decision receiving the event and its raw score (None during exclusion). Labels are never supplied to this callback or the model. Returning False skips learning, including in warm-up.

 None
learn_policy_name str | None

Required descriptive name when using learn_filter.

 None
metrics bool

Enable optional scikit-learn ranking metrics from [eval]. Disabled by default, keeping evaluation usable in the base install.

 False
metric_window_size int | None

Retain at most this many labeled, scored events. None explicitly opts into O(n) retention for exact full-stream AP and ROC AUC. The window counts labeled scored events, not arrivals.

 1000
higher_is_more_anomalous bool | None

Ranking orientation; None uses the catalog value, or True for an existing model. Unknown catalog orientation requires an explicit value. Trace scores remain in their raw scale.

 None
measure_time bool

Measure separate scoring and learning call durations.

 True

Calls to update, iter_evaluate and evaluate continue the same model and evaluator state. Create a new evaluator for each independent comparison. Stream order is preserved; seeds and dataset order are caller-controlled. Model exceptions propagate without advancing evaluator statistics. Model state rollback is governed by the supplied model's own contract.

update

Python
update(x: FeatureMap, label: object = None) -> EvaluationRecord

Process one event and return its immutable trace record.

Validate the entire event and label before calling the model. No scores are fabricated for excluded events. Callback decisions must be booleans. Input dictionaries are copied so model/policy calls cannot edit the caller's sample. Concurrent calls on one evaluator are unsupported.

iter_evaluate

Python
iter_evaluate(stream: Iterable[tuple[FeatureMap, object]]) -> Iterator[EvaluationRecord]

Lazily process an ordered stream; write yielded records to any sink.

evaluate

Python
evaluate(stream: Iterable[tuple[FeatureMap, object]]) -> EvaluationResult

Consume the stream and return a cumulative result without storing traces.

result

Python
result() -> EvaluationResult

Compute a detached snapshot; this never scores or learns a model.

EvaluationRecord dataclass

Python
EvaluationRecord(index: int, label: int | None, score: float | None, learned: bool, exclusion: Literal['warmup', 'not_ready'] | None, score_time_ns: int | None, learn_time_ns: int | None)

One successfully processed event; indices start at zero.

Excluded events have no score. Labels are evaluation data only. Timing measures the model calls, excluding validation, readiness and policy logic.

EvaluationResult dataclass

Python
EvaluationResult(n_seen: int, n_scored: int, n_learned: int, n_warmup: int, n_not_ready: int, n_labeled: int, prevalence: float | None, metric_samples: int, metrics_enabled: bool, metric_window_size: int | None, metric_prevalence: float | None, average_precision: float | None, roc_auc: float | None, score_time_ns: int | None, learn_time_ns: int | None, package_version: str, config: DetectorConfig | None, config_fingerprint: str | None, warmup: int, uses_readiness_callback: bool, learn_policy_name: str, higher_is_more_anomalous: bool)

A detached summary of the evaluator's cumulative processing.

Ranking metrics and metric_prevalence cover only metric_samples: the latest labeled, scored events within metric_window_size, or all such events when the window is None. prevalence covers all labeled, scored events. Metrics are None when disabled or mathematically undefined. Exact full-stream metrics retain O(n) score/label pairs; windowed metrics retain O(window_size). No event dictionaries or trace records are retained.