Skip to content

Calibration API

A calibrator issues a threshold with predict(key) and learns the score of that prediction with update(key, score) once the score is known.

ScoreCalibrator

Python
ScoreCalibrator(method: str, *, alpha: float, window: int = 100, **params: object)

A threshold on a stream of scores, learned online with delayed feedback.

predict(key) issues the threshold for a prediction identified by key; update(key, score) reports its score once known, which may be several steps later (a forecast's lead). Scores may be negative.

Parameters:

Name Type Description Default
method str

Identifier of a method in :data:CALIBRATORS.

 required
alpha float

Target miscoverage, so 1 - alpha of scores should fall at or below the threshold.

 required
window int

Number of recent scores in the calibration window.

 100
**params object

Method parameters; omitted ones take their defaults.

 {}

Examples:

Python
from aberrant.calibration import ScoreCalibrator

calibrator = ScoreCalibrator("aci", alpha=0.1, window=100, gamma=0.01)
for t, score in enumerate([0.2, 0.4, 0.1, 0.3]):
    threshold = calibrator.predict(t)
    calibrator.update(t, score)

spec property

Python
spec: CalibratorSpec

The method's description and parameters.

predict

Python
predict(key: Hashable) -> float

The threshold issued for key.

+inf before any score; otherwise clipped to the largest absolute score seen, as the method compares its feedback against that value.

update

Python
update(key: Hashable, score: float) -> None

Learn the score of the prediction issued for key.

CalibratorSpec dataclass

Python
CalibratorSpec(id: str, name: str, reference: str, summary: str, family: Literal['conformal', 'bayesian'], coverage: Coverage, parameters: tuple[CalibratorParameter, ...], factory: Factory)

A calibration method: what it is, its parameters and its guarantee.

coverage says what the threshold promises about the score stream:

  • "exchangeable": a new score exceeds the threshold with probability at most alpha when it is exchangeable with the calibration scores.
  • "long_run": the fraction of scores above the threshold tends to alpha for any score sequence. This bounds every exceedance, normal or anomalous, not the false-positive rate on normal scores alone.
  • "none": no coverage guarantee; calibrated only in practice.

defaults

Python
defaults() -> dict[str, float]

Every parameter at its default.

resolve

Python
resolve(params: Mapping[str, object]) -> dict[str, float]

Defaults overridden by params, each value validated.

CalibratorParameter dataclass

Python
CalibratorParameter(key: str, label: str, default: float, low: float, high: float, step: float, minimum: float, maximum: float = inf, exclusive_minimum: bool = False, exclusive_maximum: bool = False, integer: bool = False, help: str = '')

One parameter of a calibration method.

minimum and maximum bound the values the method accepts; low and high are the range worth offering in an editor.

coerce

Python
coerce(value: object) -> float

Validate a value for this parameter.

OnlineCalibrator

Bases: Protocol

A threshold on one score stream, learned with delayed feedback.

predict(key) issues a threshold when a prediction is made, and update(key, score) arrives once that prediction's score is known. A score above the threshold is a miss at the target level 1 - alpha.

ScoreHistory

Python
ScoreHistory(window: int, lookback: int = LOOKBACK)

The latest scores in arrival order, and the last window of them sorted.

Every calibrator of one score stream reads the same history. A calibrator learns a score in :meth:~aberrant.calibration.OnlineCalibrator.update before the score joins the history, so it adapts against the window it predicted from.

The history keeps the last max(window, lookback) scores, so its memory stays bounded on an endless stream.

Parameters:

Name Type Description Default
window int

Number of recent scores in the calibration window.

 required
lookback int

Number of recent scores that :meth:recent, :meth:recent_range and :meth:spread can reach.

 LOOKBACK

count property

Python
count: int

Number of scores seen.

window_sorted property

Python
window_sorted: list[float]

The calibration window in ascending order. Do not mutate.

max_abs_score property

Python
max_abs_score: float

Largest absolute score seen.

push

Python
push(score: float) -> None

Add a score, dropping the oldest one from the window when full.

recent

Python
recent(n: int) -> list[float]

The last n scores, oldest first.

recent_range

Python
recent_range(n: int) -> float

Largest absolute score among the last n scores.

spread

Python
spread(n: int) -> float

Range (max - min) of the last n scores, 0 for fewer than two values.

cap

Python
cap(q: float) -> float

Clip a threshold to the largest absolute score seen.

This also makes infinite thresholds finite. A threshold is unchanged before the first score, and NaN becomes the largest absolute score.

conformal_quantile

Python
conformal_quantile(beta: float) -> float

The ceil((n + 1) beta)-th smallest score of the window.

+inf when that rank exceeds the window, -inf when beta <= 0.