Skip to content
Edit this page View source of this page

nonconform

nonconform: Conformal Anomaly Detection in Python

Calibrate scores. Control discoveries. Monitor change.

nonconform turns anomaly scores into conformal evidence for discovery and sequential workflows:

  • Discovery control: compute conformal p-values and select anomalies with false discovery rate (FDR) control.
  • Threshold certification: build a simultaneous false-alarm/FPR certificate and select a score threshold for future inliers.
  • Sequential change monitoring: transform a stream into randomized sequential conformal p-values and accumulate evidence against exchangeability with conformal martingales.

These workflows can wrap supported scikit-learn estimators, PyOD models, or a custom detector that implements the documented protocol.

Batch discovery control

Use select(...) when a fixed batch of observations must become anomaly decisions. The example below needs only the core installation.

import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split

rng = np.random.default_rng(42)
x_train = rng.normal(size=(3_000, 2))
x_test = np.vstack([
    rng.normal(size=(200, 2)),
    rng.normal(loc=5.0, size=(20, 2)),
])

detector = ConformalDetector(
    detector=IsolationForest(random_state=42),
    strategy=Split(n_calib=1_000),
    score_polarity="auto",
    seed=42,
).fit(x_train)

discoveries = detector.select(x_test, alpha=0.05)
p_values = detector.last_result.p_values

print(f"Selected {discoveries.sum()} of {len(x_test)} observations")
print(f"Smallest p-value: {p_values.min():.4f}")

alpha=0.05 is the target FDR level for this batch, not an anomaly-score threshold and not a promise about the realized false discovery proportion in this particular run.

Raw-score false-alarm control

When the requirement is to limit false alarms among future inliers, use the raw-score certificate rather than interpreting an FDR target as a score threshold:

certificate = detector.fpr_bounds(confidence=0.95)
scores = detector.score_samples(x_test)
selected = certificate.select(scores, target_fpr=0.05)

print(f"Certified threshold: {certificate.threshold_for(0.05)}")
print(f"Selected {selected.sum()} observations")

With probability at least 95% over the clean calibration draw, conditional on the independently fitted score map, the certificate bounds the population inlier false-alarm rate simultaneously at every threshold. On that event, the chosen threshold has a future-inlier false-alarm probability at most 5%. Calibration and future inlier scores must be i.i.d. from the same distribution; exchangeability alone is insufficient.

The deterministic finite-sample one-sided KS band has a finite-threshold floor of about 3.85% with 1,000 calibration scores at 95% confidence. A target below that floor yields an infinite threshold and an empty selection. Native certificates support unweighted Split calibration independently of the p-value estimator. They do not guarantee realized finite-batch proportions, FDR, FDP, recall, or sequential Ville false-alarm control.

Sequential change monitoring

Use ExchangeabilityMonitor when observations arrive in order and the goal is to accumulate evidence that the stream has stopped being exchangeable with its reference history.

import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split
from nonconform.martingales import AlarmConfig, SimpleJumperMartingale
from nonconform.monitoring import ExchangeabilityMonitor

rng = np.random.default_rng(42)
x_train = rng.normal(size=(1_000, 2))
x_stream = np.vstack([
    rng.normal(size=(50, 2)),
    rng.normal(loc=3.0, size=(50, 2)),
])

detector = ConformalDetector(
    detector=IsolationForest(random_state=42),
    strategy=Split(n_calib=0.3),
    score_polarity="auto",
    seed=42,
).fit(x_train)

monitor = ExchangeabilityMonitor.from_split_detector(
    detector,
    martingale=SimpleJumperMartingale(
        alarm_config=AlarmConfig(restarted_ville_threshold=20.0)
    ),
    seed=42,
)

for x_t in x_stream:
    state = monitor.update(x_t)
    if "restarted_ville" in state.triggered_alarms:
        print(f"Change alarm at step {state.evidence_step}")
        break
else:
    print("No alarm in this finite stream")

A Ville threshold of 20 bounds the probability of ever crossing that threshold by 0.05 on one valid null stream. It does not control FDR across streams.

Guarantee scope

Guarantees are assumption-dependent

Standard conformal workflows require the calibration data and null test cases to be exchangeable relative to a scoring rule fixed without using the calibration or test outcomes. BH selection additionally requires valid p-values and its dependence conditions. Weighted workflows require the stated covariate-shift model, support overlap, and reliable importance weights. FPR certificates require clean, i.i.d. calibration and future inlier scores from the same distribution conditional on an independently fitted score map. Sequential Ville guarantees require conditionally valid sequential conformal p-values.

nonconform calibrates detector scores. It cannot make an unsuitable detector, contaminated reference set, adaptive analysis, or mismatched data collection process valid.

Workflow Start here Main output
Fixed batch of anomaly candidates Quick Start Conformal p-values and an FDR-controlled Boolean mask
Fixed raw-score false-alarm target FDR Control A simultaneous FPR curve and target-driven threshold
Ordered stream monitored for change Exchangeability Martingales Sequential p-values, e-values, evidence statistics, and configured alarms
Covariate shift between calibration and test Weighted Conformal Weighted p-values and WCS selections
Groups with different normal behavior, or a shifted group mix Group-conditional p-values Per-group conformal p-values and one BH selection
Custom or third-party detector Detector Compatibility A validated, anomaly-oriented score interface

Installation

pip install nonconform
uv add nonconform

See Installation for optional detector, dataset, probabilistic-estimation, and online-FDR extras.

Documentation map

Citation

If you use nonconform in academic work, cite the accompanying paper:

@misc{hennhoefer2026,
  title={Conformal Anomaly Detection in Python: Moving Beyond Heuristic Thresholds with 'nonconform'},
  author={Oliver Hennhöfer and Maximilian Kirsch and Christine Preisach},
  year={2026},
  eprint={2605.13642},
  archivePrefix={arXiv},
  primaryClass={stat.ML},
  url={https://arxiv.org/abs/2605.13642},
}