Skip to content
Edit this page View source of this page

Common workflows

This page collects complete, independent examples for the most common nonconform tasks. Every code block includes its own imports and data setup, so you can copy and run any example by itself.

Batch discovery control

Use select(...) when a batch of observations forms one multiple-testing family and you want a Boolean mask of discoveries. With an unweighted p-value strategy such as Split, select(...) computes conformal p-values and applies the Benjamini-Hochberg (BH) procedure at the requested false discovery rate (FDR) level. DerandomizedSplits instead uses e-values and e-BH, as shown below.

import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split

rng = np.random.default_rng(42)
x_reference = rng.normal(size=(800, 4))
x_test = np.vstack(
    [
        rng.normal(size=(18, 4)),
        rng.normal(loc=5.0, size=(2, 4)),
    ]
)

detector = ConformalDetector(
    detector=IsolationForest(random_state=42),
    strategy=Split(n_calib=0.3),
    seed=42,
)
detector.fit(x_reference)
selected = detector.select(x_test, alpha=0.05)

print(np.flatnonzero(selected))
print(f"discoveries: {selected.sum()}")

alpha=0.05 is a target FDR level, not a statement that every selected observation has a 95% probability of being anomalous. The FDR guarantee is a property of the selection procedure under its assumptions and across the declared testing family.

Pass method="storey" to use Storey's adaptive BH, which estimates the share of true nulls and can discover more when anomalies are common. After a call, detector.last_result.adjusted_p_values(method=...) returns the adjusted p-values whose <= alpha mask reproduces the selection. See Adaptive BH with Storey's estimator for its scope.

When groups such as sites or sensors have different normal score levels, fit with fit(x_reference, groups=site_labels) and pass the test labels to compute_p_values(..., groups=...) or select(..., groups=...). Each test point is then ranked only among calibration scores from its own group. See Group-conditional (Mondrian) p-values.

Raw-score false-alarm control

Use an FPRCertificate when the deployment requirement is a per-inlier false-alarm target rather than an expected false-discovery proportion.

import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split

rng = np.random.default_rng(42)
x_reference = rng.normal(size=(3_000, 4))
x_test = np.vstack(
    [rng.normal(size=(18, 4)), rng.normal(loc=5.0, size=(2, 4))]
)

detector = ConformalDetector(
    detector=IsolationForest(random_state=42),
    strategy=Split(n_calib=1_000),
    seed=42,
).fit(x_reference)

certificate = detector.fpr_bounds(confidence=0.95)
scores = detector.score_samples(x_test)
selected = certificate.select(scores, target_fpr=0.05)

print("threshold:", certificate.threshold_for(0.05))
print("selected indices:", np.flatnonzero(selected))
print(certificate.to_frame())

The default table includes every decision considered by threshold_for(), including a threshold above the largest calibration score when representable, and the empty rule at inf, whose FPR upper bound is zero. Use certificate.select(...) when applying a threshold reloaded from storage so lower-precision score ties remain excluded. Certificates can also be copied or pickled while retaining their prepared KS band and immutable calibration evidence.

With probability at least confidence over the clean calibration draw, conditional on the independently fitted score map, the certificate covers the population inlier false-alarm rate simultaneously at every raw-score threshold. Threshold inversion is therefore part of the certified workflow: on that event, the selected threshold has a future-inlier false-alarm probability at most target_fpr. This does not control FDR, realized FDP, recall, realized finite-batch proportions, or the probability of at least one false alarm in an arbitrarily large batch.

The band uses the deterministic finite-sample one-sided KS distribution. With 1,000 calibration scores at 95% confidence, its finite-threshold floor is about 3.85%, so the 5% target is feasible. A target below the floor returns an infinite threshold and an empty selection. Native FPR certificates support unweighted Split calibration, including detached calibration, independently of the configured p-value estimator. Coverage requires clean calibration and future inlier scores that are i.i.d. from the same deployment inlier distribution, conditional on a scoring map fixed independently of calibration. Exchangeability alone is insufficient; native scope checks do not establish these assumptions.

Inspect p-values and scores

Call compute_p_values(...) when you need the p-values themselves. The last_result snapshot also contains the test scores and calibration scores used for that call.

import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split

rng = np.random.default_rng(42)
x_reference = rng.normal(size=(600, 4))
x_test = np.vstack(
    [rng.normal(size=(8, 4)), rng.normal(loc=5.0, size=(2, 4))]
)

detector = ConformalDetector(
    detector=IsolationForest(random_state=42),
    strategy=Split(n_calib=0.3),
    seed=42,
).fit(x_reference)

p_values = detector.compute_p_values(x_test)
result = detector.last_result
assert result is not None
assert result.p_values is not None

print(np.column_stack([result.test_scores, p_values]))

last_result describes the latest scoring call

Calling score_samples(...), compute_p_value(...), compute_p_values(...), or select(...) replaces the cached result. Retrieve last_result after the call you want to inspect. The returned object is a defensive copy. When the detector uses DerandomizedSplits, inspect last_selection_result after select() instead; last_result is cleared.

Repeated-split selection with e-values

DerandomizedSplits fits repeated training/calibration splits, constructs conformal e-values separately for every split, averages the evidence uniformly, and applies e-BH once. It manages the replicas and calibration rows internally.

import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, DerandomizedSplits

rng = np.random.default_rng(42)
x_reference = rng.normal(size=(800, 4))
x_test = np.vstack(
    [rng.normal(size=(15, 4)), rng.normal(loc=4.5, size=(5, 4))]
)

detector = ConformalDetector(
    detector=IsolationForest(),
    strategy=DerandomizedSplits(n_repetitions=5, n_calib=0.3),
    seed=42,
).fit(x_reference)
mask = detector.select(x_test, alpha=0.05)
e_result = detector.last_selection_result
assert e_result is not None
print("selected indices:", np.flatnonzero(mask))
print("inner alpha:", e_result.alpha_bh)

Set advanced options on the strategy: alpha_bh=None uses alpha / 10, and tie_seed=None automatically derives a tie stream separate from fitting. An explicit tie_seed overrides only tie handling. With a detector seed, the splits and randomized secondary ranks are reproducible for deterministic scoring. Without one, randomness is generated once per fit. Fix the procedure and inner threshold before inspecting test evidence.

select() returns a boolean mask and stores a defensive EValueSelectionResult in last_selection_result. Its fields include e_values, selected, e_threshold, alpha, alpha_bh, n_repetitions, n_calibration, and the effective tie_seed. Raw scoring and refitting clear selection diagnostics. last_result is None after e-value selection.

score_samples() still aggregates raw scores using aggregation, but selection always constructs evidence from the separate model/calibration pairs. P-value methods are unsupported for this strategy. Use unweighted integrated fitting; non-Empirical estimation is rejected because p-value estimation is unused.

For existing individual split snapshots, select_conformal_e_values(...) remains available. It checks native provenance and identical test-batch content and ordering, but cannot track modifications to snapshot score arrays. Its default tie_seed=None still rejects tied scores.

Guarantee scope

Requires exchangeable normal reference and null test observations, the same fixed test family in the same row order for every repetition, valid integrated unweighted splits, and one final e-BH application. Randomized secondary ranks provide strict ordering without perturbing unequal scores. The aggregate null-evidence condition supports e-BH; individual constructed values need not be ordinary e-values. Uniform aggregation is the only supported evidence aggregation. Realized FDP in one batch is not expected FDR, and aggregation is not a promise of improved stability on every dataset.

See the Shuttle example and advanced result-list workflow.

Detached calibration for a pre-fitted detector

Use calibrate(...) with Split when the base detector has already been fitted on a proper training set and you have a separate calibration set. The scoring rule must remain fixed while calibration and test observations are scored.

import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split

rng = np.random.default_rng(42)
x_fit = rng.normal(size=(400, 4))
x_calibration = rng.normal(size=(200, 4))
x_test = np.vstack(
    [rng.normal(size=(18, 4)), rng.normal(loc=5.0, size=(2, 4))]
)

base_detector = IsolationForest(random_state=42).fit(x_fit)
detector = ConformalDetector(
    detector=base_detector,
    strategy=Split(n_calib=0.3),
    seed=42,
)
detector.calibrate(x_calibration)

p_values = detector.compute_p_values(x_test)
print(p_values)

n_calib is not used to split data in this detached workflow, because the entire array passed to calibrate(...) is the calibration set. It remains a required Split constructor argument for consistency with the strategy API.

Weighted conformal selection under covariate shift

Weighted mode estimates a target-to-calibration density ratio and uses weighted conformal p-values. select(...) then applies weighted conformalized selection (WCS), not ordinary BH.

import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split, logistic_weight_estimator

rng = np.random.default_rng(42)
x_reference = rng.normal(size=(2_000, 3))
x_test = np.vstack(
    [
        rng.normal(loc=0.5, size=(18, 3)),
        rng.normal(loc=6.0, size=(2, 3)),
    ]
)

detector = ConformalDetector(
    detector=IsolationForest(n_estimators=50, random_state=42),
    strategy=Split(n_calib=0.4),
    weight_estimator=logistic_weight_estimator(),
    seed=42,
).fit(x_reference)

selected = detector.select(x_test, alpha=0.1)
result = detector.last_result
assert result is not None
assert result.calib_weights is not None
assert result.test_weights is not None

print("selected indices:", np.flatnonzero(selected))
print(
    "weight ranges:",
    (result.calib_weights.min(), result.calib_weights.max()),
    (result.test_weights.min(), result.test_weights.max()),
)

This workflow requires more than a classifier that separates calibration and test samples. Its validity relies on an appropriate covariate-shift model, support overlap, and sufficiently reliable density-ratio estimates. See Weighted conformal inference for the assumptions and diagnostics.

Reuse explicitly prepared weights

Weight fitting is batch-specific. If you want that state transition to be explicit, prepare weights for the exact batch and disable refitting on the subsequent call.

import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split, logistic_weight_estimator

rng = np.random.default_rng(42)
x_reference = rng.normal(size=(800, 3))
x_test = rng.normal(loc=0.6, size=(40, 3))

detector = ConformalDetector(
    detector=IsolationForest(random_state=42),
    strategy=Split(n_calib=0.3),
    weight_estimator=logistic_weight_estimator(),
    seed=42,
).fit(x_reference)

detector.prepare_weights_for(x_test)
p_values = detector.compute_p_values(x_test, refit_weights=False)
print(p_values[:5])

By default, nonconform verifies both the size and content of the prepared batch. Preparing weights for one batch and applying them to another is an error.

Conditionally calibrated conformal p-values

ConditionalEmpirical applies one of the package's conditional calibration maps to classical empirical conformal p-values. The available maps include finite-sample and asymptotic options. The class is available from nonconform.scoring and is intentionally unsupported in weighted mode.

import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split
from nonconform.scoring import ConditionalEmpirical

rng = np.random.default_rng(42)
x_reference = rng.normal(size=(600, 4))
x_test = np.vstack(
    [rng.normal(size=(18, 4)), rng.normal(loc=5.0, size=(2, 4))]
)

detector = ConformalDetector(
    detector=IsolationForest(random_state=42),
    strategy=Split(n_calib=0.3),
    estimation=ConditionalEmpirical(delta=0.05, method="dkwm"),
    seed=42,
).fit(x_reference)

p_values = detector.compute_p_values(x_test)
print(p_values)

See Conformal inference before choosing a conditional calibration method. delta is the failure probability associated with the calibration map; it is not the FDR target alpha.

Sequential change monitoring

For a stream, use randomized sequential conformal ranks and a conformal martingale. The following example fits a split conformal detector, freezes its fitted scoring rule, primes rank history with the calibration scores, and then processes stream observations in order.

import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split
from nonconform.martingales import AlarmConfig, SimpleJumperMartingale
from nonconform.monitoring import ExchangeabilityMonitor

rng = np.random.default_rng(42)
x_reference = rng.normal(size=(600, 3))
x_stream = np.vstack(
    [
        rng.normal(size=(30, 3)),
        rng.normal(loc=3.0, size=(20, 3)),
    ]
)

detector = ConformalDetector(
    detector=IsolationForest(random_state=42),
    strategy=Split(n_calib=0.3),
    seed=42,
).fit(x_reference)

monitor = ExchangeabilityMonitor.from_split_detector(
    detector,
    martingale=SimpleJumperMartingale(
        alarm_config=AlarmConfig(ville_threshold=20.0)
    ),
    seed=42,
)
states = monitor.update_many(x_stream)
first_alarm = next((state for state in states if state.triggered_alarms), None)
if first_alarm is None:
    print("No alarm in this finite stream")
else:
    print(first_alarm.evidence_step, first_alarm.triggered_alarms)

Batch p-values are not sequential conformal p-values

Do not feed repeated calls to ConformalDetector.compute_p_value(...) into a martingale and infer sequential validity. ExchangeabilityMonitor constructs the required sequential randomized ranks. Its null guarantee requires an exchangeable score sequence conditional on the frozen scoring rule. Do not refit the scorer during a monitoring episode.

At a Ville threshold of 1 / alpha, a valid nonnegative martingale that starts at one has probability at most alpha of ever crossing the threshold under the null. CUSUM and Shiryaev-Roberts statistics are also exposed, but their thresholds are change-evidence triggers and need separate calibration; they do not automatically inherit the Ville guarantee. See Exchangeability martingales.

Certify realized FDP across thresholds

For a fitted unweighted empirical Split detector and one fixed test family:

import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split

rng = np.random.default_rng(42)
x_reference = rng.normal(size=(3_000, 4))
x_family = np.vstack(
    [rng.normal(size=(160, 4)), rng.normal(loc=5.0, size=(40, 4))]
)
y_family = np.r_[np.zeros(160, dtype=int), np.ones(40, dtype=int)]

detector = ConformalDetector(
    detector=IsolationForest(random_state=42),
    strategy=Split(n_calib=1_000),
    seed=42,
).fit(x_reference)

certificate = detector.fdp_bounds(x_family, confidence=0.95)
print(certificate.to_frame(thresholds=[0.01, 0.05, 0.1]))
cutoff = 0.01
mask = certificate.select(cutoff)  # p-value cutoff; returns a NumPy mask
print("discoveries:", mask.sum())
print("FDP upper bound:", certificate.bound_at(cutoff))

Labels are included only to inspect the synthetic example; certification does not use them. Reserving 1,000 calibration observations and including 40 shifted points gives this example enough information for a useful nonempty bound. The printed bound can vary because certificate sampling is fresh.

If p-values were already computed, detector.last_result.fdp_bounds(confidence=...) certifies that native snapshot without rescoring. Certificates own immutable state and survive refitting. The detector and result calls accept only confidence; they prepare one default MC-THC envelope with 1,000 fresh draws and boosting enabled. Threshold queries reuse it without resampling. confidence is simultaneous coverage of realized FDP bounds jointly over data and fresh Monte Carlo sampling, while select(x, alpha=...) targets expected FDR. It is not a guarantee conditional on every realized simulated envelope. Keep the scoring rule and family fixed, and do not cherry-pick repeated certificate constructions. Expert settings belong to FDPCertificate.from_p_values(...) and must be chosen before inspecting the curve; a fixed simulation seed gives reproducibility, not a separate conditional coverage guarantee. See the FDP guide for assumptions, method options, and migration from the removed FDP functions.

Score direction for custom detectors

nonconform normalizes scores internally so that larger values mean more anomalous observations. Built-in adapters resolve the convention for supported scikit-learn and PyOD detectors. For a custom detector, specify the convention explicitly if it cannot be inferred safely:

import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split

rng = np.random.default_rng(42)
x_reference = rng.normal(size=(300, 2))
x_test = rng.normal(size=(5, 2))

detector = ConformalDetector(
    detector=IsolationForest(random_state=42),
    strategy=Split(n_calib=0.3),
    score_polarity="higher_is_normal",
    seed=42,
).fit(x_reference)

print(detector.compute_p_values(x_test))

The explicit setting in this example matches scikit-learn's IsolationForest.decision_function, where lower values are more abnormal.