User guide¶
nonconform supports discovery and sequential workflows built on anomaly scores:
- Discovery control: construct p-values or e-values for a fixed family and select anomalies with a justified FDR procedure.
- Threshold certification: construct a simultaneous false-alarm/FPR curve and select a score threshold for future inliers.
- Sequential change monitoring: generate randomized sequential conformal p-values, accumulate evidence with a martingale, and trigger configured alarms.
Choose the workflow first. Their evidence, error targets, and evaluation metrics are different.
Start from your task¶
| Task | Read first | Then |
|---|---|---|
| Select anomalies in one batch | Conformal inference | FDR control |
| Set a per-inlier raw-score false-alarm target | FDR control | Common workflows |
| Stabilize repeated split selections | Derandomized e-values | Batch evaluation |
| Monitor an ordered stream for change | Exchangeability martingales | Streaming evaluation |
| Handle modeled covariate shift | Weighted conformal | FDR control |
| Choose split, CV, jackknife, or bootstrap | Conformalization strategies | Choosing strategies |
| Integrate a detector | Detector compatibility | Common workflows |
| Diagnose a failure | Troubleshooting | Input validation |
Foundations¶
| Page | Purpose |
|---|---|
| Statistical concepts | Short definitions of p-values, exchangeability, FDR, power, covariate shift, and Ville control |
| Conformal inference | Rank construction, data roles, marginal and conditional validity, ties, and batch versus sequential p-values |
| Conformalization strategies | Exact mechanics and statistical scope of Split, DerandomizedSplits, CrossValidation, jackknife, and bootstrap |
| Choosing strategies | Decision process based on validity needs, resolution, model-fit budget, and empirical evaluation |
Applied workflows¶
| Page | Purpose |
|---|---|
| Detector compatibility | scikit-learn, PyOD, custom protocols, blocked batch-adaptive models, and score polarity |
| Weighted conformal | Covariate-shift assumptions, density-ratio estimators, weight diagnostics, and WCS |
| FDR control | Raw-score FPR certificates, BH, BY, WCS, derandomized e-values, post-hoc FDP certificates, repeated batches, and online FDR distinctions |
| Derandomized e-values | DerandomizedSplits with automatic repetitions, e-values, and e-BH |
| Exchangeability martingales | Sequential randomized ranks, betting martingales, alarms, and Ville scope |
Evaluation and operations¶
| Page | Purpose |
|---|---|
| Batch evaluation | Reproducible labeled families with oddball and correct FDP/power bookkeeping |
| Streaming evaluation | False-alarm, detection-delay, online-testing, and window-family designs |
| Input validation | Enforced constraints, calibration resolution, fitted state, and weighted batch identity |
| Best practices | Leakage prevention, family and episode design, reproducibility, and production review |
| Logging | Actual logger namespaces and progress controls |
| Troubleshooting | Symptom-led diagnosis without weakening guarantees post hoc |
Guarantees are conditional statements
Standard split-conformal p-values require exchangeability of calibration and true-null test scores conditional on a fixed training-only scorer. FDR control additionally requires the dependence assumptions of the selection procedure. Weighted workflows require a correct shift model, overlap, and suitable weights. FPR certificates require clean, i.i.d. calibration and future inlier scores from the same distribution conditional on an independently fitted score map. Sequential Ville guarantees require conditionally valid sequential p-values and a valid e-process. Passing API validation does not establish any of these assumptions.
Review checklist¶
Before relying on a result, confirm:
- the null population, testing family, or monitoring episode was defined in advance;
- fitting, calibration, tuning, and final evaluation roles are separated;
- learned preprocessing is part of the fitted scoring construction;
- detector score polarity and fixed-state behavior are verified;
- empirical p-value resolution is adequate for the actual family;
- the selected FDR, FDP-bound, weighted, or sequential theorem matches the implementation and data design;
- labeled evaluation data did not choose the result later reported on it; and
- assumptions and failure modes accompany every statistical claim.
For agents and advanced users, the API reference exposes signatures and public docstrings, while Common workflows provides independently runnable examples.