Stream Dataset API¶

The module-level functions operate on a process-local default DatasetManager. Construct a manager directly for explicit cache ownership or dependency injection.

Registry¶

Dataset ¶

Bases: Enum

Available datasets for anomaly detection experiments.

This enumeration provides all built-in datasets that can be loaded using the load() function. Each dataset is preprocessed for anomaly detection tasks with normal and anomalous samples.

Examples:

Python

from aberrant.stream.dataset import Dataset, get_dataset_info

info = get_dataset_info(Dataset.FRAUD)
assert info.n_features == 30

DatasetInfo `dataclass` ¶

Python

DatasetInfo(name: str, description: str, filename: str, n_samples: int, n_features: int, anomaly_rate: float, source: str, category: str, sha256: str)

Metadata for an anomaly detection dataset.

Attributes:

Name	Type	Description
`name`	`str`	Human-readable name of the dataset
`description`	`str`	Brief description of the dataset and its characteristics
`filename`	`str`	Name of the file in the GitHub release (without extension)
`n_samples`	`int`	Total number of samples in the dataset
`n_features`	`int`	Number of features/dimensions
`anomaly_rate`	`float`	Proportion of anomalous samples (0.0 to 1.0)
`source`	`str`	Original source or reference for the dataset
`category`	`str`	Dataset category (e.g., 'medical', 'security', 'benchmark')
`sha256`	`str`	Trusted SHA256 checksum for release artifact validation

DATASET_REGISTRY `module-attribute` ¶

Python

DATASET_REGISTRY: Mapping[Dataset, DatasetInfo] = MappingProxyType(_DATASET_REGISTRY)

get_dataset_info ¶

Python

get_dataset_info(dataset: Dataset) -> DatasetInfo

Get metadata information for a specific dataset.

Parameters:

Name	Type	Description	Default
`dataset`	`Dataset`	Dataset enum value	required

Returns:

Type	Description
`DatasetInfo`	DatasetInfo object with metadata

Raises:

Type	Description
`KeyError`	If dataset is not found in registry

list_available ¶

Python

list_available() -> dict[str, DatasetInfo]

List all available datasets with their metadata.

Returns:

Type	Description
`dict[str, DatasetInfo]`	Dictionary mapping dataset names to DatasetInfo objects

list_by_category ¶

Python

list_by_category(category: str) -> dict[str, DatasetInfo]

List datasets by category.

Parameters:

Name	Type	Description	Default
`category`	`str`	Category to filter by (e.g., 'medical', 'security', 'benchmark')	required

Returns:

Type	Description
`dict[str, DatasetInfo]`	Dictionary of datasets in the specified category

get_categories ¶

Python

get_categories() -> list[str]

Get all available dataset categories.

Returns:

Type	Description
`list[str]`	List of unique category names

Default-manager functions¶

load ¶

Python

load(dataset: Dataset, auto_download: bool = True, *, feature_prefix: str = 'feature_', label_column: str = 'y', feature_column: str = 'X', show_progress: bool = False) -> NpzStreamer

Return a validated NPZ stream from the default manager.

download ¶

Dataset transfer and artifact validation backends.

DownloadBackend ¶

Bases: Protocol

Transfer a URL into a caller-owned destination path.

download ¶

Python

download(url: str, destination: Path) -> None

Download one artifact or raise after exhausting retries.

UrlLibDownloadBackend ¶

Python

UrlLibDownloadBackend(*, retries: int = 3, timeout: float = 30.0, backoff_seconds: float = 1.0, show_progress: bool = False, logger: Logger | None = None)

Retrying urllib transfer backend with optional progress reporting.

Parameters:

Name	Type	Description	Default
`retries`	`int`	Positive maximum number of transfer attempts.	`3`
`timeout`	`float`	Positive per-request timeout in seconds.	`30.0`
`backoff_seconds`	`float`	Non-negative base delay for exponential retry backoff. Delay before retry `n` is `backoff_seconds * 2**(n - 1)`.	`1.0`
`show_progress`	`bool`	Display a byte progress bar when downloading.	`False`
`logger`	`Logger \| None`	Logger for retry warnings. `None` uses the module logger.	`None`

DatasetArtifactValidator ¶

Validate NPZ structure and trusted SHA256 metadata.

validate ¶

Python

validate(path: Path, info: DatasetInfo) -> str

Return the verified digest or raise for an invalid artifact.

list_cached ¶

Python

list_cached() -> dict[str, CacheEntry]

Return metadata for locally cached registered datasets.

clear_cache ¶

Python

clear_cache(dataset: Dataset | None = None) -> None

Remove one dataset or all registered datasets from the default cache.

get_cache_info ¶

Python

get_cache_info() -> CacheInfo

Return a typed default-cache summary.

get_default_manager ¶

Python

get_default_manager() -> DatasetManager

Return the process-local default manager.

set_cache_dir ¶

Python

set_cache_dir(cache_dir: str | Path) -> None

Replace the default manager with one using cache_dir.

Orchestration and cache values¶

DatasetManager ¶

Python

DatasetManager(cache_dir: str | Path | None = None, github_repo: str = 'OliverHennhoefer/nonconform', release_tag: str = 'v0.9.17-datasets', download_retries: int = 3, download_timeout: float = 30.0, retry_backoff_seconds: float = 1.0, cache_lock_timeout: float = 60.0, show_progress: bool = False, logger: Logger | None = None, download_backend: DownloadBackend | None = None, validator: DatasetArtifactValidator | None = None)

Coordinate registered dataset downloads, validation, caching, and streams.

Parameters:

Name	Type	Description	Default
`cache_dir`	`str \| Path \| None`	Artifact-cache directory. `None` uses `~/.aberrant/datasets`.	`None`
`github_repo`	`str`	GitHub `owner/repository` containing the dataset release.	`'OliverHennhoefer/nonconform'`
`release_tag`	`str`	Release tag used in artifact URLs and cache metadata.	`'v0.9.17-datasets'`
`download_retries`	`int`	Transfer attempts for the default download backend.	`3`
`download_timeout`	`float`	Per-request timeout in seconds for the default download backend.	`30.0`
`retry_backoff_seconds`	`float`	Base seconds for the default backend's exponential retry backoff.	`1.0`
`cache_lock_timeout`	`float`	Seconds to wait for cross-process cache metadata transactions.	`60.0`
`show_progress`	`bool`	Show transfer progress in the default download backend. Row-iteration progress is configured separately in `load`.	`False`
`logger`	`Logger \| None`	Logger for retries and manager diagnostics. `None` uses the module logger.	`None`
`download_backend`	`DownloadBackend \| None`	Injected transfer implementation. When supplied, the default backend settings above do not configure it.	`None`
`validator`	`DatasetArtifactValidator \| None`	Injected NPZ and SHA-256 validator. `None` uses `DatasetArtifactValidator`.	`None`

cache_dir `property` ¶

Python

cache_dir: Path

Return the configured cache directory.

download ¶

Python

download(dataset: Dataset, force: bool = False) -> Path

Download, validate, and publish one registered dataset safely.

load ¶

Python

load(dataset: Dataset, auto_download: bool = True, *, feature_prefix: str = 'feature_', label_column: str = 'y', feature_column: str = 'X', show_progress: bool = False) -> NpzStreamer

Return a typed NPZ stream for one validated cached dataset.

list_cached ¶

Python

list_cached() -> dict[str, CacheEntry]

Return valid metadata entries for artifacts that exist on disk.

clear_cache ¶

Python

clear_cache(dataset: Dataset | None = None) -> None

Remove one registered artifact or all registered artifacts.

get_cache_size ¶

Python

get_cache_size() -> int

Return total cached NPZ bytes.

DatasetCacheStore ¶

Python

DatasetCacheStore(cache_dir: Path, *, lock_timeout: float = 60.0)

Own cache paths, metadata transactions, and artifact publication.

Parameters:

Name	Type	Description	Default
`cache_dir`	`Path`	Directory in which artifacts, metadata, and the process lock are stored. It is created, including parents, when necessary.	required
`lock_timeout`	`float`	Positive seconds to wait for the cross-process metadata lock before the underlying file lock raises a timeout.	`60.0`

path ¶

Python

path(filename: str) -> Path

Return the cache path for an NPZ artifact name.

read ¶

Python

read() -> CacheMetadata

Read and validate an immutable metadata snapshot.

publish ¶

Python

publish(*, dataset_name: str, temporary_path: Path, destination: Path, entry: CacheEntry) -> None

Replace an artifact, then merge metadata under one process lock.

remove ¶

Python

remove(dataset_name: str, artifact_path: Path) -> None

Remove one artifact and its metadata under one process lock.

clear ¶

Python

clear(artifacts: Mapping[str, Path]) -> None

Remove the supplied owned artifacts and reset metadata.

size `staticmethod` ¶

Python

size(artifacts: Mapping[str, Path]) -> int

Return total bytes occupied by the supplied owned artifacts.

CacheEntry `dataclass` ¶

Python

CacheEntry(sha256: str, size: int, release_tag: str)

Immutable metadata for one cached artifact.

Attributes:

Name	Type	Description
`sha256`	`str`	Verified 64-character hexadecimal SHA-256 digest.
`size`	`int`	Artifact size in bytes.
`release_tag`	`str`	Dataset release tag from which the artifact was obtained.

CacheMetadata `dataclass` ¶

Python

CacheMetadata(version: str, datasets: Mapping[str, CacheEntry])

Immutable cache metadata snapshot.

Attributes:

Name	Type	Description
`version`	`str`	On-disk metadata schema version.
`datasets`	`Mapping[str, CacheEntry]`	Read-only mapping from registry value to cache entry.

CacheInfo `dataclass` ¶

Python

CacheInfo(directory: Path, size_bytes: int, datasets: tuple[str, ...])

Typed summary of the default dataset cache.

Attributes:

Name	Type	Description
`directory`	`Path`	Configured cache directory.
`size_bytes`	`int`	Total bytes occupied by registered cached artifacts.
`datasets`	`tuple[str, ...]`	Registry values whose artifacts are present in the cache.

Download and validation¶

DownloadBackend ¶

Bases: Protocol

Transfer a URL into a caller-owned destination path.

download ¶

Python

download(url: str, destination: Path) -> None

Download one artifact or raise after exhausting retries.

UrlLibDownloadBackend ¶

Python

UrlLibDownloadBackend(*, retries: int = 3, timeout: float = 30.0, backoff_seconds: float = 1.0, show_progress: bool = False, logger: Logger | None = None)

Retrying urllib transfer backend with optional progress reporting.

Parameters:

Name	Type	Description	Default
`retries`	`int`	Positive maximum number of transfer attempts.	`3`
`timeout`	`float`	Positive per-request timeout in seconds.	`30.0`
`backoff_seconds`	`float`	Non-negative base delay for exponential retry backoff. Delay before retry `n` is `backoff_seconds * 2**(n - 1)`.	`1.0`
`show_progress`	`bool`	Display a byte progress bar when downloading.	`False`
`logger`	`Logger \| None`	Logger for retry warnings. `None` uses the module logger.	`None`

DatasetArtifactValidator ¶

Validate NPZ structure and trusted SHA256 metadata.

validate ¶

Python

validate(path: Path, info: DatasetInfo) -> str

Return the verified digest or raise for an invalid artifact.

Streams¶

DatasetStream ¶

Bases: Protocol

Common interface for dataset sample streams.

stream ¶

Python

stream() -> Iterator[Sample]

Yield individual samples.

get_metadata ¶

Python

get_metadata() -> DatasetInfo | None

Return registered dataset metadata when available.

Sample `module-attribute` ¶

Python

Sample: TypeAlias = tuple[dict[str, float], object]

NpzStreamer ¶

Python

NpzStreamer(file_path: str | Path, dataset_info: DatasetInfo | None = None, *, feature_prefix: str = 'feature_', label_column: str = 'y', feature_column: str = 'X', show_progress: bool = False)

Row-wise iterator over an NPZ dataset artifact.

Parameters:

Name	Type	Description	Default
`file_path`	`str \| Path`	Path to the NPZ archive.	required
`dataset_info`	`DatasetInfo \| None`	Optional immutable registry metadata returned by `get_metadata` and used in progress text.	`None`
`feature_prefix`	`str`	Prefix for zero-based generated feature names.	`'feature_'`
`label_column`	`str`	Name of the label array in the archive.	`'y'`
`feature_column`	`str`	Name of the two-dimensional feature array in the archive.	`'X'`
`show_progress`	`bool`	Display row-iteration progress.	`False`

stream ¶

Python

stream() -> Iterator[Sample]

Open the artifact and yield samples row by row.

get_metadata ¶

Python

get_metadata() -> DatasetInfo | None

Return registered metadata when supplied by the loader.

BatchStreamer ¶

Python

BatchStreamer(base_streamer: DatasetStream, batch_size: int = 1000)

Batch samples from any typed dataset stream.

Parameters:

Name	Type	Description	Default
`base_streamer`	`DatasetStream`	Object satisfying `DatasetStream`.	required
`batch_size`	`int`	Positive maximum samples yielded per batch. The final batch can be smaller.	`1000`

stream ¶

Python

stream() -> Iterator[tuple[list[dict[str, float]], list[object]]]

Yield feature and label batches.

get_metadata ¶

Python

get_metadata() -> DatasetInfo | None

Return metadata from the underlying stream.

Stream Dataset API¶

Registry¶

Dataset ¶

DatasetInfo dataclass ¶

DATASET_REGISTRY module-attribute ¶

get_dataset_info ¶

list_available ¶

list_by_category ¶

get_categories ¶

Default-manager functions¶

load ¶

download ¶

DownloadBackend ¶

download ¶

UrlLibDownloadBackend ¶

DatasetArtifactValidator ¶

validate ¶

list_cached ¶

clear_cache ¶

get_cache_info ¶

get_default_manager ¶

set_cache_dir ¶

Orchestration and cache values¶

DatasetManager ¶

cache_dir property ¶

download ¶

load ¶

list_cached ¶

clear_cache ¶

get_cache_size ¶

DatasetCacheStore ¶

path ¶

read ¶

publish ¶

remove ¶

clear ¶

size staticmethod ¶

CacheEntry dataclass ¶

CacheMetadata dataclass ¶

CacheInfo dataclass ¶

Download and validation¶

DownloadBackend ¶

download ¶

UrlLibDownloadBackend ¶

DatasetArtifactValidator ¶

validate ¶

Streams¶

DatasetStream ¶

stream ¶

get_metadata ¶

Sample module-attribute ¶

NpzStreamer ¶

stream ¶

get_metadata ¶

BatchStreamer ¶

stream ¶

get_metadata ¶

DatasetInfo `dataclass` ¶

DATASET_REGISTRY `module-attribute` ¶

cache_dir `property` ¶

size `staticmethod` ¶

CacheEntry `dataclass` ¶

CacheMetadata `dataclass` ¶

CacheInfo `dataclass` ¶

Sample `module-attribute` ¶