Skip to content

Alpha Spending Methods

Alpha spending functions provide flexible control of Type I error rate in sequential hypothesis testing, allowing the number and timing of interim analyses to be determined adaptively during the trial.

Overview

The alpha spending approach, developed by Lan and DeMets (1983), overcomes key limitations of traditional group sequential methods by not requiring the total number of analyses or their exact timing to be specified in advance. This flexibility is particularly valuable in clinical trials where interim analyses may be needed at unplanned times.

Available Methods

Alpha Spending Function

online_fdr.spending.alpha_spending.AlphaSpending

Bases: AbstractSequentialTest

Alpha Spending Function for sequential hypothesis testing with flexible interim analyses.

The alpha spending function approach, developed by Lan and DeMets (1983), provides flexible group sequential boundaries that control Type I error rate while allowing the number and timing of interim analyses to be determined adaptively during the trial.

This method overcomes key limitations of traditional group sequential methods by not requiring the total number of analyses or their exact timing to be specified in advance. It's particularly valuable in clinical trials where interim analyses may be needed at unplanned times for ethical or scientific reasons.

The approach works by "spending" portions of the overall alpha budget at each interim analysis according to a pre-specified spending function, ensuring that the cumulative Type I error rate never exceeds the target level.

Parameters:

Name Type Description Default
alpha float

Target Type I error rate (e.g., 0.05 for 5% error rate). Must be in (0, 1).

required
spend_func AbstractSpendFunc

Alpha spending function that determines how to allocate alpha across interim analyses. Must inherit from AbstractSpendFunc.

required

Attributes:

Name Type Description
alpha0 float

Original target Type I error rate.

rule AbstractSpendFunc

The spending function used to determine alpha allocation.

num_test int

Number of tests/analyses conducted so far.

alpha float

Current alpha level for the next test.

Examples:

>>> from online_fdr.spending.functions.bonferroni import BonferroniSpendFunc
>>> # Create Bonferroni spending function for 5 planned analyses
>>> spend_func = BonferroniSpendFunc(max_analyses=5)
>>> alpha_spending = AlphaSpending(alpha=0.05, spend_func=spend_func)
>>> # Test p-values sequentially
>>> decision1 = alpha_spending.test_one(0.01)  # First interim analysis
>>> decision2 = alpha_spending.test_one(0.03)  # Second interim analysis
References

Lan, K. K. Gordon, and D. L. DeMets (1983). "Discrete Sequential Boundaries for Clinical Trials." Biometrika, 70(3):659-663.

DeMets, D. L., and K. K. Gordon Lan (1994). "Interim Analysis: The Alpha Spending Function Approach." Statistics in Medicine, 13(13-14):1341-1352.

Jennison, C., and B. W. Turnbull (1999). "Group Sequential Methods with Applications to Clinical Trials." Chapman and Hall/CRC.

Source code in online_fdr/spending/alpha_spending.py
class AlphaSpending(AbstractSequentialTest):
    """Alpha Spending Function for sequential hypothesis testing with flexible interim analyses.

    The alpha spending function approach, developed by Lan and DeMets (1983), provides
    flexible group sequential boundaries that control Type I error rate while allowing
    the number and timing of interim analyses to be determined adaptively during the trial.

    This method overcomes key limitations of traditional group sequential methods by not
    requiring the total number of analyses or their exact timing to be specified in advance.
    It's particularly valuable in clinical trials where interim analyses may be needed
    at unplanned times for ethical or scientific reasons.

    The approach works by "spending" portions of the overall alpha budget at each interim
    analysis according to a pre-specified spending function, ensuring that the cumulative
    Type I error rate never exceeds the target level.

    Args:
        alpha: Target Type I error rate (e.g., 0.05 for 5% error rate). Must be in (0, 1).
        spend_func: Alpha spending function that determines how to allocate alpha
                   across interim analyses. Must inherit from AbstractSpendFunc.

    Attributes:
        alpha0: Original target Type I error rate.
        rule: The spending function used to determine alpha allocation.
        num_test: Number of tests/analyses conducted so far.
        alpha: Current alpha level for the next test.

    Examples:
        >>> from online_fdr.spending.functions.bonferroni import BonferroniSpendFunc
        >>> # Create Bonferroni spending function for 5 planned analyses
        >>> spend_func = BonferroniSpendFunc(max_analyses=5)
        >>> alpha_spending = AlphaSpending(alpha=0.05, spend_func=spend_func)
        >>> # Test p-values sequentially
        >>> decision1 = alpha_spending.test_one(0.01)  # First interim analysis
        >>> decision2 = alpha_spending.test_one(0.03)  # Second interim analysis

    References:
        Lan, K. K. Gordon, and D. L. DeMets (1983). "Discrete Sequential Boundaries
        for Clinical Trials." Biometrika, 70(3):659-663.

        DeMets, D. L., and K. K. Gordon Lan (1994). "Interim Analysis: The Alpha
        Spending Function Approach." Statistics in Medicine, 13(13-14):1341-1352.

        Jennison, C., and B. W. Turnbull (1999). "Group Sequential Methods with
        Applications to Clinical Trials." Chapman and Hall/CRC.
    """

    def __init__(
        self,
        alpha: float,
        spend_func: AbstractSpendFunc,
    ):
        super().__init__(alpha)
        self.alpha0: float = alpha
        self.rule: AbstractSpendFunc = spend_func

    def test_one(self, p_val: float) -> bool:
        """Test a single p-value using the alpha spending approach.

        Determines the alpha level for the current analysis based on the spending function
        and the analysis number, then tests whether the p-value meets the significance
        threshold.

        Args:
            p_val: P-value to test. Must be in [0, 1].

        Returns:
            True if the null hypothesis is rejected (p_val < alpha), False otherwise.

        Raises:
            ValueError: If p_val is not in [0, 1].

        Examples:
            >>> alpha_spending = AlphaSpending(alpha=0.05, spend_func=my_spend_func)
            >>> alpha_spending.test_one(0.01)  # First analysis
            True
            >>> alpha_spending.test_one(0.04)  # Second analysis, stricter threshold
            False
        """
        validity.check_p_val(p_val)

        self.alpha = self.rule.spend(index=self.num_test, alpha=self.alpha0)
        self.num_test += 1
        return p_val < self.alpha

Functions

test_one(p_val)

Test a single p-value using the alpha spending approach.

Determines the alpha level for the current analysis based on the spending function and the analysis number, then tests whether the p-value meets the significance threshold.

Parameters:

Name Type Description Default
p_val float

P-value to test. Must be in [0, 1].

required

Returns:

Type Description
bool

True if the null hypothesis is rejected (p_val < alpha), False otherwise.

Raises:

Type Description
ValueError

If p_val is not in [0, 1].

Examples:

>>> alpha_spending = AlphaSpending(alpha=0.05, spend_func=my_spend_func)
>>> alpha_spending.test_one(0.01)  # First analysis
True
>>> alpha_spending.test_one(0.04)  # Second analysis, stricter threshold
False
Source code in online_fdr/spending/alpha_spending.py
def test_one(self, p_val: float) -> bool:
    """Test a single p-value using the alpha spending approach.

    Determines the alpha level for the current analysis based on the spending function
    and the analysis number, then tests whether the p-value meets the significance
    threshold.

    Args:
        p_val: P-value to test. Must be in [0, 1].

    Returns:
        True if the null hypothesis is rejected (p_val < alpha), False otherwise.

    Raises:
        ValueError: If p_val is not in [0, 1].

    Examples:
        >>> alpha_spending = AlphaSpending(alpha=0.05, spend_func=my_spend_func)
        >>> alpha_spending.test_one(0.01)  # First analysis
        True
        >>> alpha_spending.test_one(0.04)  # Second analysis, stricter threshold
        False
    """
    validity.check_p_val(p_val)

    self.alpha = self.rule.spend(index=self.num_test, alpha=self.alpha0)
    self.num_test += 1
    return p_val < self.alpha

Online Fallback Procedure

online_fdr.spending.online_fallback.OnlineFallback

Bases: AbstractSequentialTest

Online Fallback procedure for controlling the familywise error rate (FWER) online.

The online fallback procedure, developed by Tian and Ramdas (2021), provides strong FWER control for testing an a priori unbounded sequence of hypotheses one by one over time without knowing the future. It ensures that with high probability there are no false discoveries in the entire sequence.

This procedure is uniformly more powerful than traditional Alpha-spending methods and strongly controls the FWER even under arbitrary dependence among p-values. It uses a gamma sequence to allocate alpha budget over time and includes a "fallback" mechanism that increases future testing power after making discoveries.

The method unifies algorithm design concepts from offline FWER control and online false discovery rate control, providing a powerful adaptive approach for sequential hypothesis testing scenarios.

Parameters:

Name Type Description Default
alpha float

Target familywise error rate (e.g., 0.05 for 5% FWER). Must be in (0, 1).

required

Attributes:

Name Type Description
alpha0 float

Original target FWER level.

last_rejected bool

Whether the previous hypothesis was rejected.

seq

Gamma sequence for alpha allocation over time.

num_test int

Number of hypotheses tested so far.

alpha float

Current alpha level for the next test.

Examples:

>>> # Create online fallback instance
>>> fallback = OnlineFallback(alpha=0.05)
>>> # Test p-values sequentially
>>> decision1 = fallback.test_one(0.01)  # First test
>>> decision2 = fallback.test_one(0.03)  # Higher power if previous rejected
>>> print(f"Decisions: {decision1}, {decision2}")
>>> # Sequential testing with FWER guarantee
>>> p_values = [0.001, 0.8, 0.02, 0.9, 0.005]
>>> decisions = [fallback.test_one(p) for p in p_values]
>>> print(f"FWER-controlled decisions: {decisions}")
References

Tian, J., and A. Ramdas (2021). "Online control of the familywise error rate." Statistical Methods in Medical Research, 30(4):976-993.

ArXiv preprint: https://arxiv.org/abs/1910.04900

Source code in online_fdr/spending/online_fallback.py
class OnlineFallback(AbstractSequentialTest):
    """Online Fallback procedure for controlling the familywise error rate (FWER) online.

    The online fallback procedure, developed by Tian and Ramdas (2021), provides strong
    FWER control for testing an a priori unbounded sequence of hypotheses one by one
    over time without knowing the future. It ensures that with high probability there
    are no false discoveries in the entire sequence.

    This procedure is uniformly more powerful than traditional Alpha-spending methods
    and strongly controls the FWER even under arbitrary dependence among p-values.
    It uses a gamma sequence to allocate alpha budget over time and includes a "fallback"
    mechanism that increases future testing power after making discoveries.

    The method unifies algorithm design concepts from offline FWER control and online
    false discovery rate control, providing a powerful adaptive approach for sequential
    hypothesis testing scenarios.

    Args:
        alpha: Target familywise error rate (e.g., 0.05 for 5% FWER). Must be in (0, 1).

    Attributes:
        alpha0: Original target FWER level.
        last_rejected: Whether the previous hypothesis was rejected.
        seq: Gamma sequence for alpha allocation over time.
        num_test: Number of hypotheses tested so far.
        alpha: Current alpha level for the next test.

    Examples:
        >>> # Create online fallback instance
        >>> fallback = OnlineFallback(alpha=0.05)
        >>> # Test p-values sequentially
        >>> decision1 = fallback.test_one(0.01)  # First test
        >>> decision2 = fallback.test_one(0.03)  # Higher power if previous rejected
        >>> print(f"Decisions: {decision1}, {decision2}")

        >>> # Sequential testing with FWER guarantee
        >>> p_values = [0.001, 0.8, 0.02, 0.9, 0.005]
        >>> decisions = [fallback.test_one(p) for p in p_values]
        >>> print(f"FWER-controlled decisions: {decisions}")

    References:
        Tian, J., and A. Ramdas (2021). "Online control of the familywise error rate."
        Statistical Methods in Medical Research, 30(4):976-993.

        ArXiv preprint: https://arxiv.org/abs/1910.04900
    """

    def __init__(
        self,
        alpha: float,
    ):
        super().__init__(alpha)
        self.alpha0: float = alpha
        self.last_rejected: bool = False
        self.seq = DefaultLordGammaSequence(c=0.07720838)

    def test_one(self, p_val: float) -> bool:
        """Test a single p-value using the online fallback procedure.

        The online fallback procedure adjusts the alpha level based on whether the
        previous test was rejected (fallback mechanism) and allocates additional
        alpha using a gamma sequence. This creates higher power for future tests
        when discoveries are made.

        Args:
            p_val: P-value to test. Must be in [0, 1].

        Returns:
            True if the null hypothesis is rejected (discovery), False otherwise.

        Raises:
            ValueError: If p_val is not in [0, 1].

        Examples:
            >>> fallback = OnlineFallback(alpha=0.05)
            >>> fallback.test_one(0.01)  # First test, uses base alpha
            True
            >>> fallback.test_one(0.03)  # Second test, higher power after discovery
            True
            >>> fallback.test_one(0.04)  # Third test, even higher power
            False

        Note:
            The alpha level increases when the previous test was rejected, implementing
            the "fallback" mechanism that provides additional testing power.
        """
        validity.check_p_val(p_val)
        self.num_test += 1

        self.alpha = self.last_rejected * self.alpha
        self.alpha += self.alpha0 * self.seq.calc_gamma(self.num_test)

        is_rejected = p_val < self.alpha
        self.last_rejected = bool(is_rejected)  # Fix SIM210: Use bool() instead of True if else False

        return is_rejected

Functions

test_one(p_val)

Test a single p-value using the online fallback procedure.

The online fallback procedure adjusts the alpha level based on whether the previous test was rejected (fallback mechanism) and allocates additional alpha using a gamma sequence. This creates higher power for future tests when discoveries are made.

Parameters:

Name Type Description Default
p_val float

P-value to test. Must be in [0, 1].

required

Returns:

Type Description
bool

True if the null hypothesis is rejected (discovery), False otherwise.

Raises:

Type Description
ValueError

If p_val is not in [0, 1].

Examples:

>>> fallback = OnlineFallback(alpha=0.05)
>>> fallback.test_one(0.01)  # First test, uses base alpha
True
>>> fallback.test_one(0.03)  # Second test, higher power after discovery
True
>>> fallback.test_one(0.04)  # Third test, even higher power
False
Note

The alpha level increases when the previous test was rejected, implementing the "fallback" mechanism that provides additional testing power.

Source code in online_fdr/spending/online_fallback.py
def test_one(self, p_val: float) -> bool:
    """Test a single p-value using the online fallback procedure.

    The online fallback procedure adjusts the alpha level based on whether the
    previous test was rejected (fallback mechanism) and allocates additional
    alpha using a gamma sequence. This creates higher power for future tests
    when discoveries are made.

    Args:
        p_val: P-value to test. Must be in [0, 1].

    Returns:
        True if the null hypothesis is rejected (discovery), False otherwise.

    Raises:
        ValueError: If p_val is not in [0, 1].

    Examples:
        >>> fallback = OnlineFallback(alpha=0.05)
        >>> fallback.test_one(0.01)  # First test, uses base alpha
        True
        >>> fallback.test_one(0.03)  # Second test, higher power after discovery
        True
        >>> fallback.test_one(0.04)  # Third test, even higher power
        False

    Note:
        The alpha level increases when the previous test was rejected, implementing
        the "fallback" mechanism that provides additional testing power.
    """
    validity.check_p_val(p_val)
    self.num_test += 1

    self.alpha = self.last_rejected * self.alpha
    self.alpha += self.alpha0 * self.seq.calc_gamma(self.num_test)

    is_rejected = p_val < self.alpha
    self.last_rejected = bool(is_rejected)  # Fix SIM210: Use bool() instead of True if else False

    return is_rejected

Spending Function Types

The alpha spending framework supports various spending functions that determine how to allocate the alpha budget across interim analyses:

  • O'Brien-Fleming type: Conservative early spending, allowing larger effects to be detected later
  • Pocock type: Equal spending at each analysis
  • Linear spending: Proportional to analysis timing
  • Custom functions: User-defined spending patterns

Key Concepts

Alpha Budget Allocation

The fundamental principle is to "spend" portions of the overall alpha budget at each interim analysis according to a pre-specified spending function, ensuring cumulative Type I error never exceeds the target level.

Flexibility vs. Control

Alpha spending provides: - Flexibility: Adapt timing and number of analyses during the trial - Control: Maintain strict Type I error rate control - Efficiency: Stop early for efficacy or futility

Comparison with Other Methods

Method Flexibility Power Complexity
Alpha Spending High Good Medium
Group Sequential Low High Low
Online FDR High Variable High

Usage Guidelines

  1. Choose appropriate spending function based on trial characteristics
  2. Monitor spending carefully to avoid early alpha exhaustion
  3. Plan for unscheduled analyses that may be required
  4. Consider futility bounds in addition to efficacy boundaries

References

Lan, K. K. Gordon, and D. L. DeMets (1983). "Discrete Sequential Boundaries for Clinical Trials." Biometrika, 70(3):659-663.

DeMets, D. L., and K. K. Gordon Lan (1994). "Interim Analysis: The Alpha Spending Function Approach." Statistics in Medicine, 13(13-14):1341-1352.

Jennison, C., and B. W. Turnbull (1999). Group Sequential Methods with Applications to Clinical Trials. Chapman and Hall/CRC.