Backtest Statistics

This chapter provides a comprehensive overview of the statistics required to evaluate a backtest, regardless of the backtesting paradigm used (historical, cross-validation, or synthetic). These metrics are essential for investors to compare strategies and identify potential flaws, such as hidden risks or low capacity.

General Characteristics

These statistics describe the high-level properties of the strategy:

• Capacity: The highest AUM the strategy can manage before performance degrades.
• Leverage: The amount of borrowing used.
• Frequency of Bets: The number of bets per year. A "bet" is a full cycle from a flat position to an exit, which is a more informative metric than "trades."
• Average Holding Period: The average time a bet is held.
• Annualized Turnover: The ratio of the average dollar amount traded per year to the average AUM.
• Correlation to Underlying: A high correlation suggests the strategy is not adding significant alpha.

Performance

These are unadjusted performance metrics.

• Time-Weighted Rate of Return (TWRR): The GIPS-compliant standard for calculating returns. It adjusts for external cash flows and geometrically links sub-period returns to provide a true performance measure.
  • Geometric Linking: $\varphi_{i, T}=\prod_{t=1}^{T}\left(1+r_{i, t}\right)$
  • Annualized Return: $R_{i}=\left(\varphi_{i, T}\right)^{-y_{i}}-1$
• Hit Ratio: The fraction of bets that resulted in a profit.
• Average Return from Hits/Misses: The average PnL for winning and losing bets, respectively.

Runs and Drawdowns

These metrics evaluate risk, particularly for non-IID (Independent and Identically Distributed) returns.

• Returns Concentration (HHI): Measures if returns are concentrated in a few lucky bets, similar to the Herfindahl-Hirschman Index. A value near 0 is ideal (uniform returns), while a value near 1 means a single bet generated all the PnL.
  • Equation (for positive returns):
    $$h^{+} \equiv \frac{\sum_{t}\left(w_{t}^{+}\right)^{2}-|w^{+}|^{-1}}{1-|w^+|^{-1}}$$
• Drawdown (DD): The maximum loss from a peak-to-trough in PnL.
• Time under Water (TuW): The time elapsed to recover from a drawdown and reach a new high-watermark.

Efficiency (Risk-Adjusted Performance)

This is the most critical category, as it corrects performance for risk, skewness, and selection bias.

• Sharpe Ratio (SR): The standard measure of mean excess return over volatility.

  • Equation:
    $$SR=\frac{\mu}{\sigma}$$

• Probabilistic Sharpe Ratio (PSR): A key metric that adjusts the SR for non-Normal returns (skewness $\hat{\gamma}_3$, kurtosis $\hat{\gamma}_4$) and short track records ($T$). It estimates the probability that the true SR is above a benchmark $SR^*$.

  • Equation:
    $$\widehat{P S R}\left(S R^{*}\right)=Z\left(\frac{\left(\widehat{S R}-S R^{*}\right) \sqrt{T-1}}{\sqrt{1-\hat{\gamma}_{3} \widehat{S R}+\frac{\hat{\gamma}_{4}-1}{4} \widehat{S R}^{2}}}\right)$$

• Deflated Sharpe Ratio (DSR): A PSR that also corrects for selection bias (multiple testing). It calculates the expected maximum SR ($SR^*$) one would get by pure chance from $N$ trials, and then uses that $SR^*$ as the rejection threshold.

  • Equation for $SR^*$:
    $$S R^{*}=\sqrt{V\left(\left\{\widehat{S R}_{n}\right\}\right)}\left((1-\gamma) Z^{-1}\left(1-\frac{1}{N}\right)+\gamma Z^{-1}\left(1-\frac{1}{N} e^{-1}\right)\right)$$
  • This leads to Marcos' Third Law of Backtesting: "Every backtest result must be reported in conjunction with all the trials involved in its production... it is impossible to assess the backtest's 'false discovery' probability."

Classification Scores (for Meta-Labeling)

These metrics are used to evaluate the secondary (overlay) model in a meta-labeling context.

• Accuracy, Precision, and Recall: Standard classification metrics.
• F1-Score: The harmonic mean of precision and recall. It is the preferred metric over "accuracy" for imbalanced datasets, which are common in meta-labeling.
  • Equation:
    $$F_{1}=2 \frac{\text { precision } \cdot \text { recall }}{\text { precision }+\text { recall }}$$
• Negative Log-Loss: A score that penalizes overconfident wrong answers. It is superior to accuracy because it evaluates the model's predicted probabilities, not just the final label.
  • Equation:
    $$L[Y, P]=-N^{-1} \sum_{n=0}^{N-1} \sum_{k=0}^{K-1} y_{n, k} \log \left(p_{n, k}\right)$$

API reference

RiskLabAI implements these in Python and Julia (signatures auto-generated from the package source):

def sharpe_ratio(returns, risk_free_rate=0):

function sharpe_ratio(returns::AbstractVector{<:Real}; risk_free_rate::Real = 0.0)

def bet_timing(target_positions: pd.Series) -> pd.Index:

function bet_timing(index::AbstractVector, target_positions::AbstractVector{<:Real})

def calculate_holding_period(
    target_positions: pd.Series,
) -> tuple[pd.DataFrame, float]:

function calculate_holding_period(
    index::AbstractVector,
    target_positions::AbstractVector{<:Real},
)

def calculate_hhi(bet_returns: pd.Series) -> float:

function calculate_hhi(bet_returns::AbstractVector{<:Real})

def calculate_hhi_concentration(returns: pd.Series) -> tuple[float, float, float]:

function calculate_hhi_concentration(index::AbstractVector, returns::AbstractVector{<:Real})

def compute_drawdowns_time_under_water(
    pnl_series: pd.Series, dollars: bool = False
) -> tuple[pd.Series, pd.Series]:

function compute_drawdowns_time_under_water(
    index::AbstractVector,
    pnl::AbstractVector{<:Real};
    dollars::Bool = false,
)

Full source: Python · Julia