Portfolio Construction (NCO)

This chapter addresses a fundamental flaw in modern portfolio theory: instability. While Markowitz's (1952) mean-variance optimization is mathematically elegant, its solutions are notoriously unstable and concentrated, often underperforming simpler 1/N portfolios in practice.

This instability arises from two sources: noise (from data sampling) and, as this chapter details, signal (from the data's true structure).

The Problem: Markowitz's Curse

The classic Markowitz solution for a minimum-variance portfolio is a convex optimization problem:

• Objective Function:
  $$\min _{\omega} \frac{1}{2} \omega^{\prime} V \omega$$
• Constraint:
  $$\omega^{\prime} a=1$$
• The Solution:
  $$\omega^{*}=\frac{V^{-1} a}{a^{\prime} V^{-1} a}$$

The critical problem is that this solution requires inverting the covariance matrix ($V^{-1}$). This leads to Markowitz's Curse:

The more correlated the assets (i.e., the more diversification is needed), the more unstable and unreliable the solution becomes.

This instability is measured by the condition number of the correlation matrix, which is the ratio of its largest eigenvalue ($\lambda_{\max}$) to its smallest ($\lambda_{\min}$).

• Example (2x2 Matrix): For a simple $C = \begin{bmatrix} 1 & \rho \\ \rho & 1 \end{bmatrix}$:
  • The eigenvalues are $\lambda_1 = 1+\rho$ and $\lambda_2 = 1-\rho$.
  • The inverse is $C^{-1} = \frac{1}{1-\rho^2} \begin{bmatrix} 1 & -\rho \\ -\rho & 1 \end{bmatrix}$.
  • As correlation $\rho \to 1$, $\lambda_2 \to 0$, the condition number ($\lambda_1 / \lambda_2$) $\to \infty$, and the inverse $C^{-1}$ explodes. This numerical instability magnifies any small estimation errors in the inputs, resulting in extreme and volatile portfolio weights.

Signal as the Source of Instability

This instability is not just random noise; it is caused by the true signal structure of financial data: clusters.

• Financial assets are naturally clustered (e.g., by sector, industry, or style).
• Assets within a cluster are highly correlated with each other.
• This cluster structure creates large eigenvalues (representing the cluster) and forces other eigenvalues to be very small to compensate (since $\text{tr}(C) = N$).
• This guarantees a high condition number and an unstable matrix. While the instability is caused by a specific cluster, inverting the matrix spreads this instability across the entire portfolio solution.

The Solution: Nested Clustered Optimization (NCO)

The Nested Clustered Optimization (NCO) algorithm is an ML-based "wrapper" method that solves Markowitz's Curse by containing this instability. It splits the optimization into a three-step, hierarchical process.

1. Correlation Clustering:

   • The covariance matrix is first denoised (using methods from Section 2, like Marcenko-Pastur) to remove random noise.
   • A clustering algorithm (like the ONC algorithm from Section 4) is applied to the clean correlation matrix to identify the underlying cluster structure (e.g., the 10 sectors).

2. Intra-cluster Allocation:

   • The optimization (e.g., minimum variance) is run separately for the assets within each cluster.
   • This step "contains" the instability. The high correlations are handled locally, without affecting the rest of the portfolio.
   • This produces a reduced covariance matrix (cov2) that represents the risk/reward of the optimized clusters themselves.

3. Inter-cluster Allocation:

   • The optimization is run a final time on the reduced covariance matrix (cov2).
   • Because the clusters are (by definition) dissimilar, this reduced matrix is stable, well-conditioned (close to diagonal), and solves the "curse."
   • The final portfolio weights are the product of the inter-cluster weights and the intra-cluster weights.

Experimental Results

Monte Carlo experiments confirm NCO's superiority. When compared to Markowitz's solution (CLA) on a "true" covariance matrix with a 10-block cluster structure:

• For the Minimum Variance Portfolio, NCO reduced the allocation error (RMSE) by 47%.
• For the Maximum Sharpe Ratio Portfolio, NCO reduced the allocation error (RMSE) by 55%.

The NCO algorithm provides a substantially more robust and stable solution by isolating cluster-specific instability instead of allowing it to infect the entire portfolio.

API reference

RiskLabAI implements these in Python and Julia (signatures auto-generated from the package source):

def inverse_variance_weights(covariance_matrix: pd.DataFrame) -> np.ndarray:

function inverse_variance_weights(covariance_matrix::AbstractMatrix{<:Real})

def cluster_variance(
    covariance_matrix: pd.DataFrame, clustered_items: list[str]
) -> float:

function cluster_variance(
    covariance_matrix::AbstractMatrix{<:Real},
    clustered_items::AbstractVector{<:Integer},
)

def quasi_diagonal(linkage_matrix: np.ndarray) -> list[int]:

function quasi_diagonal(linkage_matrix::AbstractMatrix{<:Real})

def recursive_bisection(
    covariance_matrix: pd.DataFrame, sorted_items: list[str]
) -> pd.Series:

function recursive_bisection(
    covariance_matrix::AbstractMatrix{<:Real},
    sorted_items::AbstractVector{<:Integer},
)

def pca_weights(
    cov: np.ndarray,
    risk_distribution: Optional[np.ndarray] = None,
    risk_target: float = 1.0,
) -> np.ndarray:

function pca_weights(
    cov::AbstractMatrix{<:Real};
    risk_distribution::Union{Nothing,AbstractVector{<:Real}} = nothing,
    risk_target::Real = 1.0,
)

def hrp(cov: pd.DataFrame, corr: pd.DataFrame) -> pd.Series:

function hrp(
    covariance::AbstractMatrix{<:Real},
    correlation::AbstractMatrix{<:Real},
)

def get_optimal_portfolio_weights(
    covariance: np.ndarray, mu: Optional[np.ndarray] = None
) -> np.ndarray:

function get_optimal_portfolio_weights(
    covariance::AbstractMatrix{<:Real};
    mu::Union{Nothing,AbstractVecOrMat{<:Real}} = nothing,
)

def get_optimal_portfolio_weights_nco(
    covariance: np.ndarray,
    mu: Optional[np.ndarray] = None,
    number_clusters: Optional[int] = None,
) -> np.ndarray:

function get_optimal_portfolio_weights_nco(
    covariance::AbstractMatrix{<:Real};
    mu::Union{Nothing,AbstractVector{<:Real}} = nothing,
    number_clusters::Union{Nothing,Integer} = nothing,
)

Full source: Python · Julia