Portfolio Distiller

Problem

My research universe has 75 US stocks and ETFs. That's too many to trade, and less diversified than it looks: SPY, VOO and SPXL move almost together. The question is which small subset keeps the diversification, and whether that choice can be made each month using only data available at the time.

There are many reasonable ways to pick the subset: clustering, network centrality, principal components, ranking by risk-adjusted return. I wanted to compare them fairly, so the only thing that differs between them is the selection rule itself.

Constraints

• No lookahead. Every statistic used to select at decision date t must use data from before t only. Selection always runs on a rolling window, never on the full sample.
• Daily data for selection. The source is 1-hour bars (February 2010 to June 2026). Cross-asset correlations are measured on daily returns, because intraday correlations between assets that trade at different moments are biased low.
• An ill-conditioned covariance. A raw 57-name correlation matrix has a condition number of 5,733, so the clustering, tree and PCA methods use a shrunk one.
• Reproducible. Rerunning a notebook from a fresh kernel must give the same baskets.

Design

def select_window(R, asof, years, min_cov=0.98):
    start = asof - pd.DateOffset(years=years)
    win = R[(R.index >= start) & (R.index < asof)]
    cols = win.columns[win.notna().mean() >= min_cov]
    return win[cols].dropna(how="any")

At each month-end, the harness drops the least liquid tenth of the universe and any near-duplicates (correlation of 0.97 or more). It then takes a 10-year window of daily returns that ends before the decision date, and shrinks the covariance with Ledoit-Wolf. Seven selection methods (the "lenses"), plus a vote across them, each pick 12 names from that identical window:

The 12 names are held in equal weight until the next month-end, and the harness records what they earned.

Key decisions

• One harness for every lens. The screen and covariance are built once per rebalance and handed to every lens. Differences between results come only from the selection rule, not from incidental choices about windows or weights.
• No separate backtest code. The walk-forward loop calls the same functions the inspection cells use, so the backtest can't drift away from what the notebook shows.
• Shrink the covariance before anything uses it. Ledoit-Wolf brings the condition number from 5,733 down to 86. Clustering, trees and PCA all read the shrunk matrix.
• Test the anti-pattern in the open. Ranking names by trailing Sharpe and keeping the top is the classic multiple-testing trap. Running it as one of the lenses, out of sample, makes the trap visible instead of hiding it.
• Notebooks generated from scripts. Each notebook is written by a build script, so its cells live in reviewable code. Regenerating reproduces the committed notebook cells exactly.

Evidence

Out of sample: 73 monthly rebalances from May 2020 to May 2026, 12 names in equal weight, before costs.

• Nothing decisively beats equal-weighting on Sharpe. The payoff is tradeability: 12 names instead of about 60. The PCA lenses also cut the benchmark's worst drawdown by about 40%.
• The trap shows up as predicted. Ranking by trailing Sharpe gives the second-highest growth and the deepest drawdown, with a highly correlated basket: mean |ρ| of 0.51 inside the basket, against 0.16 for HRP and 0.18 for the PCA lenses.
• The structure is readable. The first principal component is the market: it explains 20.7% of the variance and correlates 0.97 with SPY. Only 6 of 57 components clear the random-matrix noise threshold. In the spanning tree, TQQQ is the hub, connected to 38 other names.
• History length matters. With a 5-year lookback instead of 10, PCA factor reps falls from a Sharpe of 1.24 to 0.62.
• The checks pass, within their limits. The leakage probe passes in every notebook, but it's narrow: it re-checks the window function on six dates. That the screens only see past data rests on reading the code. A second run from a fresh kernel reproduces identical baskets (checked on a synthetic stand-in, since the market data is licensed).

What I'd change

• Add transaction costs to the main comparison. Turnover is already measured for every lens.
• Address survivorship. The universe is the 75 names that exist in 2026, picked with hindsight.
• Re-test the risk overlay before merging it. An unmerged branch adds trend gating and volatility targeting, net of 10 bps costs. It should compute Sharpe on returns in excess of cash, and test the chosen design on a period it wasn't tuned on.