TSBench: multi-series forecasting benchmark

Problem

TSBench compares forecasting models from several libraries on the same panel of US-market data: 75 stocks and ETFs, 15-minute bars, February 2010 to July 2026. It asks one question per target and horizon: does any model beat a simple baseline, and can I trust the answer?

Running the models is the easy part. Knowing whether a result is real is the hard part. Returns have almost no signal, so a broken harness and a working one look the same there. A leaked feature, a swapped column or a lucky winner among thirty models can all pass for skill.

Constraints

• The data has defects that never raise an error. Eleven are documented. 23 of the 75 tickers store Close where Open should be, timestamps are naive UTC+8 and drift with US daylight saving, sessions straddle midnight, and 318,294 bars are stale, with zero volume. The volume field is a partial feed: 0.9–1.7% of the average daily volume reported elsewhere.
• The libraries can't share one Python environment. statsforecast, neuralforecast, uni2ts and the Hugging Face models need conflicting dependencies.
• Overlapping horizons leak. Forecasting an h-bar return at every bar means neighbouring labels share bars, so train and test need a gap of at least h bars.
• One machine. Everything runs on my Mac. The five-horizon production run took 3.2 hours.

Design

included = [m for m in mcs.get("included", []) if m in set(sub["model_id"])]
if included:
    inside = sub[sub["model_id"].isin(included)]
    pick = inside.sort_values(["wall_clock_s", "score"]).iloc[0]
    chosen_by = "one_standard_error_within_mcs"
else:
    pick = sub.iloc[0]
    chosen_by = "argmin"   # subject to the winner's curse

The harness has five stages. Repair fixes the defects and writes one canonical panel in New York time, with explicit session and grid keys. Targets and splits define twelve forecast targets in four families (returns, volatility, volume, path) and rolling-origin folds with a final holdout. The orchestrator writes each task to disk as JSON plus a parquet slice and runs it in that library's own Python environment. Selection scores every model against its target's hardest baseline and keeps the set the data can't separate. The frozen winner is then scored once on the holdout.

Key decisions

• A subprocess boundary, not one big environment. Each library family runs in its own environment and talks to the orchestrator only through parquet and JSON on disk, never pickle. That one boundary also gives crash isolation, per-task timeouts and resumable runs. On a timeout the whole process group is killed, so a library that spawned its own workers can't keep running.
• Every target against its hardest baseline. HAR-RV for volatility, seasonal naive over one session (26 bars) for volume, and zero for returns. A generic baseline would make weak models look good.
• Selection, not the lowest error. Picking the best of about thirty models by error alone is a winner's curse. A Model Confidence Set (α = 0.1) keeps every model the data can't separate from the best, and the one-standard-error rule picks the cheapest of those.
• Known-answer tests as a gate. Log volume must beat seasonal naive by a wide margin, returns must show out-of-sample R² near zero, shuffled labels must collapse, and deleting future rows must leave every prediction unchanged. A weak volume result is treated as a harness bug, not a finding.
• Invariants written down, each tied to a measured defect. Access OHLCV columns by name, never by position. Keep the train/test gap at least h bars. Compute the MASE denominator on the training slice only: it is passed into the metric rather than computed there, so it can't come from the test fold by accident.

Evidence

Production run: 74 tickers (one mostly stale ticker dropped), 12 folds plus a holdout, five horizons from 15 minutes to one week.

The known-answer tests pass 10 of 10 on production data. Volume beats seasonal naive by 25.8% (MASE 0.772 against 1.042). Returns stay at an out-of-sample R² of 0.00000. Shuffling the labels pushes MASE from 0.772 to 2.935. Deleting future rows changes no prediction at all.

A trading track turns forecasts into 89 strategies, net of costs. The best gross Sharpe of 8.95 falls to 2.32 after costs, and the Deflated Sharpe p-values run from 0.903 to 1.000. None is significant, so nothing is promoted to a signal.

On the engineering side, 115 of 120 tests pass on a clean copy. The other five need the Mac-only worker environments or the licensed data file.

What I'd change

• Run the deep-learning and statistical families at production scale. So far neuralforecast and statsforecast have run only on development slices of 6 to 8 tickers, and the foundation models (Chronos-Bolt, Moirai) only as a one-fold smoke test.
• Read the volatility result narrowly. One horizon out of five is significant, and only on the consistent test. That is worth following up, not worth claiming.
• Prepare the repository for publishing. Keep the licensed data out of the git history, strip local paths from saved notebook outputs and run manifests, and update a README that still describes the plan rather than the code.