Strategy Orchestra

Problem

I wanted to see whether LLM agents could do the slow part of strategy research: propose an idea, test it properly, read the results and decide what to try next, overnight and without me in the loop.

Generating ideas is the easy part; an LLM will happily propose hundreds. The hard part is that every idea is tested against the same historical data, so the more you try, the more likely it is that the best result is luck. A script that loops over ideas would just find the luckiest one. The system has to search and, at the same time, account for how much searching it has done.

Constraints

• The model can't be trusted with numbers. An LLM-assigned score or verdict can drift with the prompt and the model, so every number that decides an outcome must come from code.
• Local models are slow and fragile. Proposals come from local models through LM Studio or Ollama. A reasoning model can spend minutes thinking before it answers, or use up its output tokens and return nothing.
• One shared forward window. Every candidate is judged on the same out-of-sample period, and that period becomes less trustworthy with each strategy tried.
• Unattended runs. Campaigns run overnight, so failures have to be classified and recorded, not just raised.

Design

if score_pct >= pass_score:
    blocked = ((sample is not None and not sample.get("adequate", True))
               or (dsr is not None and not dsr.get("passes", True)))
    return "refine" if blocked else "pass"
if score_pct >= refine_score:
    return "refine"
return "abandon"

The loop is a LangGraph state machine with nine nodes:

1. Strategist. An LLM writes a full strategy as JSON.
2. Builder. Code validates the draft, and any errors go back to the Strategist as a repair round.
3. Engine. Deterministic code runs a baseline backtest with Monte Carlo, an NSGA-II parameter search, four walk-forward folds and a forward test.
4. Judge. Scores the result.
5. Router. Decides what happens next.

Lessons, a hall of fame and bandit statistics carry over from one run to the next.

A second front-end, the Researcher, is a ReAct agent that picks its own next tool call from ten tools. It shares the engine and all of the cross-run memory.

Key decisions

• The LLM proposes, code decides. The score comes from an 11-factor rubric in code. The Judge's model only writes the commentary, which feeds the next proposal. An LLM-assigned score would drift between runs and between models.
• The bar rises with the search. The pass bar goes up one point for every 50 distinct strategies judged. A pass is demoted to "refine" in three cases:
  • the forward sample has fewer than 20 trades;
  • the per-trade edge has a t-stat under 1;
  • the forward Sharpe doesn't beat the expected maximum Sharpe of that many worthless strategies (a deflated-Sharpe test).
• Data the search never sees. The last 6 months of the forward window are a vault. A declared winner is evaluated there once, and the result never moves a score. A vault window retires after 5 uses. The last 3 months of all data are kept back as a further reserve.
• Infrastructure failures aren't research results. A timeout, an empty reply or an output cap spent on thinking is classified as infrastructure. It costs no bandit pull, writes no lesson, and the slot is retried before it counts. Claude is called as a fallback only for these, never for a validation error.
• Guarantees live in the tools, not the prompt. The Researcher can call tools in any order, so the tool registry enforces the rules. Duplicate checks run on both what a strategy says (its conditions) and what it did (its trades, return and Sharpe).

Evidence

The system's own read-only audit of all 232 runs, from July to October 2026:

• The guards caught the early winners.

  • Passes: 82 scorecards carry a pass, all from before the current floors. Their forward samples have a median of about 12 trades, none beat buy-and-hold over the full span, and none of them would pass under today's rules.
  • Hall of fame: it is ten variations of one idea, a daily EMA/RSI trend filter on SPY. On the vault, 6 of the 10 lost money.

• The audit found the system's own bugs.

  • A bandit bug recorded outcomes under names that never matched an arm, so all 316 pulls went to the first arm.
  • Backend crashes had been counted as failed strategies.
  • The old vault window had been used 78 times.

  Each of these is now fixed in the code.

• Latency and cost. The Strategist ran for 79 hours in total, and 41% of that time produced no candidate. Strategist calls that led to a judged candidate averaged 3.2 minutes. Cost per run isn't measured, because neither backend reports token counts.

• Engineering. 692 of 696 tests pass with no LLM and no market data, and ruff is clean. The 4 skipped tests check the engine bar for bar against the original notebook, which lives in another repository.

What I'd change

• Run a campaign under the current rules, on the clean vault window, with a goal that can produce 20 or more forward trades, such as intraday bars or several tickers.
• Re-seed the hall of fame with different strategy families now that the bandit records outcomes correctly, and mark the entries from before the floors as legacy.
• Record one run as a demo from its per-run story page, so a reader can step through it.
• Measure cost. Add token counts to the backends so each run has a price.
• Run the equivalence test in CI against the reference notebook, so the engine port is checked on every change.