Reinforcement-learning trading strategy

Problem

I wanted an agent that trades a small, diversified basket of ETFs on intraday bars. Every 15 minutes it decides which ETF to touch, how big a position to hold, and whether to hold, trade or close.

Costs, position limits and drawdown all depend on the path the agent takes, so each decision changes the next one. That makes it a natural fit for reinforcement learning, where an agent learns from the results of its own trades rather than from a forecast plus fixed rules.

Constraints

• Intraday data with uneven history. 15-minute bars for 23 US ETFs, February 2010 to July 2025. The 10 ETFs I kept only share history from November 2014, when PDBC's data begins.
• Costs on every fill. 0.1% of the traded value, with each position capped at 30% of equity.
• A very wide state. 2,143 indicator columns for each ETF, so 21,455 inputs per step once the portfolio and risk numbers are added.
• Unstable training. An earlier A2C run diverged to NaN during training, so training needed guards.

Design

target_position_value = position_size * portfolio_value_before * self.max_position_size
current_position_value = self.positions[asset_idx] * current_price
trade_value = target_position_value - current_position_value
trade_shares = trade_value / current_price

transaction_cost_amount = abs(trade_value) * self.transaction_cost

if self.balance >= transaction_cost_amount:
    self.positions[asset_idx] += trade_shares
    self.balance -= transaction_cost_amount

The data path runs across the top. A greedy search keeps the 10 ETFs with the lowest average correlation. Each one gets 2,143 indicator columns, centred on the median and divided by the interquartile range. A custom Gymnasium environment steps through the bars.

The middle row is the learning loop. The environment passes the state and a shaped reward to two stable-baselines3 agents, PPO and A2C, and the ensemble averages their actions into one. The bottom row is evaluation: saved models, a Monte Carlo risk-of-ruin run, a walk-forward backtest and a risk manager.

The agents answer with three numbers:

The reward is the step's return divided by one plus recent volatility. It subtracts 0.001 for each trade and twice the worst drawdown so far. It adds three small bonuses: for holding several positions, for a win rate above 50%, and for average gains larger than 1.5 times average losses.

Key decisions

• One continuous action instead of one action per ETF. Three numbers cover the ETF, the size, and hold, trade or close. A standard continuous-action PPO or A2C policy can drive both the choice and the size. The cost is that the ETF choice becomes a number cut into ten bins.
• A shaped reward instead of raw P&L. Raw return lets an agent ignore risk and trade freely. Scaling by volatility, charging for trades and drawdown, and rewarding a spread of positions and a good win/loss ratio push it toward steadier behaviour.
• Two algorithms, averaged. PPO for stable training, A2C for fast training. Averaging two different learners is meant to reduce the chance that one of them overfits.
• Guards around training. Training runs in four stages, from 1,000 to 10,000 steps. After each checkpoint the code checks the outputs for NaN, and it reinitialises a model whose weights have turned NaN. A2C gets a smaller tanh network, gradient clipping and advantage normalisation. Any input above a million makes the ensemble fall back to an action that trades nothing.
• Low correlation first. I compared three ways to pick 10 ETFs from 23. The greedy search gave a mean correlation of 0.123, against 0.156 for clustering and 0.597 for a Sharpe-weighted score, so I kept it.

Evidence

• The pipeline runs end to end. Both agents finish all four training stages with no NaN, and then save, reload and act. Fourteen ensembles were saved between September 2025 and January 2026, across three versions of the inputs: 315, 21,455 and 28,742 numbers per step.
• A Monte Carlo run completed. In 1,000 noisy replays of 500 bars each, no run lost half its equity. The mean return was −0.11% and the average maximum drawdown was 26%. These numbers don't measure skill, for the reasons below.
• No valid out-of-sample result yet. I don't report a Sharpe ratio or a return here, because nothing run so far measures one honestly.

What didn't work

Each of these problems invalidates the results on its own:

• Opening a position doesn't take cash. The environment deducts the 0.1% cost but not the price of the shares. With prices held flat, one full-size buy lifts equity from 100,000 to 129,970, and a short removes about 30%. The reward follows equity, so buying is rewarded for its own sake.
• Smaller design issues.
  • Averaging the asset number mixes two choices: ETF 0 from PPO and ETF 9 from A2C becomes ETF 5.
  • "Close" only fires at the very top of its range.
  • The ETFs are read by row number, not timestamp, so the same step can be a different moment for different ETFs.

A later version, from June 2026, moves to daily bars and replaces the placeholder with the real environment.

What is next to be enhanced

• Give each ETF its own action. Use target weights for all 10, or a single agent, instead of averaging one continuous asset number.
• Train across the whole training period, not just its first few weeks.
• Compare with simple baselines out of sample. Use buy-and-hold, equal weight with regular rebalancing, and a random agent that trades as often. Compare after costs, on dates never used for selection, scaling or training.