The Same Prompt Finished 11.64 Points Apart
Three runs of the same Claude Sonnet 5 trading prompt returned -10.29%, +1.35% and -0.31%. The best and worst finished 11.64 points apart. Their ending balances were $8,971.30, $10,135.33 and $9,968.54.
We replayed past markets using the live arena's validator, a $10,000 paper account and a 0.1% fee per trade. The window ran from February 2 to June 23, 2026, with 142 daily decision cycles at 16:00 UTC. The universe contained 60 tradeable assets: 10 crypto and 50 US stocks. BTC and SPY were benchmarks and were not traded.
The model decided from bars closed before 16:00 UTC. Fills used that day's close plus slippage of 10 basis points for crypto and 3 for stocks. To open a position, the model had to state a confidence of at least 0.80 and choose a stop, a price at which the position closes. The replay checked each stop against later daily bars and closed the position when price touched it. We call that a stop exit. The arena calls the stop an invalidation price.
The baseline prompt was our copy of the live arena prompt with a forecast section added. The three baseline runs used the same model, prompt, data, asset codes and dates. In the names-hidden setup, every ticker was replaced by a code and every date by a relative offset. Price levels were not hidden. Run 1 ran on November 10, and Runs 2 and 3 repeated it on November 12.
The three runs split on the first day, February 2, on a byte-identical prompt. Run 2 opened a short on MU at 15% of equity and confidence 0.82. It closed on February 19 for +$67.71. Runs 1 and 3 made no trade that day. From the second day, each run carried its own positions and notes into its next prompt.
How much stop exits lost differed by run. Run 1's 22 stop exits realized -$1,443.76 with 1 winner, more than the whole run lost. Stop exits realized -$414.90 in Run 2 and -$210.96 in Run 3. A stop starts on the losing side of entry, and the model can move it later, so some stop exits are winners. Stop exits losing money in total is still expected.
All three finished below buy and hold SPY at +5.49% and above buy and hold BTC at -20.33%. The table also includes the two prompt edits and the Sonnet 5.5 recall runs discussed below. The first five rows are Sonnet 5 with names hidden.
The Seven Runs
| Run | Return | Max drawdown | Opens | Days in cash |
|---|---|---|---|---|
| Run 1 (baseline) | -10.29% | 12.69% | 29 | 3 |
| Run 2 (baseline) | +1.35% | 5.35% | 24 | 15 |
| Run 3 (baseline) | -0.31% | 7.84% | 36 | 7 |
| Free mandate (edit) | +10.89% | 5.40% | 41 | 2 |
| Exit note (edit) | +8.28% | 7.20% | 24 | 4 |
| Sonnet 5.5, names hidden | -0.97% | 0.97% | 1 | 140 |
| Sonnet 5.5, names shown | -0.02% | 0.02% | 1 | 141 |
No Run Showed Skill at Next-Day Direction
Alongside each trading decision, the model forecast the chance that the next daily close would be higher for every asset it was shown candles for. It also forecast the same chance for 7 closes ahead and gave a 10% to 90% price range.
No run showed skill at calling next-day direction. We compared the calls with a drift formula that gives each asset the average daily move of its own history. For each day, we counted whether the run or the formula made more correct next-day calls, on the forecasts both made. The p-value measures how often a split at least this lopsided would turn up if the run and the formula were equally good. A p-value of 0.0052 is about 0.5% of the time.
Of the three baseline runs, only Run 1 was below p = 0.05 taken alone, at p = 0.0052. Runs 2 and 3 had 0.18 and 0.11 in the same direction. The lab's report also corrects each p-value for the 12 prompt candidates its research contract allows. That puts Run 1 at 0.0619, so after that correction no run is below 0.05.
All seven runs lost more days than they won against the drift formula. We noticed this after the runs. Two were below p = 0.05 on their own. The runs share the same days and market, so they are not seven independent results.
The largest difference in next-day direction hit rate between the three baseline runs was 2.72 points. Each of the three pairwise 95% intervals included zero. Shared calls in the table are the next-day direction calls both runs made. We wrote these comparisons down before the two repeats ran.
The three pairwise differences on the price-range score also had intervals that included zero. That score reads the 10% to 90% price range over both the next-day and 7-day forecasts. Lower is better. Against the drift formula with recent volatility, no run was better or worse on this score at p < 0.05.
Pairwise Differences in Next-Day Direction Hit Rate
| Comparison | Shared calls | Hit-rate difference | 95% interval |
|---|---|---|---|
| Run 2 minus Run 1 | 1,102 | +2.72 points | -1.54 to +6.98 |
| Run 3 minus Run 1 | 1,201 | +0.67 points | -2.99 to +4.37 |
| Run 3 minus Run 2 | 1,129 | -2.04 points | -5.70 to +1.73 |
Days Won and Lost Against the Drift Formula
| Run | Won | Lost | Tied | p |
|---|---|---|---|---|
| Run 1 | 46 | 78 | 18 | 0.0052 |
| Run 2 | 48 | 63 | 31 | 0.18 |
| Run 3 | 49 | 67 | 26 | 0.11 |
| Free mandate | 48 | 67 | 27 | 0.093 |
| Exit note | 46 | 73 | 23 | 0.017 |
| Sonnet 5.5, names hidden | 50 | 63 | 28 | 0.26 |
| Sonnet 5.5, names shown | 57 | 68 | 17 | 0.37 |
Two Edited Prompts Finished at +10.89% and +8.28%, One Run Each
The free mandate returned +10.89%, and the exit note returned +8.28%. The free mandate ran on November 10. The exit note run started November 10 and finished just after midnight UTC on November 11. One run of each does not establish whether either edit changes returns.
The free mandate replaced the baseline's medium-term investor framing. It allowed any trading style and holding period, treated long and short as equal options, and said rank was determined by final equity. It also removed four lines: the one asking for an investment thesis on every new position, the one asking for a daily re-check of open theses, the one saying doing nothing is a valid decision, and the one saying uninvested cash is a position.
The exit note added a sentence to the stop rule. It said the stop is checked every 15 minutes on the live price and that a level only a couple of typical daily moves from entry is likely to be touched within days. The two edited runs also used a different assignment of codes to assets from the baseline runs.
Against Run 1 alone, the edits looked like gains of +21.18 points and +18.57 points. The free mandate was 9.54 points above the best baseline run, and the exit note was 6.93 points above it. Both differences were smaller than the baseline's own 11.64-point spread. We chose that comparison after seeing the repeats. Neither edit has been repeated, so its own spread is unknown.
In all six comparisons of an edited run against a baseline run, the 95% interval for the difference in next-day hit rate included zero. Both edited runs scored worse than Run 2 on the price-range score, with 95% intervals that exclude zero. The free mandate's score was 3.35% worse than Run 2's, with an interval of +0.43 to +6.33. The exit note's was 3.27% worse, with an interval of +0.04 to +6.77. Against Run 1 and Run 3, the intervals included zero. Two identical baseline runs differed by at most 1.67% on this score. Read by days won and lost, three of the six comparisons had p below 0.05, each against the edited run. We noticed the price-range result after the runs. Neither edit improved the forecasts.
How positions were closed did change, which we noticed after the runs. The exit note run had 10 stop exits, against 22, 14 and 26 for the baseline runs. The model closed 14 positions itself, against 4, 5 and 5.
We added the exit note to the live arena prompt on November 11, 2026. The written reason for that change cited one lab run of each prompt: where the model placed its stops, and the drop in stop exits from 22 to 10. With the repeats, 10 is still below all three baseline runs, which had 22, 14 and 26.
Sonnet 5.5, Trained Through June 2026, Did Not Beat the Drift Formula
A model whose training data covers a test window can look skilled by recalling it. Sonnet 5.5's training data runs through June 2026, according to Anthropic's models overview, and the window ends June 23, 2026. A backtest of Sonnet 5.5 on this window is not valid. We used its runs as a recall test, once with names and dates shown and once with them hidden. Sonnet 5's training data runs through January 2026, according to Anthropic's page for Sonnet 5, before the window.
Sonnet 5.5 did not beat the drift formula with names shown or names hidden. With names shown, it won 57 days, lost 68 and tied 17 against the formula, with p = 0.37. With names hidden, it won 50 days, lost 63 and tied 28, with p = 0.26. The names-hidden row covers 141 days. On March 29, that run gave equal chances of up and down on all 5 of its forecasts, so the day has no direction call to count. The test found no sign of recall in forecast accuracy in either run.
Names shown minus names hidden gave a -0.32-point difference in next-day direction hit rate. The 95% interval was -3.27 to +2.33, across the 950 next-day calls both runs made, which fall on 141 days. The rule written beforehand required the difference to be above zero. It required the interval to exclude zero. It also required the difference to exceed 2.72 points, the largest gap between the Sonnet 5 runs. The result met none of the three.
The names-hidden run was not blind. Our rule, set beforehand, said that if Sonnet 5.5 named more than 10% in the naming check, the run was not blind and the comparison would understate recall. We showed it 20 of Run 1's prompts and asked it to name the asset behind each code. One question was one code in one prompt. Sonnet 5.5 named the asset correctly on 360 of 1,234 questions, or 29.17%. It was right on 170 of 220 questions about crypto codes and 190 of 1,014 about stock codes. ETH, BNB, XRP and DOGE were named correctly in all 20 prompts. The comparison is therefore weak evidence and cannot rule recall out.
Sonnet 5 answered the same questions on the same prompts. It named the asset correctly on 407 of 1,234 questions, or 32.98%. Its training data ends before the window, so naming an asset does not hand it the window's outcomes. For both models, in every run in this article, names hidden hides less than the label suggests. The prompts hide names and dates but show prices. A price level is enough to place a large asset.
The first version of this check reported 1.70%. A counting error in the lab caused it, and we found it while checking this article: the check counted each code once across all 20 prompts, then divided by all 1,234 questions. Runs 2 and 3 and both Sonnet 5.5 runs had already finished. We changed neither the rule nor the runs after the correction.
Sonnet 5.5 Opened One Position in Each Run
Sonnet 5.5 opened one position in each of its 142-day runs. With names hidden, it opened an ETH long on March 17 at 10% of equity and confidence 0.80, with a stop at 2,100. The stop exit on March 19 realized -$96.43. The account was fully in cash on 140 of 142 days.
With names shown, it opened a JPM long on June 23, the last day of the window. The position was 15% of equity, at confidence 0.80. The account was fully in cash on 141 of 142 days. The -0.02% return was the fee and slippage on that one fill.
Sonnet 5 opened 29, 24 and 36 positions under the same baseline prompt. When Sonnet 5.5 held, its written reason named the 0.80 confidence bar in 56 of 76 hold decisions with names hidden and in 51 of 60 with names shown. Staying in cash returned about zero while buy and hold BTC lost 20.33% and buy and hold SPY gained 5.49%.
Both models ran at the medium effort setting, and that setting does not mean the same amount of work on both. Sonnet 5.5 wrote an average of 2,187 and 2,479 tokens per call. Sonnet 5 wrote 5,752, 4,424 and 5,296.
“No position; crypto is oversold (RSI ~24) but still falling, and I have no 0.80-confidence entry, so I stay in cash.”
Sonnet 5 Put Confidence at 0.82, Just Above the 0.80 Bar
Sonnet 5 put confidence at 0.82 on 129 of 154 opens across its five runs, or 84%. We noticed this after the runs. The rule required at least 0.80 to open. The other opens were 12 at 0.81, 9 at 0.80 and 4 at 0.85.
Across the three baseline runs alone, Sonnet 5 made 89 opens. It used 0.82 on 75, 0.81 on 6, 0.80 on 6 and 0.85 on 2. None was above 0.85. Sonnet 5.5 made 2 opens, both at exactly 0.80.
The number landing two hundredths above the bar on 84% of opens does not look like a measured probability. It reads as a pass mark. Sonnet 5 wrote the pass mark and traded. In most of its written holds Sonnet 5.5 said nothing reached the bar, and it did not trade. Under the baseline prompt, Sonnet 5 opened 24 to 36 positions per run and Sonnet 5.5 opened 1 per run. That comparison mixes model, training data and effort.
The Lab Repeats the Unchanged Prompt
We do not judge a prompt change on the return of one run. We measure the forecast record against the drift formula, reading the result day by day with the paired sign test. A repeat of the unchanged prompt sets the size of difference that counts.
We do not backtest a model on a window inside its training data. The Sonnet 5.5 runs here are a recall test, and we label them that way. The lab's separate holdout period was not touched by any run in this article.
An earlier article reported a wider gap. Two backtests of Jev over this window, which differed by a renamed input field and fresh samples, finished 35 points apart.
One live season is also one run per model. The live LLM trading benchmark runs every model forward on prices that have not happened yet. You can follow it on the live leaderboard.
Limits of These Runs
This study has three baseline runs, one run of each edit and one Sonnet 5.5 run with names shown and one with names hidden. It covers one window of 142 days, when BTC fell 20.33% and SPY rose 5.49%.
June 2026 is Sonnet 5.5's cutoff month. A cutoff says what a model could have seen. It does not say how much of the last months it retained. When we say recall was not shown, we mean one run with names shown and one with names hidden could not show it. That does not mean Sonnet 5.5 has no memory of the period. The 2.72-point gap used in the recall test was measured on Sonnet 5 and applied to Sonnet 5.5.
The baseline runs used the same model name and client version. A provider-side change between November 10 and November 12 cannot be ruled out. The replay fills at the daily close and checks stops on daily bars. The live arena checks stops every 15 minutes. The replay is less informed than live. These were paper trades with no real money.
This article describes backtests and simulated trades and is for educational and entertainment purposes. It is not financial advice. Backtests and simulated results do not predict future results.