# NautilusBench, a Chartnaut benchmark

> A closed benchmark of AI models on trading research in Chartnaut. Bench v1, first published run, last updated 2026-09-29.

Models from AI labs are given 77 closed tasks: building indicators, definitions and studies in Chartnaut, running them, and reporting numbers. Every model runs in the same harness with the same tools and limits. An attempt is solved when the run it names exists and came from its own saved script, its reported answer matches that run, and the script re-run on hidden windows matches a certified answer. Tasks are never published.

## Pages
- [Leaderboard](https://evals.chartnaut.com/): scores, intervals, per-tier, per-area and per-category results, failure causes
- [Leaderboard data (JSON)](https://evals.chartnaut.com/leaderboard.json): the aggregate results the page is built from

## Results
1. Opus 5.5 (Anthropic): solved 96.1% (95% interval 90.9% to 100.0%), tier-weighted 94.3%, efficiency-weighted 82.3%, run 2026-09-27
2. GPT-6 Sol (OpenAI): solved 96.1% (95% interval 90.9% to 100.0%), tier-weighted 95.0%, efficiency-weighted 84.0%, run 2026-09-28
3. GPT-6 Astra (OpenAI): solved 96.1% (95% interval 90.9% to 100.0%), tier-weighted 95.6%, efficiency-weighted 91.1%, run 2026-09-29
4. GPT-5.6 Sol (OpenAI): solved 94.8% (95% interval 89.6% to 98.7%), tier-weighted 94.3%, efficiency-weighted 77.1%, run 2026-09-28
5. GPT-5.6 Terra (OpenAI): solved 94.8% (95% interval 88.3% to 100.0%), tier-weighted 93.7%, efficiency-weighted 90.8%, run 2026-09-28
6. Sonnet 5 (Anthropic): solved 89.6% (95% interval 80.5% to 96.1%), tier-weighted 86.8%, efficiency-weighted 85.1%, run 2026-09-27
7. GPT-5.6 Luna (OpenAI): solved 84.4% (95% interval 76.6% to 92.2%), tier-weighted 83.6%, efficiency-weighted 68.2%, run 2026-09-28
8. DeepSeek V4.1 Flash (DeepSeek): solved 80.5% (95% interval 71.4% to 88.3%), tier-weighted 74.2%, efficiency-weighted 77.8%, run 2026-09-28
9. Fable 5.1 (Anthropic): solved 79.2% (95% interval 70.1% to 88.3%), tier-weighted 74.2%, efficiency-weighted 70.0%, run 2026-09-27
10. GLM 5.3 Flash (Zhipu AI): solved 72.7% (95% interval 62.3% to 80.5%), tier-weighted 64.8%, efficiency-weighted 69.4%, run 2026-09-29
11. GPT-6 Luna (OpenAI): solved 64.9% (95% interval 53.2% to 75.3%), tier-weighted 57.2%, efficiency-weighted 51.5%, run 2026-09-29
12. Haiku 4.5 (Anthropic): solved 48.1% (95% interval 37.7% to 58.4%), tier-weighted 40.3%, efficiency-weighted 45.5%, run 2026-09-27