Benchmark
HonestQuant-Bench
Can an AI agent do honest quantitative research? Find the pitfalls, report the right number and refuse to ship what does not pass.
Most finance benchmarks we found test Q&A, trading returns or whether a backtest runs. This one plants the mistakes that turn a losing backtest into a winning one, and checks whether the agent catches them.
Five task families
- A
Audit
Find and fix what is wrong with a backtest that looks too good, and report the honest number.
- B
Build
Build a strategy from a spec where the natural path falls into a trap such as survivorship or omitted funding.
- C
Decide
Ship or not, given a log of trials and a holdout: multiple testing, a reused holdout, beta passed off as alpha.
- D
Simulation vs live
Estimate what a strategy can really fill: price at the signal, latency, queue position, one side of the book.
- E
ML leakage
Estimate out-of-sample skill honestly: shuffled time-series splits, overlapping labels, selection on the full sample.
How grading works
- Numbers: the corrected metric must match the reference solution within a tolerance.
- Error codes: the report uses a closed list of codes, scored on precision and recall, so listing everything does not pay.
- Hidden re-run: the grader runs the agent’s code again on a fresh series with another seed. A hand-tuned number does not survive.
- Invariance: cutting off the future must not change past signals.
- ML tasks: the agent’s model is scored on hidden future rows, and its own estimate must match what it really achieves.
- Binary reward: a task passes only if every check passes. No LLM judge.
Pilot status
- 10
- tasks, 2 per family, all auto-graded
- 10/10
- passed by the reference solution
- 0/10
- passed by a null agent, and 0/10 by a naive baseline that ignores the pitfalls
- 30/30
- reference passes on 3 alternative seeds per task; the naive baseline fails all 30
Not done yet: model calibration (several models, 4 rollouts each) and the public release. We will publish the results, including any tasks that turn out too easy.
Leaderboard
| Model | Pass rate | Rollouts |
|---|---|---|
| Results coming. No model has been scored yet; this table fills in after calibration. | ||
Code and data
Hugging Face · chaskilabs GitHub · chaskilabs
Repositories are not published yet. These links go to our organisation pages.
Planned licences: code under Apache-2.0, data under CC-BY-4.0. Every text file carries a canary string.