Benchmark

HonestQuant-Bench

Pilot · public release coming (working name)

Can an AI agent do honest quantitative research? Find the pitfalls, report the right number and refuse to ship what does not pass.

Most finance benchmarks we found test Q&A, trading returns or whether a backtest runs. This one plants the mistakes that turn a losing backtest into a winning one, and checks whether the agent catches them.

Five task families

  • A

    Audit

    Find and fix what is wrong with a backtest that looks too good, and report the honest number.

  • B

    Build

    Build a strategy from a spec where the natural path falls into a trap such as survivorship or omitted funding.

  • C

    Decide

    Ship or not, given a log of trials and a holdout: multiple testing, a reused holdout, beta passed off as alpha.

  • D

    Simulation vs live

    Estimate what a strategy can really fill: price at the signal, latency, queue position, one side of the book.

  • E

    ML leakage

    Estimate out-of-sample skill honestly: shuffled time-series splits, overlapping labels, selection on the full sample.

How grading works

  • Numbers: the corrected metric must match the reference solution within a tolerance.
  • Error codes: the report uses a closed list of codes, scored on precision and recall, so listing everything does not pay.
  • Hidden re-run: the grader runs the agent’s code again on a fresh series with another seed. A hand-tuned number does not survive.
  • Invariance: cutting off the future must not change past signals.
  • ML tasks: the agent’s model is scored on hidden future rows, and its own estimate must match what it really achieves.
  • Binary reward: a task passes only if every check passes. No LLM judge.

Pilot status

10
tasks, 2 per family, all auto-graded
10/10
passed by the reference solution
0/10
passed by a null agent, and 0/10 by a naive baseline that ignores the pitfalls
30/30
reference passes on 3 alternative seeds per task; the naive baseline fails all 30

Not done yet: model calibration (several models, 4 rollouts each) and the public release. We will publish the results, including any tasks that turn out too easy.

Leaderboard

Pass rate on the public split
ModelPass rateRollouts
Results coming. No model has been scored yet; this table fills in after calibration.

Repositories are not published yet. These links go to our organisation pages.

Planned licences: code under Apache-2.0, data under CC-BY-4.0. Every text file carries a canary string.

Want your agent evaluated before the public release?

Ask about an agent evaluation →