Data lab

RL environments, evals and benchmarks for quant finance and crypto

We build tasks that check whether an AI agent does quantitative research honestly. Each one ships with synthetic data we generate and own, a reference solution and an automatic grader, and each one targets a mistake we made and documented in our own research.

Request a pilot → Ask about an agent evaluation

What you can buy

Prices in USD. Every engagement gets a fixed quote in writing after a scope call.

  • Pilot pack

    from $7,500typically $7,500–12,500 · per pack

    25 private, calibrated tasks in quant finance or crypto, in Harbor/TestGen format, with a calibration report. Non-exclusive licence.

    Included

    • 25 tasks with oracle and verifier
    • Calibration report
    • QC record per task
    • One round of rework
  • Custom generator

    from $15,000typically $15,000–40,000 · per project

    A seeded generator plus 100+ tasks in a domain you choose, such as perps and funding, order-book execution or time-series ML.

    Included

    • Seeded task generator
    • 100+ tasks
    • Calibration report
    • Licence or IP assignment by contract
  • Agent evaluation

    from $2,000typically $2,000–5,000 · per run

    We run held-out tasks from our benchmark against your agent and send you a failure report: which pitfalls it falls into, and where.

    Included

    • Held-out tasks, not public
    • Per-check failure report
    • Traces deleted after the project
  • Expert hours

    from $90typically $90–150 · per hour

    Quant review, rubric writing or task authoring by the hour, in your format.

    Included

    • Quant and crypto market structure
    • Backtest and ML leakage review
    • English or Spanish

Exclusive licence: the same pack withdrawn from our catalogue, at about 4× the non-exclusive price.

How we build a task

  1. 01

    Start from a real failure

    Each task begins with a mistake documented in our own research, such as a fill rate of 2.6% live against 30–38% simulated, or drift that looked like edge.

  2. 02

    Generate the data

    Seeded synthetic data, built so the leaky answer looks good and the honest one does not. Same seed, same bytes.

  3. 03

    Write the reference solution

    An oracle solves the task end to end and must pass. A null agent and a naive agent that ignores the pitfalls must fail.

  4. 04

    Grade with code

    The number within a tolerance, error codes scored on precision and recall, the agent’s code re-run on a hidden series, invariance tests. No LLM judge.

  5. 05

    Check for leaks

    A scan for expected values, grader files and solutions in anything the agent can see, and alternative seeds to show the gap is structural.

  6. 06

    Calibrate

    Several models, several rollouts each. Tasks that are too easy get hardened or dropped, and you get the report.

What you receive

  • Task packages in Harbor/TestGen format: instruction.md, task.toml, a pinned offline Dockerfile, data, the oracle (solution/solve.sh) and the verifier (tests/).
  • A calibration report: pass rates per model over several rollouts, and which checks each model fails.
  • A QC record per task: the oracle passes, null and naive baselines fail, builds are deterministic, alternative seeds hold, the leak scan is clean.
  • A canary string in every text file, so contamination can be detected.

Licensing and data provenance

Data
100% synthetic, generated by our own code from hand-written parameters.
Never included
Exchange or venue market data, raw or derived; scraped social media; identified wallets; personal data; anything made for another client.
Authoring
Designed and reviewed by a quant researcher, with AI assistance. Every task passes the automated QC above before delivery.
Licence
Non-exclusive by default. Exclusive licence or full IP assignment by contract. Private tasks carry a no-publication obligation.
Public benchmark
When released: code under Apache-2.0, data under CC-BY-4.0.

Where we are

Our benchmark is a 10-task pilot. The reference solution passes all 10 and the naive baseline fails all 10, but it has not yet been calibrated against frontier models or published. We will keep saying so until it has.

See the benchmark →

Request a pilot → Ask about an agent evaluation