Data lab
RL environments, evals and benchmarks for quant finance and crypto
We build tasks that check whether an AI agent does quantitative research honestly. Each one ships with synthetic data we generate and own, a reference solution and an automatic grader, and each one targets a mistake we made and documented in our own research.
Request a pilot → Ask about an agent evaluation
What you can buy
Prices in USD. Every engagement gets a fixed quote in writing after a scope call.
Pilot pack
from $7,500typically $7,500–12,500 · per pack
25 private, calibrated tasks in quant finance or crypto, in Harbor/TestGen format, with a calibration report. Non-exclusive licence.
Included
- 25 tasks with oracle and verifier
- Calibration report
- QC record per task
- One round of rework
Custom generator
from $15,000typically $15,000–40,000 · per project
A seeded generator plus 100+ tasks in a domain you choose, such as perps and funding, order-book execution or time-series ML.
Included
- Seeded task generator
- 100+ tasks
- Calibration report
- Licence or IP assignment by contract
Agent evaluation
from $2,000typically $2,000–5,000 · per run
We run held-out tasks from our benchmark against your agent and send you a failure report: which pitfalls it falls into, and where.
Included
- Held-out tasks, not public
- Per-check failure report
- Traces deleted after the project
Expert hours
from $90typically $90–150 · per hour
Quant review, rubric writing or task authoring by the hour, in your format.
Included
- Quant and crypto market structure
- Backtest and ML leakage review
- English or Spanish
Exclusive licence: the same pack withdrawn from our catalogue, at about 4× the non-exclusive price.
How we build a task
- 01
Start from a real failure
Each task begins with a mistake documented in our own research, such as a fill rate of 2.6% live against 30–38% simulated, or drift that looked like edge.
- 02
Generate the data
Seeded synthetic data, built so the leaky answer looks good and the honest one does not. Same seed, same bytes.
- 03
Write the reference solution
An oracle solves the task end to end and must pass. A null agent and a naive agent that ignores the pitfalls must fail.
- 04
Grade with code
The number within a tolerance, error codes scored on precision and recall, the agent’s code re-run on a hidden series, invariance tests. No LLM judge.
- 05
Check for leaks
A scan for expected values, grader files and solutions in anything the agent can see, and alternative seeds to show the gap is structural.
- 06
Calibrate
Several models, several rollouts each. Tasks that are too easy get hardened or dropped, and you get the report.
What you receive
- Task packages in Harbor/TestGen format: instruction.md, task.toml, a pinned offline Dockerfile, data, the oracle (solution/solve.sh) and the verifier (tests/).
- A calibration report: pass rates per model over several rollouts, and which checks each model fails.
- A QC record per task: the oracle passes, null and naive baselines fail, builds are deterministic, alternative seeds hold, the leak scan is clean.
- A canary string in every text file, so contamination can be detected.
Licensing and data provenance
- Data
- 100% synthetic, generated by our own code from hand-written parameters.
- Never included
- Exchange or venue market data, raw or derived; scraped social media; identified wallets; personal data; anything made for another client.
- Authoring
- Designed and reviewed by a quant researcher, with AI assistance. Every task passes the automated QC above before delivery.
- Licence
- Non-exclusive by default. Exclusive licence or full IP assignment by contract. Private tasks carry a no-publication obligation.
- Public benchmark
- When released: code under Apache-2.0, data under CC-BY-4.0.
Where we are
Our benchmark is a 10-task pilot. The reference solution passes all 10 and the naive baseline fails all 10, but it has not yet been calibrated against frontier models or published. We will keep saying so until it has.