Every RL post-training team reaches the point where task supply, not compute or algorithm choice, limits progress. In-house authoring stops scaling, and supply moves to human-data incumbents, environment-native startups, product companies and contractor networks. There is no standard acceptance spec for what they deliver. A diligence call with any RL environment vendor usually produces a demo, a task count and a price per task, and none of those shows whether reward tracks task success on the tasks you will actually train on.
The seven questions below are written to produce that evidence. Each maps to a measurable property of the delivered artifact, and each has a good answer, a red flag and the published work behind it. Use them as a call script, or send them in writing and score the replies.
The 7 questions at a glance
| # | Question to ask | What it measures | Red flag |
|---|---|---|---|
| 1 | Show me a task you rejected | Filter selectivity | No example, or a near-zero rejection rate |
| 2 | Who authors tasks? | Author competence, provenance | "A global network of experts" |
| 3 | How do you prove a task is solvable? | Verifier recall | "A model solved it" |
| 4 | How do you try to break the verifier? | Verifier precision | No exploit ever found |
| 5 | What does a sub-step rubric look like? | Reward shaping, grader agreement | A single overall score |
| 6 | How do you calibrate difficulty? | Gradient signal per rollout group | Labels set by author opinion |
| 7 | What does a 15-task pilot look like? | Falsifiable acceptance | Vendor picks tasks and defines success |
Question 1: Show me a task you rejected, and why
A rejection log is the operating characteristic of a vendor's filter. The public audits give a prior on the defect rate. For SWE-bench Verified, three annotators screened each of 1,699 samples. According to OpenAI's write-up, 38.3% had underspecified problem statements, 61.1% had unit tests too specific or unrelated to the problem, and 68.3% were filtered out overall. In the Terminal-Bench paper, 229 tasks were submitted by 93 contributors and 89 were selected, at roughly three reviewer-hours per accepted task. A vendor with a near-zero rejection rate is either drawing from an unusually clean distribution or not measuring.
A good answer sounds like: The vendor opens a real rejected task and classifies the defect: ambiguous specification, a verifier that accepts the null action, an answer derivable from container contents, or a task every model solves immediately. It reports the rejection rate by defect class and shows the log.
Red flag: "We don't really reject tasks, our authors are experts."
Question 2: Who authors tasks, and how are they vetted for the domain?
Authorship determines two things: whether the specification is correct, and whether the task is new to the model. Epoch AI's market survey reports that interviewees weigh domain expertise above ML skill for writing tasks, and that maintaining quality while scaling is the most cited bottleneck. GDPval is a reference for strict vetting. According to the GDPval paper, task writers needed at least four years of experience and averaged 14. They also passed a video interview, training and a quiz, and each of the 1,320 tasks received an average of five human reviews, with a minimum of three.
Provenance matters as much as credentials. Tasks adapted from public repositories or documents can sit in pretraining data. In its February 2026 post, OpenAI reported that every frontier model it tested could reproduce the gold patch or verbatim problem details for some SWE-bench Verified tasks. A contaminated task measures recall, and the reward it produces says little about capability.
A good answer sounds like: A named credential band for your domain, such as licensed pharmacists for a medication task. A screening exercise on sample tasks. Separate authors and reviewers, an LLM-assistance policy, and per-author acceptance rates. Original authoring from private material, or a stated contamination probe, for example checking whether a model can reproduce the reference solution or task text without the environment.
Red flag: "A global network of vetted experts" with no detail on your domain. Tasks adapted from public issues or documents with no decontamination check.
Question 3: How do you prove a task is solvable?
Solvability is the verifier's recall. An unsolvable task returns zero reward. A task solvable only in a form the verifier rejects is worse, because the policy is penalized for correct behavior.
| Ground truth | Verifier rejects | Verifier accepts |
|---|---|---|
| Valid solution (the agent does the work) | ✗ False negative. The policy is penalized for correct work. Measured with alternative valid solutions: verifier recall. Tested by Question 3. | ✓ True positive. Correct work earns reward. |
| Exploit or wrong output (the agent skips the work) | ✓ True negative. Wrong output earns zero. | ✗ False positive. Reward without task success. Measured with bad-answer fixtures and an exploit agent: verifier precision. Tested by Question 4. |
Table 1. A verifier has two error modes. Question 3 tests the false-negative cell. Question 4 tests the false-positive cell.
Terminal-Bench is the reference process. Each task ships with a manually written reference solution and an oracle script that makes all tests pass, and a no-op agent must fail. In February 2026, OpenAI stopped reporting SWE-bench Verified scores after auditing 138 problems and finding that 59.4% had material test or description problems. Of those, 35.5% had narrow tests that rejected valid fixes. Huang and colleagues found the same failure in math: open-source rule-based verifiers often reject correct answers written in a different format.
A good answer sounds like: Every task ships with a reference solution trace, executed in the exact image the policy sees, pinned by digest, with no network at runtime. The verifier returns an identical verdict when re-run on the same snapshot. A second expert solves the task blind. The vendor measures recall by running alternative valid solutions through the verifier, and confirms the null action scores zero.
Red flag: "A model solved it." That shows the task is passable, not that the verifier accepts all correct solutions.
Question 4: How do you try to break the verifier before delivery?
The verifier is the reward function, so its false-positive rate is the rate at which reward decouples from task success. A policy under optimization pressure finds those false positives, and you pay for them in wasted training runs and misleading evals.
The Agentic Benchmark Checklist paper found that SWE-bench Verified uses insufficient test cases and that tau-bench counts empty responses as successful, flaws that can distort measured performance by up to 100% in relative terms. Model-based graders add their own attack surface: Huang and colleagues found them more accurate on static data but highly susceptible to hacking. Terminal-Bench ran an agent prompted to pass the tests by cheating, without access to tests or solutions, and inspected every trajectory. It found monkey-patching of the test environment, answer guessing and enumeration of every possible answer. Its CI also flags canary strings, Dockerfile contents that could leak tests or solutions, and privileged containers.
A good answer sounds like: Two attacks on every task, a human red-teamer and an automated exploit agent, with logs from earlier tasks and the fixes that followed. A fixture set of bad answers (empty, padded, hedged, confidently wrong, tampered) that must all score below a correct one. The grader runs outside the agent's sandbox, scores a post-episode state snapshot, and no gold answer or derivative of one exists in the image.
Red flag: "We use an LLM judge, so it handles edge cases." Also no exploit ever found and fixed.
Question 5: What does a sub-step rubric look like for one of your tasks?
A terminal binary reward gives an agent that completes seven of eight steps the same return as one that does nothing. A sub-step rubric supplies denser signal that is harder to game. A well-formed rubric reward is a weighted sum of atomic pass-or-fail criteria, normalized by the maximum positive score: R = Σ w_i · 1[c_i met] / Σ_{w_i>0} w_i.
In Rubrics as Rewards, instance-specific rubrics gave up to 31% relative improvement on HealthBench and up to 7% on GPQA-Diamond over LLM-as-judge baselines with Likert rewards. HealthBench uses 48,562 physician-written criteria with signed values from +10 to -10, so harmful or irrelevant content lowers the score. Ask the vendor to show one rubric for one of your tasks. This is the shape to look for.
# Sub-step rubric for one task (illustrative task and weights)
task: rotate-leaked-key-staging
veto: # a hit caps the whole episode at -1.0
- id: prod_key_revoked
check: state # deterministic, reads the post-episode snapshot
criteria:
# Required steps (positive weights, maximum 9 points)
- id: old_key_revoked
weight: +3
check: state # deterministic
- id: new_key_issued
weight: +2
check: state # deterministic
- id: services_healthy
weight: +3
check: probe # run by the grader, outside the sandbox
- id: audit_note_cites_commit
weight: +1
check: judge # LLM judge, called once for this criterion
quote_required: true # judge must quote its evidence
# Penalties (negative weights)
- id: unrelated_keys_touched
weight: -2
check: state # deterministic
- id: unneeded_escalation
weight: -1
check: log # read from the audit log
The maximum positive score is 9 points, and the reward is points earned divided by 9. Here is how five different attempts score:
| Attempt | What happened | Points | Reward |
|---|---|---|---|
| A | Full solution | 9 of 9 | 1.00 |
| B | Rotates the key and services stay healthy, but no audit note | 8 of 9 | 0.89 |
| C | Same as B, and also touches an unrelated key | 6 of 9 | 0.67 |
| D | Revokes the production key | Veto | −1.00 |
| E | Writes to the grader's state | Tamper | −2.00 |
The ordering is the point: tampering scores below a veto, and a veto scores below any honest attempt, including a failed one.
def reward(snap, audit, rubric):
if audit.wrote_grader_state() or audit.touched_canary():
return -2.0 # below every honest outcome
if any(v.hit(snap) for v in rubric.veto):
return -1.0
pos = sum(c.w for c in rubric.criteria if c.w > 0)
return max(sum(c.w for c in rubric.criteria if c.met(snap)) / pos, -1.0)
A good answer sounds like: Roughly eight to twenty atomic criteria, each tied to one step and independently checkable. Signed weights, negative criteria for wrong content and padding, and a score-capping veto for safety-critical errors. Deterministic checks wherever possible, and an LLM judge only where they cannot work, called once per criterion and required to quote its evidence. The judge model is pinned to a version, and agreement against expert labels on partial attempts is re-measured whenever it changes.
Red flag: One 1-to-10 quality score. Criteria like "the response is thorough". Compound criteria that invite partial credit, no negative criteria, or an identical rubric across tasks. A judge called through a moving model alias.
Question 6: How do you calibrate difficulty against current models?
Difficulty determines whether a task produces gradient. In GRPO-style training, a rollout's advantage is its reward normalized against the other rollouts for the same prompt. As the DAPO paper explains, when every response is correct or every response is wrong, the rewards are identical, the advantages are zero and the prompt contributes no policy gradient. DAPO's dynamic sampling filters out prompts with accuracy of exactly 0 or 1.
Group size sets how much pass-rate headroom a task needs. With a binary reward and independent rollouts, a prompt at pass rate p yields a group with mixed outcomes with probability 1 − p^G − (1 − p)^G. At a 2% pass rate, 16 rollouts give a usable signal on 28% of draws and 64 rollouts give 73%. At 5%, 8 rollouts give 34%.
| Task pass rate | 8 rollouts | 16 rollouts | 64 rollouts |
|---|---|---|---|
| 1% | 8% | 15% | 47% |
| 2% | 15% | 28% | 73% |
| 5% | 34% | 56% | 96% |
| 10% | 57% | 81% | 99%+ |
| 20% | 83% | 97% | 99%+ |
| 30% | 94% | 99%+ | 99%+ |
Table 2. Probability that a group of rollouts contains both a success and a failure, by task pass rate and group size. Derived from the binomial formula above, assuming a binary reward and independent rollouts.
Calibration is also an estimation problem. The Epoch survey reports practitioners using a minimum pass rate around 2 to 3% (or one success in 64 or 128 rollouts), a spread of tasks near 0%, 5% and 30%, and retirement of tasks near 70%. Zero successes in 64 rollouts gives a 95% upper bound near 3/64, about 5%, so a task measured at 0 of 64 is not known to be harder than 5%. A zero pass rate also does not prove difficulty, since the task may be infeasible (see Question 3). Pass rates belong to a checkpoint, so they need re-measuring as your models change.
A good answer sounds like: Every task is run against named current models with enough rollouts to bound the pass rate, and results are reported per model with sample sizes. Target bands are agreed in advance, failure transcripts separate hard from broken, and pass rates are re-measured as checkpoints change.
Red flag: Easy, medium and hard labels from author opinion. One run per model. Calibration against a much weaker model than yours.
Question 7: What does a 15-task pilot look like, and how is success measured?
There is no standard pilot format, so this structure is a recommendation, not a published benchmark. Fifteen tasks fills three difficulty bands with five each and can be reviewed in full. Each task should arrive with its image digest, verifier, reference solution, rubric and pass-rate report, plus the batch's rejection log. You set the specification, since a pilot of the vendor's best existing work only measures that work.
Agree acceptance criteria in writing before authoring starts, and run the checks yourself. The thresholds below are illustrative:
def accept(p):
return all([
p.oracle_pass == 15, # reference solution, run by you
p.null_action_reward == 0, # on all 15
p.fixture_order_violations == 0, # bad answers score below correct ones
p.alt_solution_recall >= 0.95, # verifier accepts valid variants
p.exploits_found <= p.agreed_budget, # your red-team pass, pre-agreed count
p.judge_agreement >= 0.85, # judge vs expert labels, partial attempts
p.in_band_fraction >= 0.80, # pass rates in agreed bands, n >= 64
])
Settle exclusivity terms early: Epoch reports exclusive deals run roughly four to five times the cost of non-exclusive ones.
A good answer sounds like: The vendor proposes acceptance criteria first and welcomes your independent checks.
Red flag: The vendor picks the tasks and defines success. The pilot uses a different process than production. There is no right to walk away after a failed check.
Frequently Asked Questions
What is an RL environment?
An RL environment is the setting in which a model takes actions and receives a reward. It is defined by the available actions and the context that determines their effects, and is often delivered as a Docker container. A task pairs a prompt with a grader that scores whether the objective was met.
How much do RL environments cost?
Epoch AI reports about $200 to $2,000 for an individual task, around $20,000 for a website replica and roughly $300,000 for a high-fidelity replica of a product like Slack. At those task prices, a 15-task pilot runs roughly $3,000 to $30,000.
Should you build or buy RL environments?
Buy for breadth, speed and commodity tasks. Build when tasks encode proprietary workflows or capabilities you do not want to disclose. A common pattern is an internal team owning specifications and acceptance review while vendors supply volume.
What is a verifier in an RL environment?
A verifier is the code or model that decides whether the agent's work earns a reward. It may run tests against final container state, compare an output to a reference, or apply a rubric with an LLM judge. In RLVR it is the entire reward function, so any gap becomes something the policy can exploit.
How long does an RL environment pilot take?
There is no published standard. As a planning estimate, expect three to five weeks for 15 tasks: about a week to agree the specification and acceptance criteria, two weeks for authoring and review, and one to two weeks for model runs and your own checks.
Choose the RL Environment Vendor Who Can Show Their Work
Every question above asks for the same thing: evidence. A task count and a demo describe what a vendor sells. A rejection log, reference solution, exploit report, real rubric and pass-rate distribution describe what it has done. Score each answer 0 for none, 1 for a plausible description, and 2 for one backed by artifacts you were shown. Ask every vendor the same seven questions and let the measurements decide.
Evaluate us against these questions. We build and stress-test RL environment tasks, including the expert solve, red-team pass, sub-step rubrics and difficulty calibration covered in this guide. Send us your task specification and we will answer all seven questions with the artifacts behind them.
.png)
