Dockerless: Environment-Free Program Verifier for Coding Agents

Wenhao Zeng, Yuling Shi, Xiaodong Gu, Chao Hu, Chaofan Wang, Yuhao Cui, Hongting Zhou, Mengnan Qi, Jianqiao Wangni, Zhaojian Yu, Shuzheng Gao, Kai Cai, Shilin He

Dockerless replaces per-repository test execution with agentic repository exploration to enable environment-free coding agent training.

How can we train coding agents using verification rewards without the massive compute and setup overhead of per-repository Docker environments?

Coding agents typically rely on per-repository Docker environments to verify patches, but building these environments is prohibitively expensive and often impossible for legacy or private codebases. Dockerless is an environment-free verifier that replaces test execution with agentic repository exploration: it generates diagnostic questions about a patch, dispatches sub-agents to gather evidence via shell tools, and aggregates that evidence into a correctness score. This approach enables a fully environment-free post-training pipeline that matches the performance of standard environment-based methods while requiring zero per-repository setup.

Paper Primer

Dockerless operates in two stages: first, it derives verification questions from the issue and reference patch; second, it dispatches parallel sub-agents to explore the repository using read-only shell commands. The verifier is like a detective: it doesn't just look at the crime scene (the patch), it actively interviews witnesses (the codebase) to confirm the fix's validity.

Dockerless significantly improves patch verification accuracy over existing methods.

On a trajectory-level verifier evaluation benchmark, Dockerless achieves 81.0 AUC on SWE-bench Verified. It outperforms the strongest open-source verifier by 14.3 AUC points.

Fully environment-free post-training matches the performance of standard environment-based pipelines.

The resulting Dockerless-RL-9B model achieves a 62.0% resolve rate on SWE-bench Verified. It surpasses the Qwen3.5-9B baseline by 2.4 points on Verified and 8.7 points on Multilingual benchmarks.

Why is this approach necessary if frontier models can already evaluate code?

Frontier models used as zero-shot judges rely on surface-level textual comparison, which fails to capture deep repository context like function call dependencies or module integration, leading to lower verification accuracy.

Does the agentic exploration add significant latency to the training pipeline?

No; while reward evaluation takes longer than simple textual scoring, it accounts for only 7.2% of the total per-rollout time, as the process is dominated by the agent's initial code generation rollouts.

The Bottleneck of Environment-Based Verification

We expose the verification bottleneck caused by Docker environments and introduce Dockerless, an environment‑free verifier.

Current program verifiers rely on executing unit tests inside per‑repository Docker containers, incurring heavy engineering and runtime costs. Dockerless replaces these environments with an LLM‑based verifier that generates and answers verification questions to score patches without any execution.

Running a test suite inside a Docker image for each repository requires building custom images, resolving dependencies, and scripting test execution, which dramatically slows down verification and limits the number of patches that can be evaluated.

A benchmark collection of real‑world software engineering issues, each paired with a reference patch and test suite, used to evaluate coding agents and their verification components.

**Figure 1.** Comparison of verifiers for SWE agents. Docker-based tests are accurate but depend on costly per-repository environments. LLM scorers sidestep that cost but score patches based on surface-level information, without actively inspecting the repository. Dockerless instead deeply explores the codebase to judge the patch, requiring no per-repository environment while retaining repository grounding.

The high cost of Docker‑based verification limits the scale of agent training.

The Dockerless Verification Engine

Dockerless replaces costly Docker verification with a two‑stage LLM‑driven question‑answer judge.

Per‑repository Docker environments dominate verification cost, making large‑scale post‑training impractical.

Think of the verifier as a detective: it first drafts a handful of probing questions about the bug, then sends parallel assistants to fetch evidence from the codebase before delivering a final verdict.

Sub‑agent 1 runs

Sub‑agent 2 runs

The judge receives $(Q_1,A_1)$ and $(Q_2,A_2)$, concatenates them with $x$, $y_{\text{ref}}$, and $y$, and computes logits $\ell_0=-0.3$, $\ell_1=1.2$.

Applying the scoring formula yields $r_\phi(x,y)=\exp(1.2)/(\exp(-0.3)+\exp(1.2))\approx0.88$, indicating high confidence that $y$ resolves $x$.

The verifier’s confidence stems from concrete evidence (file location and test name) rather than a blind string similarity between $y$ and $y_{\text{ref}}$.

How does Dockerless differ from the traditional execution‑based verifier that runs tests inside a Docker image?

Dockerless never spins up a full container; instead it asks targeted questions, gathers static evidence with read‑only tools, and makes a judgment based on that evidence. The execution‑based verifier, by contrast, actually runs the candidate patch against the test suite, incurring the full Docker build and test runtime.

RFT treats the verifier as a filter: it scores a large pool of rollouts collected in a minimal base image, then keeps only the top‑K patches that receive the highest $r_\phi$ scores before fine‑tuning.

Why does keeping only the top‑K rollouts improve fine‑tuning compared to using all collected rollouts?

Because the verifier’s score reflects how well a patch aligns with the issue and the repository context; discarding low‑scoring patches removes noisy or incorrect examples, letting the model focus on high‑quality corrective behavior.

In the RL stage, the verifier supplies a reward for each rollout; the policy is updated with GRPO using these rewards, eliminating the need for test execution.

How does using Dockerless as a reward source differ from conventional RL that relies on executed test outcomes?

Conventional RL must run the candidate patch in a full environment to observe a pass/fail signal, which is costly. Dockerless instead provides a learned probability that the patch resolves the issue, based on static evidence, enabling reward computation without any execution.

**Figure 2.** Architecture of Dockerless. The verifier takes the issue $x$, reference patch $y_{ref}$, and candidate patch $y$, and proceeds in two stages. (1) Question generation and exploration: the verifier first generates $K$ verification questions and dispatches parallel sub-agents to collect evidence-backed answers from the codebase. (2) Judgment: the verifier conditions on the issue, the patches, and the collected $(Q_k, A_k)$ pairs to produce a binary verdict token, whose logits define the continuous score $r_{\phi}(x, y)$.

**Figure 3.** Training pipeline for Dockerless: teacher-generated question-answer-judge trajectories are rejection-sampled by matching the predicted verdict against the ground-truth, and used to fine-tune a base model.

**Figure 4.** Env-free post-training pipeline for Dockerless. (A) Environment-free RFT: candidate rollouts are scored by Dockerless, and the top-K are kept to fine-tune the base model, yielding the SFT model. (B) Environment-free RL: starting from the SFT model, GRPO uses Dockerless as the per-rollout reward source, yielding the RL model.

Evaluation Framework and Results

Dockerless attains near‑execution accuracy while eliminating container overhead.

Dockerless RL model reaches 62.4% resolve rate on the Verified benchmark, surpassing the best open‑source baseline by 5.2%.

Table 1 shows the RL model (+Test‑Execution) achieving 62.4% versus 57.2% for the top open‑source model.

The verifier spins up the repository’s Docker image, runs the candidate patch, and checks whether the test suite passes.

Why is the execution‑based verifier slower than Dockerless?

It must launch a per‑repository Docker container for every patch, paying image download, container startup, and dependency installation overhead each time.

The verifier feeds the patch text to a large language model, which predicts the likelihood of passing the tests without actually running the code.

How does the LLM‑scorer differ from Dockerless verification?

LLM‑scorer predicts outcomes purely from the patch’s text, while Dockerless asks targeted verification questions and aggregates multiple LLM answers, yielding higher fidelity scores.

**Figure 5.** Verifier AUC vs. number of verification questions $K$ on SWE-bench Verified verifier evaluation benchmark.

**Figure 8.** Frontier-model resolve rate (%) on SWE-bench Verified, Multilingual, and Pro under env-based and env-free settings. Solid bars are env-free; hatched extensions show the additional gain from per-repository environments, so the full bar height equals the env-based score.

Dockerless achieves near‑execution‑level accuracy without the container overhead.

Performance Analysis and Ablations

Environment‑free training matches or exceeds the best env‑based baselines.

Recall that the bottleneck in coding‑agent training is the need for per‑repository Docker environments; Dockerless replaces those with an LLM‑based verifier that asks and answers $K$ verification questions.

Dockerless‑RL‑9B reaches 62.0 % Verified, 50.0 % Multilingual, and 35.2 % Pro resolve rates, improving the base Qwen3.5‑9B by +2.4, +8.7, and +2.9 points respectively.

Table 1 shows the 62.0/50.0/35.2 numbers for Dockerless‑RL‑9B and the 59.6/41.3/32.3 baseline.

Env‑free SFT matches env‑based SFT: Dockerless‑SFT‑9B attains 60.6 % Verified versus 60.0 % for the env‑based counterpart.

Table 1 compares Dockerless‑SFT‑9B (60.6) with Env‑SFT‑9B (60.0) on the Verified metric.

Dockerless‑RL‑9B performs on par with Test‑Execution RL, differing by only –0.4 % on the Verified metric (62.0 vs. 62.4).

Table 1 lists the 62.0 value for Dockerless‑RL‑9B and 62.4 for Test‑Execution RL.

Dockerless achieves the highest verifier AUC: 81.0 on SWE‑bench Verified and 72.1 on Multi‑SWE‑bench Flash, beating the next‑best open‑source verifier by +14.3 and +9.2 points.

Table 2 reports Dockerless AUC scores of 81.0 and 72.1, versus 66.7 and 62.9 for DeepSWE Verifier.

When filtering SFT data, Dockerless‑4K yields a Pro resolve rate of 35.3 %, surpassing Random‑4K’s 32.0 %.

Table 3 lists 35.3 for Dockerless 4K and 32.0 for Random 4K under the Pro column.

**Figure 9.** Per-language comparison of SFT (w/o env) and SFT (w/ env) resolve rate, aggregated across SWE-bench Verified, Multilingual, and Pro. Bubble size encodes the number of test instances per language. Points above the diagonal mark languages where SFT (w/ env) wins, below the diagonal where SFT (w/o env) wins.

**Figure 10.** Distribution of total per-rollout wall-clock time (rollout + reward) under three reward sources, on 7680 rollouts collected during RL training. All three sources produce near-identical distributions; the long right tail is set by slow rollouts, not by the verifier.

**Figure 6.** Per-rollout wall-clock breakdown during RL under three reward sources.

Across all ablations, removing the env‑free component either leaves performance unchanged (SFT) or incurs only a modest drop (RL), confirming that Dockerless‑generated verification questions provide a reliable substitute for per‑repository execution.

Contextualizing Coding Agents

We position Dockerless within prior software‑engineering agents and verification approaches.

Large language models have progressed from generating isolated code snippets to tackling full software‑engineering tasks, and most SWE agents are fine‑tuned with a two‑stage SFT‑then‑RL recipe built on scaffolds such as SWE‑agent and OpenHands.

Environment‑free rollout pipelines share a single base image across repositories, but they still restrict agents by exposing only a static tool set, simulating tool returns with learned models, or imposing prompt‑level execution limits.

Dockerless removes these constraints: the agent can issue any shell command in a minimal Linux image, and the same env‑free verifier replaces both the RFT‑stage filter and the RL reward source.

Case Study – Matplotlib offsetText color issue: the candidate patch updates both XAxis and YAxis initialization, preserving the inherit semantics for tick colors, and Dockerless assigns a high verification score (0.996) despite low textual similarity to the reference patch.

Verifier research for SWE agents spans execution‑trained classifiers, a 30B mixture‑of‑experts critic, group‑wise textual reasoning, and rubric‑supervised or RL‑distilled variants, none of which invoke tools or inspect the repository at scoring time.

Dockerless places the verifier itself as an agent that actively explores the repository before issuing a patch‑outcome verdict, thereby unifying verification and reward modeling without any per‑repository environment.

Supplementary Data and Failure Modes

Appendix details env‑free evaluation, dataset construction, training, and prompts, highlighting modest performance trade‑offs.

Env‑free evaluation removes per‑repository Docker images and replaces them with a minimal Ubuntu 22.04 LTS image. Across four frontier models and three benchmarks, the drop in resolve rate never exceeds 13.9 points, averaging only 7.1 points, and the strongest model stays within 3–4 points of its env‑based counterpart.

Table 4 shows that Dockerless‑RL‑9B remains the top sub‑10B model under env‑free conditions, achieving 53.8 % on Verified, 42.3 % on Multilingual, and 30.6 % on Pro. It outperforms Dockerless‑SFT‑9B by 1.2–1.8 points on each benchmark.

Comparing env‑base to env‑free scores reveals average drops of 9.4 points for Env‑SFT‑9B, 7.1 points for Dockerless‑SFT‑9B, and 6.8 points for Dockerless‑RL‑9B. The smaller gap for env‑free‑trained models indicates they are more robust to the distribution shift introduced by removing Docker.

Appendix C describes the three components of the dataset construction pipeline: verifier training data, env‑free rollout data, and a balanced verifier evaluation benchmark.

C.1 builds a corpus from execution‑labeled patches in SWE‑Gym and Multi‑SWE‑RL, filters to trajectories of 4–30 turns, caps the negative‑to‑positive ratio at 4:1, and ends up with 3.7 K unique issues. Each example contains a question‑generation trajectory, multiple answer trajectories, and a final judgment trajectory, all truncated to 10 000 characters.

C.2 collects 16 K env‑free rollouts on SWE‑Rebench‑v2 using a minimal Linux image, then selects the top 4 K globally ranked by Dockerless for downstream use.

C.3 assembles a verifier benchmark with 500 samples from SWE‑bench Verified and 276 from Multi‑SWE‑bench Flash, balanced 1:1 positive/negative, and labels each trajectory via the standard per‑repository Docker test suite.

Appendix D details the training procedures for the agentic verifier, the env‑free SFT model, and the env‑free RL policy.

D.1 fine‑tunes the verifier from Qwen3.5‑9B using cross‑entropy on the filtered trajectories, stopping after 150 optimizer steps with AdamW (lr 1e‑5 → 1e‑6, weight decay 0.01, batch 256, seq 32 768).

D.2 scores each candidate rollout with two independent agentic passes, averages the dense scores, and selects the top‑ranked 4 K rollouts (DRFT) for SFT training, which runs for three epochs with the same AdamW configuration.

D.3 initializes the RL policy from the SFT checkpoint and runs GRPO using Dockerless scores as rewards; each issue samples eight rollouts, each scored with two passes, and the group‑normalized advantages from equation 3 drive the update over 50 RL steps.

E. Per‑language analysis shows that the overall env‑free vs env‑base gap is small, but compilation‑heavy languages (Rust +7.0, C +13.3) exhibit larger drops, reflecting the loss of compiler diagnostics in the env‑free setting.

F. Latency distribution plots (omitted here) illustrate the runtime characteristics of the env‑free pipeline across different model sizes.

G. Prompt templates enumerate the exact prompts used by Dockerless for question generation, sub‑agent exploration, final scoring, and zero‑shot LLM‑as‑judge evaluation.

G.1 Question generation prompt instructs the model to act as an expert code reviewer, producing 2–4 diagnostic questions each tagged with a category and rationale.

G.2 Sub‑agent exploration prompt defines a read‑only shell loop, requiring a THOUGHT section and exactly one bash command per response, with no file modifications.

G.3 Final scoring prompt asks the model to evaluate the generated patch against the issue description and golden patch, outputting a binary verdict token inside an tag.

G.4 Zero‑shot LLM‑as‑judge prompt queries frontier LLMs directly with the issue, golden patch, and generated patch, parsing the binary verdict from the same tag format.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers