Facet: Preserving Source Intent and Executable State in Terminal Task Synthesis

Kou Shi, Zun Wang, Qisheng Su, Shiting Huang, Ziao Zhang, Zhen Fang, Qingnan Ren, Jin Liu, Yu Zeng, Yiming Zhao, Lin Chen, Zehui Chen, Feng Zhao

FACET synthesizes verifiable terminal tasks by grounding instructions, solutions, and verifiers in a shared, realized execution state.

How can we automatically synthesize high-quality, executable terminal tasks that maintain consistency between user instructions, environment state, and reference solutions?

Synthesizing terminal tasks is prone to "artifact drift," where instructions, solutions, and verifiers become mutually inconsistent because they are generated independently without a shared execution context. FACET addresses this by reconstructing coherent scenarios from heterogeneous skills and using a realized container state as a shared grounding interface for all task artifacts. Fine-tuning models on 1.2K trajectories from these tasks consistently improves performance across model scales, with the 27B model reaching 47.57 on Terminal-Bench 2.1.

Paper Primer

Terminal tasks are complex bundles of instructions, environments, solutions, and verifiers. When these components are generated in isolation, they often fail to align—for example, an instruction might reference a file that the environment never creates, or a verifier might test a state the solution cannot reach.

FACET (Fine-grained Agentic Construction of Executable Tasks) treats task synthesis as a coordinated workflow: it reconstructs multi-dimensional scenarios from existing skill packages, materializes the execution environment first, and then generates the instruction, solution, and verifier against that specific, realized state. This sequential, environment-grounded generation ensures that every artifact is contractually aligned with the actual files, schemas, and dependencies present in the container.

FACET provides data-efficient supervision that transfers across model scales.

Supervised fine-tuning on 1.2K successful trajectories improved Qwen3.5 models (4B, 9B, 27B) on Terminal-Bench 2.1. The 27B model achieved a 47.57 score, closing most of the performance gap to the 397B model while using 15x fewer parameters.

Environment-grounded construction significantly improves task validity.

A sequential generation order (Instruction → Solution → Verifier) achieved a 46.5% initial validity rate, compared to 24.2% for reverse-order generation. The sequential approach recovered 83% of tasks after repair, outperforming joint or reverse-order generation schemes.

Why is "executable-state grounding" necessary for terminal tasks?

It prevents cross-artifact contract mismatches. By exposing the realized container state as a shared interface, the framework ensures that the instruction, solution, and verifier are all generated against the same actual files, ports, and dependencies.

How does FACET handle failures during the synthesis process?

It uses a targeted repair pipeline. A router identifies the specific artifact responsible for a failure (e.g., environment, solution, or verifier) and triggers a repair only for that component, preserving the valid parts of the task bundle.

For terminal-agent training, the quality of the executable task bundle—specifically the alignment between the solution and the verifier—is more critical than the raw volume of synthetic data. FACET demonstrates that coordinating these artifacts through a shared environment state is a scalable path to high-quality supervision.

Introduction and Motivation

We expose the core gap: current terminal task synthesis misaligns instructions, environments, and verification.

Training agents that operate in terminal environments demands reliable, executable supervision. Existing pipelines generate task components independently, so mismatches between instruction, environment, solution, and verifier frequently render tasks invalid.

Current terminal‑task synthesis treats the four essential artifacts—instruction, environment, solution, and verifier—as separate outputs. Because they are produced without a shared grounding, inconsistencies easily arise, making the overall task unsolvable or incorrectly evaluated.

The disconnect between instruction and execution in current terminal task synthesis prevents reliable, scalable supervision.

The FACET Framework

We describe the FACET pipeline that turns skill collections into executable task bundles via a shared realized container state.

Independent generation of instructions, environments, and solutions often yields mismatched artifacts that cannot be executed together. The core pain is the lack of a shared grounding that guarantees consistency across all task components.

A Task Bundle packages everything needed to run a terminal task: the user instruction, the environment spec, the reference solution, a verifier, and runtime metadata.

How does this Task Bundle differ from a simple “instruction + code” pair commonly used in code generation benchmarks?

The bundle adds an explicit environment spec $E$ and a verifier $V$, plus runtime metadata $M$, which together guarantee that the solution $S$ can be executed and validated in the exact context it was generated for.

The FACET pipeline couples skill collection, scenario reconstruction, and reference building through the realized container state, ensuring that every artifact shares a common execution context.

Why not simply concatenate the collected skills into a single script instead of reconstructing a scenario?

Direct concatenation discards cross‑skill dependencies and intermediate states; the reconstruction step explicitly recovers these dependencies, preserving the logical flow required for a runnable task.

Initialize environment $e_0 = \text{Init}(E)$ where $E$ specifies a Ubuntu container with the \texttt{cp} utility.

Run solution $S$: execute \texttt{cp /data/src.txt /data/dest.txt} producing state $e_T$.

Verifier $V$checks that \texttt{/data/dest.txt} exists in$e_T$ and returns success.

Metadata $M$ records execution time $= 0.12\,$s and container image hash.

This tiny example shows how the same $e_0$ underlies both the solution and the verifier, guaranteeing that the generated instruction $I$ (“copy src to dest”) is realizable.

The reconstruction stage expands a raw skill combination into a compositional scenario by iteratively applying five specialized modules.

How does this multi‑module reconstruction differ from a single “scenario generation” model that directly outputs text?

By decomposing the process into explicit modules, the pipeline can enforce structural constraints (e.g., tool compatibility) and recover intermediate states, which a monolithic generator cannot guarantee.

The scenario $D_c$ is expressed as a set of five complementary descriptions, each capturing a distinct aspect of the task.

Why not collapse the five dimensions into a single “scenario description”?

Collapsing would force the model to embed heterogeneous information into one text block, increasing the risk of omitted dependencies and ambiguous state transitions.

**Figure 1.** The overall framework of Harbor, which consists of three stages: Information Source Acquisition, Scenario Reconstruction & Reference Building, and Environment-driven Task Construction.

Environment-Driven Construction

We detail how the environment-driven construction builds and validates executable tasks from a specification.

Stage 3 resolves the unreliability of terminal‑task synthesis by grounding construction in the realized container state. By first building a concrete environment, the method ensures that all subsequent artifacts share a common, observable context.

Instead of generating files, scripts, and tests in isolation, the method first builds a concrete container environment and then uses the observed state of that environment as shared context for all downstream artifacts.

How does this differ from the naïve approach of generating the instruction, solution, and tests without first building an environment?

The naïve approach treats artifact generation as independent text‑completion, which often yields mismatched file paths, missing dependencies, or non‑executable scripts. By constructing a concrete environment first, we obtain a real filesystem snapshot ($e_0$) that all later artifacts can reference, guaranteeing consistency and enabling precise, targeted repairs.

Plan manifest: create a Dockerfile with Python 3.9, copy `data.csv`, and declare `solve.sh` as an executable.

Materialize: write `data.csv` with three rows, write `solve.sh` that runs `python -c "import pandas as pd; print(pd.read_csv('data.csv').sum())"`.

Retrieve: download a small helper library `pandas` and install it in the image.

Augment: add a fourth dummy row to `data.csv` to increase complexity while keeping the sum unchanged.

Initialize: build the image and run a test command; the container fails because `pandas` is missing.

Repair: install `pandas` via `pip install pandas`; rebuild and re‑run the test, which now succeeds.

Observe $e_0$: the container now contains `data.csv`, `solve.sh`, and the installed `pandas` package.

Generate $I$: “Run `solve.sh` inside the container to compute the sum of the first column of `data.csv`.”

Generate $S$: the script `solve.sh` as written above.

Execute $S$: the script outputs `42`.

Generate $V$: a test that checks the script’s output equals `42`.

Package $\mathcal{T}$: bundle the Dockerfile, `data.csv`, `solve.sh`, and the verifier test.

The environment‑driven pipeline guarantees that the generated instruction and verifier refer to actual files and installed packages, eliminating the mismatch errors that plague purely text‑based generation.

**Algorithm 1** Executable-state-grounded task construction **Require:** Reconstructed specification $Z = (C, R_S, R_I)$ **Ensure:** Validated task bundle $\mathcal{T}$ or failure 1: Plan an environment manifest from $Z$ 2: Materialize the required files, services, dependencies, and assets 3: Retrieve and localize public resources when needed 4: Augment or perturb fixtures while preserving task semantics 5: **repeat** 6: $\quad$ Build and initialize the environment 7: $\quad$ **if** initialization fails **then** 8: $\quad \quad$ Repair the environment from the failure trace and $Z$ 9: $\quad$ **end if** 10: **until** the environment is valid or the repair budget is exhausted 11: **if** the environment remains invalid **then** 12: $\quad$ **return** failure 13: **end if** 14: Observe the realized environment state $e_0$ 15: Generate instruction $I$ from $R_I$ and $e_0$ 16: Generate solution $S$ from $I, R_S$, and $e_0$ 17: Execute $S$ to obtain the resulting state $e_T$ 18: Generate verifier $V$ from $I, R_S, e_0$, and $e_T$ 19: Package the task bundle $\mathcal{T}$ 20: **repeat** 21: $\quad$ Validate $\mathcal{T}$ from a clean initial state 22: $\quad$ **if** validation fails **then** 23: $\quad \quad$ Identify the responsible artifact from the execution trace 24: $\quad \quad$ Repair only the identified artifact 25: $\quad$ **end if** 26: **until** $\mathcal{T}$ is valid or the repair budget is exhausted 27: **if** $\mathcal{T}$ is valid **then** 28: $\quad$ **return** $\mathcal{T}$ 29: **else** 30: $\quad$ **return** failure 31: **end if**

Plan manifest from $Z$ (directories, files, services).

Materialize manifest inside a minimal base image.

Retrieve and localize any public resources.

Augment fixtures while preserving task semantics.

Run initialization checks; if they fail, repair using the failure trace and repeat (max 3 iterations).

Observe realized container state $e_0$.

Generate instruction $I$ from $R_I$ and $e_0$.

Generate solution $S$ from $I$, $R_S$, and $e_0$.

Execute $S$ to obtain resulting state $e_T$.

Generate verifier $V$ from $I$, $R_S$, $e_0$, and $e_T$.

Package task bundle $\mathcal{T}$.

Validate $\mathcal{T}$; if validation fails, identify the offending artifact and repair, then repeat (max 3 iterations).

Return $\mathcal{T}$ if valid; otherwise return failure.

Dataset Comparison

We compare terminal‑agent datasets and explain FACET’s shared realized container state for artifact generation.

Recall that FACET ties instructions, environments, and solutions together by exposing a realized container state that all downstream generators read. This section first situates FACET among existing terminal‑agent datasets and then details how the shared state eliminates cross‑artifact inconsistencies.

TW is the prior baseline dataset used for terminal‑task synthesis, representing a conventional approach that does not share a realized container state across artifacts.

**Table 1.** Trajectory- and task-level comparison of terminal-agent datasets. Trajectories are collected using the Terminus-2 scaffold, and task performance is evaluated using DeepSeek-V4-Pro with Terminus-2. Turns and Tests denote the average interaction turns per trajectory and executable checkpoints per task. Detailed protocols are provided in Appendix A.6.

Experimental Validation

We report FACET’s validation pipeline, repair process, and performance gains across model scales.

Validation packages each candidate as a Harbor bundle containing environment, solution, tests, instruction, and task metadata. The pipeline checks that the environment builds, the verifier fails initially, the solution runs from a clean state, and the final state passes verification.

If validation fails, a constrained router pinpoints the offending component from the execution trace and invokes a targeted repair that preserves all other valid components. Each task receives up to five repair iterations before being discarded.

We fine‑tune Qwen3.5‑4B, Qwen3.5‑9B, and Qwen3.5‑27B on 1.2 K successful trajectories generated by the Terminus‑2 agent (DeepSeek‑V4‑Pro). All models are evaluated on Terminal‑Bench 2.1 with three attempts per task and mean pass rate reported.

Compared to prior terminal‑agent datasets, FACET offers 1.2 K training trajectories (average 11.86 turns) and 6 078 validated tasks with 22.77 tests per task—the highest test density among baselines.

Fine‑tuning on FACET yields up to +8.24 points higher pass rate on Terminal‑Bench 2.1.

Qwen3.5‑9B improves from 27.34 to 35.58, a gain of +8.24 points.

The 27 B model reaches 47.57 on Terminal‑Bench 2.1, only 1.49 points shy of the 49.06 achieved by the much larger 397 B model, despite using a model roughly 15× smaller.

Task‑level analysis reveals that 89.40 % of individual attempts achieve partial correctness, yet full task success remains low because agents must satisfy all conjunctive requirements.

FACET significantly improves task yield and agent success rates.

Task Performance Analysis

Task Analysis quantifies performance gaps and difficulty patterns on Terminal‑Bench 2.1.

Reference models achieve a top score of 78.00 on Terminal‑Bench 2.1, the highest among all evaluated systems.

Table 2 lists GPT‑5.5 (xhigh) with a score of 78.00.

**Table 2.** Results on Terminal-Bench 2.1. Scores under our evaluation setting are averaged over three independent attempts per task.

Check‑level success is low: only 20.94 % of completed rollouts achieve full task success, even though many trajectories make substantial progress.

Among failures, 54.00 % miss just one or two verifier checks, often because a minor field value or secondary deliverable is incorrect, revealing that most errors are near the success boundary.

Generation Scheme Analysis

We evaluate how generation order impacts task validity and final yield.

We compare three artifact‑generation orders—Forward (I→S→V), Reverse (I→V→S), and Joint (all together)—using the same 100 scenario–skill pairs to see how order affects validation success.

Generation schemes dictate the sequence in which the instruction, solution, and verifier are produced, and this ordering determines what information each component can condition on.

Why does generating the verifier before the solution (Reverse) cause more contract‑mismatch failures?

When the verifier is produced without seeing the solution, it can only guess the solution’s behavior from the instruction. This guess often disagrees with the actual solution that is later generated, leading to mismatched expectations about inputs, outputs, and side effects—hence the high contract‑mismatch rate.

Forward outperforms Reverse on a controlled set of 88 pairs, succeeding where Reverse fails significantly more often.

Forward succeeds on 29 pairs while Reverse succeeds on only 9 of the same pairs (p = 0.0017, two‑sided exact sign test).

**Figure 2.** Analysis of execution and synthesis failures. (a) Distribution of failed or errored verifier checks among unsuccessful teacher rollouts; three rollouts without a parsed FAILED/ERROR result are omitted. (b) Distribution of initial validation failure types under the Forward, Reverse, and Joint generation schemes.

Table 7 reports additional statistics for the three generation orders, such as command usage frequencies and turn‑dynamics patterns, confirming that Forward’s more distributed failure profile aligns with higher overall task yield.

Validation and Feasibility

We outline the practical constraints and assumptions underlying the task‑generation pipelines.

The central premise—that coupling instructions, environments, and solutions via a shared realized container state improves reliability—has been introduced earlier. This section enumerates the concrete limits observed when applying that premise.

Repair budgets differ across pipelines: staged generation permits up to five repair rounds, while contract‑first and joint generation allow only three. Under these budgets, staged generation recovers 37 of 53 initial failures for a final yield of 83/100, contract‑first recovers 41 of 69 for 63/100, and joint recovers 29 of 60 for 65/100.

For a fair paired comparison we restrict attention to the 88 semantic paths that achieve validation under all three schemes. Staged generation alone succeeds on 29 paths versus contract‑first’s 9 (two‑sided exact sign test p = 0.0017). Against joint generation, staged succeeds alone on 27 paths while joint succeeds alone on 18 (p = 0.233).

Oracle feasibility—whether the reference solution executes from a clean state—is comparable: 46 of 99 staged tasks and 45 of 96 joint tasks succeed. However, partial‑solution coverage is highly uneven: joint generation covers 95/96 cases, staged only 7/99, and contract‑first 9/91.

Generation‑cost analysis isolates the artifact‑generation stage. Joint generation incurs a single model call with 2.80 minutes of referenced latency per task, whereas staged and contract‑first generation require three calls, averaging 4.14 minutes and 5.19 minutes respectively. These figures exclude shared‑prefix processing, validation, and repair, serving only as proxies for generation cost.

Each generated task follows the Harbor directory structure: task/ containing instruction.md, task.toml, an environment/ folder, solution/, and tests/. Dockerfiles, fixtures, dependencies, and initialization scripts are derived from the reconstructed task specification.

The environment is built and repaired before final artifacts are generated. After a successful image launch, the system records the realized initial state $e_0$, capturing files, directories, schemas, dependencies, and local services. This read‑only $e_0$ is supplied to the instruction, solution, and verifier generators, ensuring a consistent context without mutable sharing.

Verification proceeds in clean containers: first the verifier runs on the untouched $e_0$ (expecting reward 0), then the reference solution executes (expecting reward 1). Failures are classified as instruction, environment, solution, or verifier defects, triggering a full Docker lifecycle repeat for up to five repair rounds.

Verifiers evaluate observable final‑state content rather than exact command sequences, normalizing nondeterministic values such as timestamps, generated identifiers, and irrelevant ordering. Local services are accessed through deterministic interfaces, and verifier failures are localized to guide the corresponding repair agent.

Training and evaluation settings are summarized in Table 9. All supervised fine‑tuning runs use LLaMA‑Factory on eight NVIDIA H200 GPUs, with Qwen3.5 models (4 B, 9 B, 27 B) trained for three epochs, batch size 64, learning rate $1\times10^{-5}$, cosine schedule with 0.1 warmup, and a maximum sequence length of 32 768 tokens. Evaluation employs the Terminus‑2 agent on Terminal‑Bench 2.1, three attempts per task, a two‑hour timeout, temperature 1.0, and the same context length.

**Figure 5.** Command-level patterns in successful teacher trajectories. (a) Command-occurrence frequency. (b) Adjacent-turn transition probabilities.

**Table 9.** Principal training and evaluation configurations.

Source-Skill Taxonomy

Appendix A details skill taxonomy, task accounting, and construction‑funnel statistics.

We organize the $71,341$ retained skills into five top‑level families and $34$ fine‑grained categories. Table 3 reports the category counts and shares computed over the complete corpus without sampling.

**Table 3.** Top-level source-skill distribution.

The final dataset contains $6,078$ validated tasks, of which $6,074$ have rollout records. Four jobs failed to produce a record, and $1,270$ of the completed runs earned reward 1, yielding a teacher success rate of $20.94\%$.

**Table 4.** Fine-grained source-skill categories. “Global” is the share of all 71,341 skills; “within parent” is the share inside the corresponding top-level category.

Task‑level success varies across skill tags, with pass rates ranging from $7.14\%$ to $35.00\%$. Structured‑data tasks tend to succeed more than narrative‑document tasks, while longer instructions and broader verifiers reduce success.

Table 5 summarizes the construction funnel, tracking candidate tasks from initial seeds ($7,852$) through environment repair, task validation, and final retention ($6,078$). Retention percentages are computed relative to the preceding stage.

The table displays the progression of data through various stages, including counts and retention percentages for each stage.

Dataset Comparison Details

This appendix details dataset statistics, command behavior, and compares generation schemes across shared semantic paths.

Table 1 aggregates trajectory‑level and task‑level statistics for several terminal‑agent datasets. The trajectory columns describe agent interactions, while the task columns describe executable task bundles; these are sampled independently, so the numbers reflect dataset‑level characteristics rather than paired measurements.

In the 1,270 parseable teacher trajectories that earned reward 1, the commands cat, python3, and ls account for 69.5 % of occurrences, revealing a compact “observe–act–verify” interaction pattern. Turns often alternate between observation‑only and action‑only, with 53.1 % of action turns followed by an observation turn.

**Figure 3.** Distributions of source skills and synthesized tasks. (a) The retained skill corpus spans five top-level families and 34 fine-grained categories. (b) The 6,078 validated tasks are distributed across nine task families, whose individual shares range from 9.59% to 11.99%.

**Figure 4.** Strict task-level pass rates for the 20 most frequent skill tags among the 6,066 completed rollouts. Error bars indicate Wilson 95% confidence intervals, and the dashed line denotes the overall pass rate. Tags are selected by frequency and ordered by observed pass rate.

The table compares three schemes: Forward (Ours), Reverse, and Joint, across three metrics: Reached validation, Initially valid, and Final yield.

Implementation Details

Details on adapters, pipeline variants, and evaluation results for the reproduced pipelines.

Adapters for the reproduced pipelines preserve original prompts, generation order, and artifact‑construction logic, modifying only the handling of common skill‑pair records, Harbor‑compatible packaging, and connection to shared validation.

Baseline is a stripped‑down version that directly turns each skill pair into a static task blueprint, which then drives environment, instruction, solution, and verifier generation without any scenario reconstruction.

TW reproduces the TerminalWorld workflow, keeping the original prompts and construction logic while adding the necessary Harbor packaging and validation interfaces; it separates environment construction from the remaining artifacts.

FACET first reconstructs an executable scenario from the related skills, propagates the recovered requirements through instruction and solution, then generates artifacts in stages, explicitly building and repairing the environment before validation.

Evaluation reports construction yield, validated task count, and difficulty metrics (P@1, P@3, average terminal commands) over 500 common skill‑pair inputs, distinguishing complete Harbor packages from those that also pass oracle validation.

The table outlines an 8-step process for task materialization, validation, and repair. The columns are "Step", "Stage", and "Operation".

The table compares three pipelines: Baseline, TW, and FACET (Ours) across six metrics: Packages, Validated, Yield, P@1, P@3, and Avg. Cmds.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers