Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction

Tencent WorkBuddy Bench Team, Siqi Cai, Shaopeng Chen, Xiang Fei, Yong Mao, Zihan Xu, Zhiheng Lyu, Zhijian Shao, Yuchen Shi, Shuwen Zhang, Chaofan Qiu, Linjie Che, Xiaoxi Zhao, Feng Wu, Kai Zhang, Chaofan Zhu, Yubin Qi, Xiaoyun Liang, Peijie Dong, Yunhao Zhang, Yuanjie Zhu, Ling Jiang, Xianjun Zhang, Zhehang Chu, Anyuan Sang, Zhen Feng, Sen Nie, Shi Wu, Yuanzhen Xu, Xin Li, Ning Yang, Zhiqiang Dong, Hande Dong, Qiang Lin, Yi Liu, Yunsheng Wu, Ke Li, Xing Sun

A multi-domain coding-agent benchmark using contamination-resistant task construction and open-source auditing.

How can we build a multi-domain coding agent benchmark that resists data contamination while accurately measuring performance across code, web, office, and security tasks?

Public coding benchmarks are increasingly vulnerable to contamination, as models memorize the specific issue threads and pull requests that populate static test sets. Tencent WorkBuddy Bench addresses this by reverse-engineering tasks from real-world commits and business scenarios, then rewriting them as colloquial, role-played requests that withhold the solution context. The suite provides a fully open, reproducible evaluation harness across four domains—Code, Web, Office, and Security—allowing third parties to audit every task and environment directly.

Paper Primer

The benchmark replaces static, web-searchable problem statements with a distribution-informed construction protocol. By matching task difficulty and intent to real-world usage taxonomies rather than copying raw user data, the suite ensures that agents must perform genuine repository-level reasoning and requirement disambiguation to succeed.

The core mechanism is a unified task-directory format: each task is a self-contained Docker environment that isolates the agent from grading assets until the episode concludes. This "hidden-at-solve-time" design, combined with dataset versioning, forces agents to solve the task from scratch rather than relying on pre-indexed solutions.

The benchmark achieves contamination resistance by construction rather than secrecy.

Tasks are reverse-engineered from real commits and business scenarios, then rewritten as colloquial, role-played requests that omit the root cause and reference diffs. The suite covers 260 total tasks across four domains (80 Code, 70 Web, 50 Office, 60 Security) with no suite-wide average, as scoring instruments are domain-specific.

Why does the suite report no aggregate score across the four domains?

The domains use fundamentally different scoring instruments—ranging from hidden unit tests in Code to deterministic scorers in Security and LLM-judge rubrics in Web and Office—making a single cross-domain average statistically meaningless.

How does this benchmark differ from vendor-production suites like CursorBench?

While vendor benchmarks are closed and opaque, Tencent WorkBuddy Bench is fully open; all task directories, environment images, evaluation harnesses, and reference solutions are public, allowing any third party to reproduce and audit the results.

Introduction to WorkBuddy Bench

We present Tencent WorkBuddy Bench, a contamination‑resistant, multi‑domain agent benchmark.

Public agent benchmarks either expose full issue text (e.g., SWE‑bench) or remain closed (e.g., CursorBench), making scores vulnerable to memorization or opaque selection bias. A realistic evaluation must resist web‑searchable prompts while remaining fully auditable.

TWB unifies four work domains—Code, Web, Office, Security—into a single, open‑source suite whose tasks are reverse‑engineered from real engineering artifacts and rewritten as short, colloquial, role‑played requests, guaranteeing that prompts cannot be recovered by a web search.

We make four concrete contributions: (1) the TWB suite itself, (2) a distribution‑informed, contamination‑resistant construction pipeline, (3) an evaluation methodology that combines baseline/oracle admission gates with domain‑specific scoring, and (4) a cross‑model leaderboard evaluated under both CodeBuddy Code and Claude Code harnesses.

**Figure 1.** Tencent WorkBuddy Bench at a glance. Real commits, pull requests, office workflows, and security cases are reverse-engineered into colloquial, role-played requests, with task distributions matched to real usage (left); the four tracks – Code, Web, Office, and Security – share one open task-directory format (center); every task is scored in an isolated sandbox under two agent harnesses (right).

The key shift is from chat‑based, prompt‑only evaluation toward agentic, artifact‑based benchmarks that test end‑to‑end workspace creation and are provably resistant to prompt contamination.

Contamination-Resistant Task Construction

How the benchmark tasks are built, packaged, and kept contamination‑resistant.

The benchmark constructs each task through a uniform pipeline while allowing subset‑specific scoring, and it deliberately excludes any raw user data.

Source a concrete origin – either an upstream commit/PR, a CVE, or a business scenario.

Rewrite the origin into a short, colloquial, underspecified natural‑language request; for Code, select one of five requester personas, for Security assign a professional role.

Assemble the agent‑visible workspace (code repository, data fixtures, or document set) and place it under a Docker‑based environment.

Isolate evaluation assets (tests, rubrics, scoring scripts) in a separate

Package everything into a self‑contained directory following the Harbor‑style layout.

Tasks are built from real artifacts but rewritten so that the instruction text cannot be recovered by a simple web search, and the released dataset is versioned to limit long‑term exposure.

How does “contamination‑resistant” differ from simply “no user data”?

“No user data” only guarantees that private prompts are absent; contamination‑resistance also guarantees that the exact wording of the task cannot be found in any public source because the request is freshly rewritten from a real artifact, eliminating the searchable‑prompt leakage vector.

**Table 2.** Open release: the dataset ships every component needed to run, grade, and audit a task.

Because the entire suite (tasks, environments, and grading assets) is released openly, any third party can rerun a task, inspect its workspace, and verify the evaluation logic.

The Code Subset

Defines the Code subset, its provenance, difficulty, and evaluation pipeline.

The Code subset treats an agent like a junior engineer who must read a real repository, locate the right file, and apply a change based solely on a role‑played request.

How does the Code Subset differ from traditional bug‑fix benchmarks?

Traditional benchmarks give a pre‑localized bug file and often a starter patch. The Code Subset drops the agent into a full repository at a baseline commit, requires it to discover the relevant module, and evaluates the change against hidden tests, so success depends on navigation, context extraction, and synthesis, not just fixing a known line.

Define a task directory containing instruction.md, task.toml, a Docker image of the target repository, and a tests/ folder with hidden unit tests.

Run the oracle‑gated admission: first execute the verifier on the untouched baseline; reject if reward > 0.3.

Apply the diagnostic gold patch; run the verifier again and require oracle reward = 1.0.

Expose the task to the agent; after the agent produces a patch, rebuild the Docker image and run the verifier on the patched workspace.

Record the per‑run hidden‑test score; aggregate across runs to obtain the headline Code metric.

First verifier run on the untouched repository returns 0.25 ≤ 0.3, so the task passes the baseline gate.

Apply the gold patch; second verifier run returns 1.0, satisfying the oracle gate.

The task is admitted; during evaluation the agent’s patch receives a hidden‑test score of 0.68, which contributes to the final Code metric.

This concrete trial shows that both a low baseline reward and a perfect gold‑patch reward are required for admission, preventing trivial or unsolvable tasks from entering the benchmark.

**Table 3.** Code subset provenance families. Counts sum to the 80-task release: A = 34, B + C = 46.

**Figure 2.** Code and Web task composition. (a) Six Code usage domains merged from 18 fine-grained categories; bug fixing accounts for 10 of 80 tasks. (b) Seven Web task categories. (c) Six Web lifecycle modes; From Scratch accounts for 35 of 70 tasks.

**Figure 3.** Code task and evaluation workflow. The coding agent reads a natural-language request, explores the repository, and emits a patch (left); the patch is graded by hidden unit tests and, as a diagnostic reference, by a rubric-weighted LLM judge (center). The headline Code metric is the hidden-test score; the LLM-judge reading and its blend are reported as reference values only and never enter the headline metric (right).

The Web Subset

Defines the Web tasks, their categories, and evaluation workflow.

The Web subset evaluates whether agents can produce a functional front‑end artifact rather than merely plausible HTML.

The Web subset measures an agent’s ability to deliver a runnable, checkable front‑end artifact that satisfies a concrete output‑path contract.

Enforce the artifact‑not‑chat contract: the agent must write a runnable artifact to a declared output path.

Divide the 70 tasks into seven categories, recording counts for each (e.g., 21 page‑interaction, 15 data‑visualization).

Assign half the tasks as “From Scratch” generation; allocate the remaining tasks to bug‑fix, feature‑extension, review, test generation, and format conversion according to their counts.

Label each task with an interaction‑complexity axis (static, single‑flow, persistence, multi‑step, light interaction) and record the distribution (25 static, 45 interactive).

Run each task in a sandbox without internet or external credentials, then evaluate using the three‑stage rubric (rule checks → LLM/VLM judges → agent‑judge).

Aggregate rubric scores via the checklist formula shown in Figure 4 to obtain the final metric.

**Figure 4.** Web task and evaluation workflow, from query and agent rollout to the delivered artifact (left), through rule, LLM/VLM, and agent judges over extracted evidence (center), to rubric-item checklist scoring (right), combining deterministic checks, semantic and visual judgment, and live interaction over the delivered artifact.

The Office Subset

The Office subset defines 50 mixed‑format office tasks and their evaluation pipeline.

The Office subset probes an agent’s ability to handle realistic office workflows that span spreadsheets, documents, JSON, and file‑system structures.

A curated suite of 50 office‑automation tasks that require an agent to read, modify, and keep consistent a heterogeneous collection of files in a local workspace.

How does the Office Subset differ from the Code or Web subsets?

Code and Web tasks focus on single‑file code execution or web‑page retrieval, respectively. The Office subset forces an agent to orchestrate multiple heterogeneous files, maintain consistency across them, and produce structured artifacts that are evaluated by inspecting the whole workspace rather than a single output string.

Read the ward table (2 rows) and admission log (3 rows) into memory.

Join on the ward identifier, count admissions per ward, and divide by the total bed count from the policy table.

Write the “detail” sheet with the per‑ward utilization numbers and the “summary” sheet with the overall average.

Verify that the sum of the detail sheet matches the summary sheet value.

This trial shows that success hinges on correct cross‑file key matching and on preserving consistency between the detail and summary sheets—simply outputting a plausible percentage would not pass evaluation.

Identify a target capability or real‑world workflow to benchmark.

Build the agent‑visible workspace (input files) and the hidden evaluation assets (reference answers, rule‑check specifications, semantic rubrics).

Run the evaluator on saved submissions to ensure it can detect both correct and incorrect behaviours.

Calibrate task difficulty by measuring solution‑path complexity and adjusting the tier label.

Perform release checks: verify that every task uses both deterministic rule checks and the evidence‑grounded LLM Judge, and that all assets are correctly linked.

**Figure 5.** Composition of the 50-task Office release by construction route and calibrated difficulty.

**Figure 6.** Office coverage across task type, diagnostic scenario, output family, and evaluation mechanism, arranged as a single row of four bar charts. Output families and mechanisms use multi-label counts; scenario groups describe benchmark coverage and are not estimates of production request traffic.

**Figure 7.** Office task, evaluation, and scoring flow. The agent acts on the workspace and leaves a final state (left); deterministic rule checks evaluate the verifiable workspace state while the LLM Judge evaluates only fixed post-task evidence (center); each task combines the two scores with its own weight, trials are averaged within each task, and task scores are macro-averaged with equal weight (right).

Together, these components make the Office subset a comprehensive probe of an agent’s ability to orchestrate multi‑file workflows, maintain consistency, and satisfy both exact and semantic evaluation criteria.

The Security Subset

Describes the security‑team tasks, their evaluation pipeline, and anti‑cheat safeguards.

The Security Subset targets the full spectrum of security‑team work, from red‑team exploitation to blue‑team triage, using deterministic scoring without any LLM judge.

The Security Subset is a collection of realistic, contamination‑resistant tasks that emulate the end‑to‑end workflow of security researchers and analysts.

For each task, the deterministic scorer executes inside an isolated Docker container.

The task is run three times; the three trial scores are recorded.

Compute the mean of the three trial scores to obtain the task’s base score $S_i$.

Calculate rule‑check pass rate $R = \text{passed rule checks} / N$ and evidence pass rate $J = \text{passed rubrics} / K$.

Weight $S_i$ by $R$ and $J$, then average the weighted scores over all 60 tasks to produce the overall benchmark score.

Compute the mean: $(0.8 + 0.7 + 0.9) / 3 = 0.8$ — this is the base score $S_i$ for the task.

If the rule‑check pass rate $R$ is 0.9 and evidence pass rate $J$ is 0.95, the weighted score becomes $0.8 \times 0.9 \times 0.95 = 0.684$.

When aggregated with the other 59 tasks, this contribution influences the final benchmark average.

This example shows how deterministic scoring, multiple runs, and pass‑rate weighting combine to produce a robust, repeatable metric.

**Figure 8.** WorkBuddy Bench Security overview. Tasks are built from real, historical vulnerabilities and authored scenarios into reproducible, self-contained cases (left); they span six red- and blue-team task types – whitebox source audit, blackbox binary exploitation, web exploitation, agent security, malware analysis, and security operations – across 38 red-team and 22 blue-team tasks (center); and each is scored by a per-task deterministic program inside an isolated Docker container that emits a numeric reward (right).

The Evaluation Harness

Describes the open, sandboxed harness that runs each task and computes track scores.

The evaluation harness treats task execution as a first‑class contribution: every task runs in an isolated container, the model is accessed via a configurable proxy, and all scoring logic is public.

It is a public, reproducible pipeline that isolates the agent, connects it to a model backend, runs the task, and then applies a track‑specific scoring rule.

How does this harness differ from a typical benchmark that simply calls a model API?

Typical benchmarks embed the model call inside the task script, so hidden tests can be inadvertently exposed. Here the model is contacted only through a proxy after the agent has acted, and the sandbox isolates the agent from any verification assets, guaranteeing that the score reflects only the agent’s behavior.

Package each task as a directory containing a Docker image, evaluation script, and gold solution.

Launch the task inside a sandboxed container; the agent receives only the declared workspace.

Connect the agent to the model backend via a proxy that logs requests and rewrites model names.

After the agent finishes, invoke the task‑specific verifier to compute $r_t$.

Aggregate per‑task rewards across the track using the mean formula $S = \frac{1}{|T|}\sum_{t\in T} r_t$.

Record the model’s sampling hyperparameters (temperature, top‑p, reasoning toggle) in a central schema.

Publish the full task directory, container image, and scoring code alongside the leaderboard.

Trial 1 passes $P_{m,i,1}=4$ rule checks → $R_{m,i,1}=4/5=0.8$.

Judge outcomes for the three rubrics are $b_{m,i,1,1}=1$, $b_{m,i,1,2}=0$, $b_{m,i,1,3}=1$ → $J_{m,i,1}=(1+0+1)/3=0.667$.

Combined trial score $S_{m,i,1}=0.8 \cdot 0.8 + 0.2 \cdot 0.667 = 0.746$.

Trial 2 passes $P_{m,i,2}=5$ rule checks → $R_{m,i,2}=1.0$.

Judge outcomes are $b_{m,i,2,1}=1$, $b_{m,i,2,2}=1$, $b_{m,i,2,3}=1$ → $J_{m,i,2}=1$.

Combined trial score $S_{m,i,2}=0.8 \cdot 1.0 + 0.2 \cdot 1 = 1.0$.

Task‑level average $\bar{S}_{m,i}=(0.746+1.0)/2=0.873$.

This concrete averaging shows how the high rule weight $w_i$ can dominate the final score, while a single judge failure only modestly reduces the trial score.

Benchmark Results and Leaderboard

We present the benchmark leaderboard and analyze how model rankings vary across domains and harnesses.

The Tencent WorkBuddy Bench evaluates agentic capability across four realistic work domains and two distinct evaluation harnesses, answering the core question of how agents rank under varied conditions.

A leaderboard orders models by their average scores on each domain‑harness cell, making it easy to spot which agent excels where.

How does this leaderboard differ from a typical ML benchmark leaderboard?

Standard benchmarks usually report a single aggregate score per model, whereas this leaderboard reports per‑domain and per‑harness scores, exposing how evaluation conditions (the harness) can reshape the ranking.

Model rankings shift substantially between the two harnesses, with the Security track showing the largest reordering (mean absolute shift 8.6 points).

Table 6 shows that, for Security, the top model changes from GLM‑5.2 (cbc) to GLM‑5.2 (cc) while several other models move up to five points, and the overall mean absolute shift across the seven models is 8.6 points.

**Table 6.** Tencent WorkBuddy Bench leaderboard.

**Figure 9.** Office results by difficulty and task type. (a) Mean Office score by calibrated difficulty within each harness, averaged over all models with scored runs in that panel. (b) Equal-weight task average for the seven diagnostic task types represented by at least four tasks, shown for the three models discussed in the text. Scores use a 0–100 scale. Bold marks the best score in the full model set for each task type and harness, so a displayed three-model group may contain no bold value. HY-3’s per-task slices in this figure predate its updated Office run, so its contribution to the panel means reflects the earlier scoring.

**Table 7.** Per-run averages on the Code subset, by harness: turns (unique assistant messages, including subagent activity), output tokens, and cache-inclusive input tokens, in thousands. Output tokens are comparable across harnesses; input tokens are not, because the two harnesses manage context and caching under different conventions. ‡Claude Opus 4.8's Claude Code figures come from the modified-instruction run described in Table 6.

Performance variance across domains is substantial: some models dominate specific tasks while others lag, and the ranking can change dramatically when the evaluation harness changes.

Related Work

We compare WorkBuddy Bench to prior agent benchmarks across code, web, office, and security domains.

We first situate Tencent WorkBuddy Bench (TWB) among existing agent benchmarks in the code, web, office, and security domains. The suite’s own task‑construction methodology, subset design, and evaluation harness are described in later sections, so here we only provide design‑time comparisons.

The Code subset occupies a similar problem space to the SWE‑bench family and to library‑from‑scratch benchmarks such as Commit0, yet it differs in instruction style and role diversity. SWE‑bench supplies detailed GitHub issues, while Commit0 uses test‑driven specifications; WorkBuddy’s Code tasks are short, colloquial requests that leave implementation details underspecified, and they span five requester roles across 18 categories.

Contamination resistance is handled differently across the family. LiveCodeBench relies on problems released after model‑training cutoffs, whereas WorkBuddy’s Code subset uses freshly authored task directories—real upstream commits, clean‑room reimplementations, and synthetic workspaces—kept hidden until benchmark release.

End‑to‑end and production coding‑agent benchmarks provide useful reference points. Vibe Code Bench evaluates zero‑to‑one web‑application development from text specifications, while CursorBench traces committed code back to authentic production requests. Vibe focuses on from‑scratch construction; WorkBuddy Web additionally covers modification, review, front‑end testing, analysis, and conversion. CursorBench is closed‑source, so its task set and selection bias cannot be independently audited.

In the Web domain, Design2Code and Interaction2Code assess static and lightly interactive page reproduction, FrontendBench broadens front‑end generation tasks, and WebArena plus VisualWebArena evaluate agents operating within a browser environment. Each excels on one or two axes—static reproduction, interactive generation, or browser‑agent operation—but none combine page/UI work, data and chart artifacts, project documents, tests, analyses, non‑scratch lifecycle coverage, runtime checks, and rule/LLM/VLM/agent judging in a single suite.

**Table 8.** Web benchmark capability matrix. • = full coverage, ◦ = partial coverage, blank = not covered or not applicable. Axes are task-design and verification coverage, not a head-to-head measured evaluation of agent performance; model-judge bias for VLM and agent-judge layers is discussed separately in Section 7.

Office benchmarks address complementary aspects of office‑agent work. Workspace‑Bench 1.0 evaluates heterogeneous file dependencies, ClawsBench simulates Gmail, Calendar, Docs, Drive, and Slack interactions, OdysseyBench targets long‑horizon multi‑application workflows, and SpreadsheetBench 2 focuses on complex workbook construction and visualization.

WorkBuddyBench‑Office concentrates on complete handoffs in local workspaces containing multiple file formats. Agents must preserve source information, keep related files consistent, and respect execution constraints, while deterministic rule checks and evidence‑grounded LLM judges score binary semantic rubrics from fixed post‑task evidence.

Security benchmarks each cover a slice of the red‑team side. Cy‑bench and NYU CTF Bench score professional‑level and competition challenges; InterCode‑CTF casts CTF solving as interactive coding; CVE‑Bench measures autonomous exploitation of real‑world web‑application CVEs; Meta’s CyberSecEval series assesses model‑generated insecure code and offensive assistance.

The Security subset differs on coverage and scoring. Its 60 tasks span red‑ and blue‑team work—vulnerability discovery, safe exploitation, malware analysis, and security operations—scored by deterministic per‑task scripts behind five anti‑cheat layers, with no LLM judge involved.

Overall, WorkBuddy Bench adds breadth and framing rather than new task types. It pairs repository‑scale code tasks with diverse requester roles, enforces contamination resistance through fresh, closed‑room task directories, unifies web benchmarks across multiple axes, treats office workflows as complete handoffs with deterministic rule checks, and provides a comprehensive security suite with deterministic scoring. All artifacts—task directories, environment images, evaluation code, grading tests, and reference solutions—are released openly for independent audit.

Limitations and Future Work

We outline the benchmark’s current limitations, near‑term work, and list the contributors.

One leaderboard cell uses a modified instruction setup: Claude Opus 4.8’s Code score was run with a “do‑not‑ask, complete‑in‑one‑pass” prompt, differing from the other runs and thus not folded into the Code harness‑shift aggregate.

The Code subset’s open release is dominated by Python tasks; only a few tasks port behavior from JavaScript, TypeScript, or Rust, so findings should not be assumed to generalize to other languages.

Because the benchmark is fully open—task directories, environment images, evaluation code, and reference solutions are public—published task content can be scraped into future model training data, eroding contamination resistance over time.

Judge‑based components (Web, Office, Code) combine rule‑based checks with LLM/VLM judges, introducing model‑judge bias that may favor familiar response styles independent of task correctness.

Scores are tied to the specific serving endpoint (e.g., HY) and to the particular harness builds used; metric values may shift as endpoints or harness versions evolve.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers