Know Before Fix: QA-Driven Repository Knowledge Acquisition for Software Issue Resolution

Haotian Lin, Silin Chen, Xiaodong Gu, Yuling Shi, Chengxi Pan, Jiaqi Ge, Mengfan Li, Jianghong Huang, Mengchieh Chuang, Beijun Shen, Haibing Guan

ACQUIRE improves software issue resolution by explicitly acquiring repository knowledge via targeted QA before repair.

How can we improve the success rate of LLM-based coding agents by explicitly acquiring repository-specific knowledge through a QA-driven process before attempting a repair?

Coding agents often fail to resolve software issues because they lack deep repository understanding, leading them to rely on shallow keyword-based localization that misses cross-module dependencies and API contracts. ACQUIRE decouples knowledge acquisition from patch generation: it first uses a category-guided Questioner and an autonomous Answerer to generate evidence-grounded QA pairs, then injects this structured knowledge into a Resolver to guide the repair process. On SWE-bench Verified, this approach raises Pass@1 by up to 4.4 percentage points, consistently outperforming existing pre-repair methods while maintaining competitive efficiency.

Paper Primer

The framework hinges on a two-stage design that mirrors expert human debugging: first, decompose the issue into targeted questions across four knowledge dimensions (Mechanism, Design, Structure, and Ecosystem); second, perform autonomous, read-only repository exploration to ground the answers. By injecting these QA pairs as a structured context block before the repair loop, the agent gains a stable, factually reliable understanding of the codebase that steers it toward causally relevant code regions.

ACQUIRE significantly improves resolution success rates across diverse LLM backbones.

Pass@1 on SWE-bench Verified increased by 3.8% for GPT-5-mini and 4.4% for DeepSeek-V3.2 compared to the base repair agent.

The QA-driven approach accelerates the repair trajectory by reducing trial-and-error.

Step-level analysis shows a 23.8% reduction in steps spent locating contracts and a 24.1% reduction in fixing steps, allowing more effort to be dedicated to reproduction and verification.

Why is this QA-driven approach more effective than simply asking the agent to generate a repair proposal directly?

A single-pass proposal forces the agent to simultaneously identify root causes and formulate a fix, which often leads to superficial, entangled reasoning. ACQUIRE’s decomposition isolates specific knowledge gaps, allowing the Answerer to produce evidence-grounded responses that are more factually reliable.

Does the category-guided question template actually improve performance, or is it just a heuristic?

It is essential; ablations show that removing the category-guided template (ACQUIRE-FreeQ) drops Pass@1 by 3.8 percentage points. The categories ensure broader coverage of diagnostic angles and prevent questions from clustering around a single, potentially irrelevant, diagnostic focus.

Introduction

We expose the repository knowledge gap and introduce ACQUIRE to acquire QA-driven context before repair.

Current LLM‑based coding agents excel at generating patches but frequently produce factual errors because they lack a deep understanding of the target repository’s internal structure, dependencies, and implicit contracts.

The gap is the missing, concrete information about a codebase that an agent needs to reason correctly about a bug.

To close this gap we propose ACQUIRE, a QA‑driven framework that first generates targeted question‑answer pairs about the repository and then feeds this structured knowledge into the repair agent.

**Fig. 1.** Agent round distribution $w$/o vs. w/ QA injection under DeepSeek-V3.2.

The core failure mode of current coding agents is insufficient repository context.

The Repository Knowledge Gap

Insufficient repository knowledge hampers code agents, motivating question‑driven knowledge injection.

Recent empirical studies consistently point to an insufficient repository knowledge gap as the primary failure mode of software‑engineering agents, leading to shallow keyword‑based localization, missed cross‑module faults, violated API contracts, and ignorance of external protocols.

Current pre‑repair approaches remain fundamentally fix‑oriented: they either rank suspicious locations or enrich context with structural summaries, yet they never ask what repository knowledge the agent still lacks before attempting a repair.

In contrast, seasoned developers confronting an unfamiliar bug first pose diagnostic questions—e.g., how a component behaves, what API contracts enforce, or where relevant modules reside—building the necessary understanding before committing to any fix.

The oracle experiment is deliberately limited: only one question may be asked and the Answerer can access only the files touched by the golden patch, not the full repository.

Manual inspection of the 116 oracle questions uncovered four recurring patterns—Mechanism & Behavior (70.7 %), Design & Usage (18.1 %), Locating & Structure (7.8 %), and Ecosystem & Standards (3.4 %)—covering the dimensions of repository knowledge needed for repair.

These patterns form a structured question template that enables an automated Questioner to generate targeted queries without privileged information, while the Answerer is replaced by an autonomous agent that discovers relevant files on its own.

The ACQUIRE Framework

We detail ACQUIRE’s two‑stage process that first gathers targeted QA pairs and then uses them to guide code repair.

LLM agents that lack repository‑specific context frequently emit patches that violate hidden invariants or miss required imports.

ACQUIRE first asks the codebase a handful of focused questions, then hands the answers to a repair agent so it can act with full situational awareness.

How does ACQUIRE differ from simply prompting the LLM with the issue description?

ACQUIRE augments the issue description with a curated set of concrete QA pairs that expose hidden repository facts; a plain prompt lacks this structured, evidence‑grounded context.

Answerer 1 searches the code tree, finds

Answerer 2 inspects

The two pairs (q₁,a₁) and (q₂,a₂) are assembled into K = {(q₁,a₁),(q₂,a₂)}.

K is serialized and prepended to the Resolver’s prompt before any edit is attempted.

Even with only two questions, the Resolver gains precise locations and external constraints that would otherwise be guessed.

**Fig. 1.** Overview of ACQUIRE.

Stage 1 therefore supplies the repair model with a compact, evidence‑backed snapshot of the repository before any code is touched.

Before the Resolver starts editing, the entire set of (question, answer) pairs is inserted as a single text block, guaranteeing that every subsequent decision can reference the same factual grounding.

Why does the Resolver receive the QA set as a static pre‑injection rather than querying on‑the‑fly?

Static injection preserves modularity (the two stages remain decoupled) and guarantees that the repair loop always has the full knowledge context, eliminating early‑stage decisions made under partial information.

Questioner receives the issue description I and generates N templated queries, each assigned to one of the four knowledge categories.

Each query qᵢ is dispatched to an independent Answerer instance operating in read‑only mode.

Answerer i explores the repository, gathers concrete evidence (file paths, function signatures, external specs), and returns answer aᵢ.

If an Answerer cannot locate sufficient evidence, it returns an explicit “unknown” response.

All (qᵢ, aᵢ) pairs are collected into the knowledge set $K$ and handed off to Stage 2.

Resolver loop – knowledge‑informed repair

Stage 2’s Resolver now navigates, edits, and validates code while constantly consulting the static QA block, which narrows the search space and informs design‑level decisions.

Experimental Setup

Defines the evaluation protocol, datasets, metrics, baselines, and implementation details.

We evaluate ACQUIRE by answering four research questions that probe its effectiveness, the quality of generated QA knowledge, the contribution of its components, and the trade‑off between QA quantity and cost.

A curated subset of real GitHub issues that isolates functional bugs and provides deterministic unit‑test evaluation.

A stripped‑down repair agent that repeatedly issues shell‑style actions, observes execution, and updates its plan until a passing patch is produced.

Research Questions: RQ1 asks how effective ACQUIRE is at fixing issues; RQ2 probes the influence of the generated QA knowledge on repair behavior; RQ3 isolates the impact of question decomposition and category‑guided generation via ablation; RQ4 studies how the number of QA pairs trades off performance against cost.

We evaluate on SWE‑bench Verified and use two contrasting backbone LLMs: DeepSeek‑V3.2, an open‑source Mixture‑of‑Experts model with 671 B total parameters (≈37 B active per token); and GPT‑5‑mini, a proprietary, high‑throughput model optimized for low latency.

Effectiveness is measured by Pass@1 (the fraction of issues solved on the first attempt). Economic impact is captured by Average Cost (dollar cost per instance, including pre‑repair and repair stages) and Average Time (wall‑clock seconds from start of pre‑repair to patch generation).

Baselines include Mini‑SWE‑Agent (standalone and shared repair backbone), LocAgent (heterogeneous repository graph traversal), CoSIL (incremental local call‑graph expansion), LingmaAgent (knowledge‑graph + Monte Carlo Tree Search), and SWE‑Debate (multi‑agent debate for repair planning).

All baselines adopt Mini‑SWE‑Agent’s repair settings: Docker isolation, 360 s step timeout, 250‑step max trajectory, \$3.00 per-instance cost cap, and temperature 0.0. ACQUIRE generates two questions per issue (temperature 0.7 for diversity), uses the same execution sandbox, limits answer steps to 150 with a \$2.00 cost cap, and inherits Mini‑SWE‑Agent’s resolver configuration.

Performance and Impact

ACQUIRE delivers top Pass@1 scores with modest cost, confirming QA‑driven knowledge boosts repair efficiency.

ACQUIRE attains the highest Pass@1 scores across both backbone models.

Table I shows ACQUIRE reaching 62.2 % on GPT‑5‑mini and 70.8 % on DeepSeek‑V3.2, surpassing all baselines.

**Table I.** Main results on SWE-bench Verified under GPT-5-mini and DeepSeek-V3.2.

**Table II.** API-call shift between Mini-SWE-Agent and ACQUIRE on SWE-bench Verified under DeepSeek-V3.2.

Beyond raw accuracy, we evaluate the factual reliability of the generated QA pairs and their impact on downstream repair behavior.

99.1 % of ACQUIRE’s generated QA pairs are supported by repository evidence.

Human audit of 232 QA pairs (230 supported, 2 ungrounded) conducted by four master’s‑level reviewers.

Failure Analysis

ACQUIRE’s QA injection boosts repair success and ablations confirm each design choice’s impact.

ACQUIRE attains 70.8% Pass@1, surpassing the Proposal and FreeQ ablations by 4.8 and 3.8 percentage points respectively.

Table II shows Pass@1 of 70.8 for ACQUIRE, 66.0 for ACQUIRE‑Proposal, and 67.0 for ACQUIRE‑FreeQ.

Among 22 observed regressions, only five are misleading—where injected QA steered the Resolver away from the correct fix—while the remaining seventeen stem from Resolver‑side issues despite reliable QA.

Fact‑checking shows that nine of the ten QA pairs in the misleading cases are grounded in repository evidence, indicating that errors arise from plausible but incorrect framing rather than factual inaccuracies.

**Fig. 2.** Trajectory stage composition on 44 Fail$arrow$Pass instances w/o vs. w/ QA Injection. Darker shading in the right ring marks QA-related steps.

Ablations and Case Study

We ablate ACQUIRE components and examine how QA‑pair count affects repair.

We evaluate each ACQUIRE component via ablations and study how the number of QA pairs influences repair performance and cost.

Replacing decomposition with a single‑pass proposal harms repair.

Pass@1 falls to 66.0 %, below Mini‑SWE‑Agent’s 66.4 %.

Dropping the category‑driven template reduces performance.

Pass@1 drops to 67.0 %.

Category‑guided questions score higher on diagnostic utility (+0.38), reasoning depth (+0.38), and coverage (+0.78) while answerability remains unchanged.

Adding a single QA pair ( $N$ = 1) raises Pass@1 by 2.6 pp to 69.0 %.

Cost increases by only \$0.005 per instance.

Performance peaks at $N$ = 2 with Pass@1 70.8 % and then drops at $N$ = 3, while average cost keeps rising.

**Fig. 3.** Pass@1 and average cost per instance under varying the number of QA pairs.

**Fig. 4.** Case Study of ACQUIRE with instance sphinx-doc__sphinx-9230.

In the sphinx‑doc__sphinx‑9230 case, ACQUIRE’s Questioner asks about the “:param” type‑parsing logic, steering the Answerer to the true culprit in `docfields.py`. The Resolver then patches the split with a bracket‑aware replacement, completing the repair in 52 steps.

ACQUIRE solves the bug in 52 steps, a 55 % reduction from Mini‑SWE‑Agent’s 117 steps.

The focused QA pair enables precise fault localisation and a minimal, correct patch.

**Table II.** Ablation results on downstream repair performance on SWE-bench Verified under DeepSeek-V3.2.

Discussion and Limitations

We examine ACQUIRE’s practical impact, its limitations, and avenues for future improvement.

ACQUIRE demonstrates that explicitly acquiring repository knowledge before repair is both practical and effective for repository‑level issue resolution.

By separating question generation, repository‑grounded answering, and patch synthesis, ACQUIRE converts implicit knowledge gaps into explicit QA pairs that the agent consumes before editing.

Empirically, ACQUIRE yields consistent gains across backbone models with only modest additional cost, improving Pass@1 and reducing blind locating effort.

The four‑category question template works well, yet broader coverage could be achieved via hierarchical or learned category schemas.

Currently, knowledge is injected as pre‑generated QA context; a promising extension is dynamic acquisition that refreshes knowledge during the repair trajectory, though this adds computational cost.

We acknowledge three validity threats: construct validity (metrics may miss aspects of repair quality), internal validity (possible confounds like prompt design), and evaluation bias from LLM‑as‑judge scoring.

To mitigate evaluation bias, we repeat each LLM judgment ten times, use both scoring and pairwise voting, and apply position‑bias correction as in prior work.

Prompt Templates

Defines the prompt templates used to generate repository questions and evaluate them.

The Questioner Prompt defines a system prompt that asks the model to produce exactly N repository‑focused questions, grouped into four predefined categories.

JSON skeleton returned by the Questioner Prompt.

The Answerer Instance Template wraps a user‑provided question in XML tags and prescribes a THOUGHT section followed by a single bash command.

Skeleton of the Answerer Instance Template.

The evaluation prompts describe a five‑dimensional scoring rubric (relevance, answerability, utility, depth, clarity) and the exact JSON output the evaluator must emit.

JSON schema for a single question evaluation.

Prompt Evaluation Criteria

Evaluating diagnostic question sets for repository‑focused debugging.

The evaluator judges each diagnostic question using three hard flags: `requires_external_knowledge`, `too_generic`, and `rephrases_issue_without_added_value`. Repository Answerability measures where the answer originates, while Diagnostic Utility measures how much the answer reduces debugging uncertainty. A question that can be answered without opening the repository should not receive a high Repository Answerability score, and a question that merely suggests a starting point without reducing uncertainty should not receive a high Diagnostic Utility score.

Coverage is the sole scoring dimension, rated from 1 to 10. It reflects core relevance, complementarity, non‑redundancy, and overall debugging usefulness, with anchors ranging from very poor (1‑2) to excellent (9‑10). Evaluators consider whether the set collectively targets the main failure mechanism, probes distinct diagnostic angles, avoids repetition, and would substantially narrow the search space.

Calibration examples illustrate good and bad question sets: a strong set pinpoints the failing file or subsystem and then asks how the relevant control flow works, while a weak set repeats location‑seeking questions. The evaluator returns a JSON object with a numeric coverage score, a brief summary, and a rationale tied to relevance, complementarity, redundancy, and debugging value. For voting between two sets, the output JSON indicates the winner, confidence, and a concise justification, or TIE if benefits are equal.

Supplementary Analysis

Detailed breakdown of ACQUIRE’s runtime, cost, and QA impact.

Section A dissects the total wall‑clock time and monetary cost reported in Table I into a pre‑repair exploration phase and a downstream repair phase for each method.

Across all models, ACQUIRE’s pre‑repair stage runs in 187 s and costs 0.286 $, which is 2.5–11× faster and 5–26× cheaper than the comparable stages of LingmaAgent and SWE‑Debate, while still delivering the highest Pass@1 score.

Section B examines how QA injection reshapes the agent’s interaction pattern.

In B.1 the mean number of agent rounds drops by 7.1 % on the full 500‑instance set and by 17.1 % on the 44 Fail→Pass cases, confirming that front‑loading repository knowledge curtails costly trial‑and‑error loops.

B.2 reports the API‑call shift ($\Delta$ = `API_ACQUIRE` − `API_Mini`‑SWE‑Agent) shown in Table II: Fail→Pass instances see a mean reduction of 15.18 calls, whereas the overall Pass→Fail increase is modest (+0.55) and is driven almost entirely by the misleading subset (+11.40), while non‑misleading Pass→Fail cases actually reduce calls (‑3.64).

Section C characterizes the content of the generated QA pairs; Table III shows that 74.4 % belong to the “Mechanism & Behavior” category, reflecting the dominance of internal logic and data‑flow questions.

Nevertheless, 20 of the 44 Fail→Pass instances (45.5 %) also require at least one QA from a non‑Mechanism category—design, locating, or ecosystem—indicating that cross‑category knowledge is essential for many successful repairs, as also observed in the category‑guided ablation.

Human Audit Details

Human audit details and regression analysis of QA injection.

We first break down the 132 Supported QA pairs with minor deviations into three error categories. Localized reference or detail inaccuracies dominate at 67.9%, where the overall answer is correct but specific code references, examples, or line ranges are imprecise. Evidence‑scope overreach accounts for 24.6% and involves extrapolating beyond verifiable repository evidence, while implementation boundary overgeneralization makes up the remaining 7.5%.

The audit also identified two QA pairs labeled Contains ungrounded claim that still offered useful repository context but featured incorrect central mechanism claims. In django__django-12663_q0 the answer mistakenly asserted that SimpleLazyObject inherits from Promise, whereas the code places it under the LazyObject hierarchy. In sympy__sympy-23534_q0 the answer incorrectly claimed that the cls argument propagates through nested tuple recursion, while the implementation simply recurses without forwarding cls.

We manually inspected all 22 Pass→Fail regression trajectories from Section 5.2 to separate failures caused by misleading QA from those where QA was not the primary cause. Five of the 22 cases are classified as misleading, meaning the injected QA steered the Resolver toward an incorrect repair direction; the remaining 17 are non‑misleading, where QA generally aided the Resolver but other factors caused the final failure.

Table IV summarizes the repair‑behavior breakdown of the five misleading cases under DeepSeek‑V3.2. Two cases involve a wrong modification location, where the QA highlights a related but non‑gold file and the Resolver continues editing there. Two cases exhibit wrong logic repair, with QA‑provided mechanisms conflicting with the gold fix. One case suffers overbroad edits, as the QA suggests an overly wide set of files or methods, prompting unnecessary changes.

This analysis confirms a limitation of ACQUIRE: injected QA can occasionally misguide the Resolver, accounting for 5 of the 22 inspected regressions. Nevertheless, the overall effect is positive, yielding a net gain of +22 instances, which shows that the risk is bounded and outweighed by the benefit of providing repository‑grounded context.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers