PACE: Towards Surfacing Hidden Conflicts in User Requests

Yoojin Kim, Jihyoung Jang, Hyounghun Kim

PACEMAKER is a multi-agent framework that surfaces latent situational conflicts in user requests by traversing egocentric knowledge graphs.

How can we evaluate and improve an LLM's ability to detect when a user's request conflicts with their own personal context (e.g., health, schedule, or ethical constraints)?

Personalized assistants often fail to identify when a seemingly reasonable request—like booking a restaurant—conflicts with hidden personal constraints, such as a friend's allergy or a prior commitment. PACEMAKER addresses this by reformulating queries to target latent conflict signals, traversing a k-nearest neighbor graph to collect distributed evidence, and filtering out irrelevant distractors. On the PACE benchmark, this approach consistently outperforms standard retrieval methods, significantly improving the assistant's ability to correctly identify and justify conflict-based refusals.

Paper Primer

The core challenge is that conflict-relevant facts are often scattered across thousands of atomic entries in a user's knowledge base (KB) and lack direct lexical overlap with the user's query. PACEMAKER treats retrieval as a diagnostic selection task: it plans multiple query views to probe for potential conflicts, expands the search via multi-hop graph traversal to find indirectly related evidence, and applies a final filter to isolate the specific facts that determine feasibility.

PACEMAKER significantly improves conflict-aware reasoning performance compared to standard retrieval baselines.

In the open-source setting, PACEMAKER achieves a 68.82% PASS rate on the PACE benchmark, outperforming dense and sparse retrieval baselines (62.39% and 62.73% respectively). It improves Conflict PASS rates by up to 11.4 percentage points over the strongest non-oracle baselines.

Multi-hop graph traversal is the most critical component for surfacing latent conflict evidence.

Ablation studies show that removing the traversal stage causes the largest performance drop, reducing the PASS rate on Conflict queries from 75.19% to 52.41%. A 22.78 percentage point decrease in Conflict PASS rate.

Why is this problem harder than standard safety or risk-detection tasks?

Standard safety benchmarks focus on explicit risks in the input, whereas PACE requires integrating distributed, implicit facts from a personal KB that appear benign in isolation but create conflicts when combined.

How does this framework differ from existing graph-based retrieval methods?

Unlike general-purpose graph retrieval, PACEMAKER is specifically designed for diagnostic evidence selection; it uses conflict-aware query planning and targeted filtering to prioritize facts that determine request feasibility rather than just topical relevance.

The Challenge of Conflict-Aware Personalization

We expose the gap between request execution and situational awareness in personalized assistants.

Existing assistants excel at executing user requests but ignore whether those requests fit the user’s current circumstances. This blind execution can lead to inappropriate actions, while overly cautious assistants degrade usefulness. The core problem is the missing link between request execution and situational awareness.

Personalization that not only tailors responses but also checks whether a request aligns with the user’s current circumstances.

**Figure 1.** Overview of conflict-aware personalization. The user-centric knowledge base contains information about the user, related people, and surrounding situations. Although the reservation request appears executable, reasoning over evidence about the companion's dietary constraint and the restaurant's fixed menu reveals a latent conflict, requiring the ideal assistant to refuse the booking and suggest an alternative.

Bridging the gap between request execution and situational awareness is essential for trustworthy assistants.

The PACE Benchmark

Introducing the PACE benchmark and its core components for conflict‑aware evaluation.

Personalized assistants often miss hidden conflicts that only become apparent when a user’s own contextual facts are considered. To expose this blind spot, the authors release Personalized Assistants for Conflict Evaluation (PACE), a dataset that forces models to reason over an egocentric knowledge base and decide whether a request is feasible.

The PACE benchmark bundles realistic user requests with a personal knowledge base, asking models to detect when a request silently clashes with the user’s own context.

An egocentric KB stores facts about the user (the “ego”) and a small set of close contacts (the “alters”), capturing only the details that matter for the user’s decisions.

Each request is labeled as “conflicting” if any KB fact makes the request impossible or inappropriate, otherwise it is “non‑conflicting”.

The dataset categorizes conflicts into three situation types. Temporal conflicts stem from schedule or travel constraints, Personal conflicts violate health or personal values, and State conflicts arise from external conditions such as venue closures or road hazards.

The PACEMAKER Framework

PACEMAKER orchestrates multi‑agent retrieval to expose hidden conflicts in user‑centric knowledge bases.

Standard retrieval pipelines retrieve documents that match the query surface, but they miss the hidden contradictions that arise from a user’s personal context. PACEMAKER adds conflict‑aware agents that steer retrieval toward evidence of such contradictions.

Think of PACEMAKER as a detective team: one member sketches the crime scene (query planning), another gathers clues from both obvious and obscure sources (hybrid retrieval), a third follows the trail through a network of related leads (graph traversal), and the final member decides which clues prove the suspect’s alibi (evidence selection) before delivering the verdict (answer generation).

How does PACEMAKER differ from a standard Retrieval‑Augmented Generation (RAG) pipeline?

RAG retrieves documents solely based on lexical or dense similarity to the original query. PACEMAKER first expands the query into conflict‑focused views, then fuses results with WRRF that up‑weights those conflict‑oriented views, and finally traverses a $k$‑NN graph to pull in indirect evidence. This extra reasoning loop explicitly surfaces contradictions that a vanilla RAG would miss.

Conflict‑aware query planning: the conflict planner extracts up to three probing cues (e.g., scheduling constraints) and the multi‑view generator emits the original query plus counter‑views targeting those cues.

Hybrid retrieval and fusion: each view is sent to a dense encoder and a BM25 retriever; WRRF merges all results, assigning higher weight to counter‑view hits.

Pre‑hop filtering: the top‑N fused documents are screened by a pre‑hop filter agent to form the seed set.

Multi‑hop graph traversal: starting from the seed set, a breadth‑first search expands each frontier document to its top‑$M$ neighbors, repeating up to depth $H$.

Post‑hop filtering: a second filter agent selects the most decision‑relevant $N$ documents from the expanded pool.

Answer generation: the answer generator consumes the final evidence set and produces a feasibility verdict with an explanatory rationale.

Build a $k$‑NN graph: each fact links to its two closest facts, yielding 12 edges.

Query planning produces two views: the original request and a counter‑view targeting a scheduling conflict.

Hybrid retrieval returns 5 documents per view; WRRF fuses them, giving the counter‑view documents a weight of 1.5 versus 1.0 for the original view.

Pre‑hop filter selects the top‑3 weighted documents as seeds.

Graph traversal (depth 1) expands each seed to its $M=2$ nearest neighbors, adding up to 6 new documents.

Second traversal (depth 2) repeats, potentially adding another 6 documents, but duplicates are removed.

Post‑hop filter ranks all collected documents by conflict relevance and keeps the top‑3 as final evidence.

Even though the original query matches only a few facts, the multi‑hop expansion uncovers related contradictions hidden two edges away in the graph.

**Figure 2.** Workflow of PACEMAKER.

Experimental Setup

Benchmarking retrieval and response quality across baselines.

We evaluate both retrieval effectiveness and the quality of generated responses on the PACE benchmark, comparing PACEMAKER against several baselines.

PACEMAKER improves the PASS rate over dense retrieval.

On the Qwen3 configuration, PASS rises from 60.35 % (Dense) to 68.98 % (PACEMAKER).

**Table 2.** Main benchmarking results on PACE. All metrics are reported on a percentage scale (%).

The metrics used to evaluate retrieval and response quality drive the comparison.

Main Benchmarking Results

We revisit the conflict‑aware task and show how PACEMAKER outperforms baselines.

Recall that personalized assistants must detect conflicts between a user request and the user’s own egocentric knowledge base. This section quantifies how well our PACEMAKER framework resolves that challenge.

PACEMAKER achieves a 68.82 % PASS rate on the open‑source PACE setting, outperforming all retrieval‑based baselines.

Table 3 shows the PASS/WRONG/FAIL breakdown for each method.

**Table 3.** Response quality results by feasibility status in three configurations.

Conflict vs. Non-Conflict Performance

Analysis quantifies PACEMAKER’s advantage on conflict queries and evidence coverage.

PACEMAKER improves Conflict PASS rate over the strongest non‑oracle baseline by up to 11.40 percentage points.

Table 3 reports Conflict PASS gains of 11.40, 3.47, and 4.02 points for Qwen, GPT, and Gemini configurations respectively.

Figure 3 shows that partial evidence already yields a non‑trivial PASS rate, but full gold coverage consistently raises performance, confirming that conflict resolution often requires multiple atomic facts. The ablation in Table 4 reveals that removing multi‑hop traversal drops Conflict PASS to 52.41 %, underscoring its critical role. Table 5 further highlights PACEMAKER’s edge over structured retrieval baselines, especially on Conflict queries.

**Figure 3.** PASS rates by gold evidence coverage for Conflict queries.

Ethics and Limitations

This section discusses limitations, ethical considerations, and acknowledges support.

We introduced the PACE benchmark and the PACEMAKER framework, showing that PACEMAKER outperforms retrieval‑based baselines and that retrieval completeness is especially critical for Conflict queries.

The benchmark isolates feasibility judgment rather than full task execution, so it does not assess downstream capabilities such as recommendation or planning, and the current evaluation is training‑free, leaving room for specialized agents.

Our dataset uses synthetic identity profiles, avoiding real personal data, yet it mimics realistic personal contexts; we filter harmful or unsafe queries but advise users to handle the data responsibly.

We thank the reviewers and action editor, and acknowledge funding from the Institute of Information & Communications Technology Planning & Evaluation (IITP) grants supporting this work.

Human Evaluation

Human evaluation shows >93% agreement between the automatic judge and annotators, confirming reliability.

The automatic judge aligns with human annotators on 93.5% of evaluated instances.

Aggregated across PASS, WRONG, and FAIL labels for 600 sampled instances.

Annotation quality was ensured by restricting workers to >10 000 approved HITs and ≥98% approval rate, with each instance reviewed by three annotators and resolved by majority vote. Disagreements concentrate on Conflict queries, where humans sometimes adopt a more permissive feasibility interpretation than the stricter construction pipeline.

Ablation Study

Personalized assistants miss conflicts; this section tests which PACEMAKER components matter.

Recall that the paper’s goal is to surface hidden conflicts by reasoning over egocentric knowledge bases. This section isolates each PACEMAKER component to see how much it contributes to that goal.

**Table 21.** Ablation study of agent components for retrieval and overall performance.

PACEMAKER outperforms every ablated variant on both retrieval and response quality.

Table 21 reports the highest scores for the full model across all metrics.

**Table 23.** Retrieval and overall performance across different models.

**Table 24.** Query performance by feasibility status across different models.

GraphRAG queries are limited by a 300‑second timeout; 2.89 % of 3,249 queries timed out, with completed calls averaging 16.3 s.

Implementation Details

Implementation details of PACEMAKER, covering indexing, planning, retrieval, traversal, and evidence selection.

PACEMAKER is built from several independent agents, each with a small set of hyperparameters that govern its behavior. Because the system is training‑free, we employ off‑the‑shelf models without any additional fine‑tuning.

**Table 6.** Query distribution of PACE.

The image displays two examples of situation-based queries, their feasibility status, gold judgments, and model responses. The first example involves a physical task (carrying a box) that conflicts with a health constraint, while the second involves a reminder task that is compatible with a health routine.

**Table.** Ablation study results comparing different methods across Overall Quality, Conflict Query, and Non-conflict Query metrics, categorized by PASS, WRONG, and FAIL outcomes.

**Table 14.** Retrieval and overall performance under different hyperparameter settings.

Indexing encodes every KB document with the embedding model and builds a $k$‑NN graph ($k=10$). Query planning creates up to three conflict dimensions, which feed a generator that produces up to three counter queries per original query.

Hybrid retrieval pulls the top‑$K=10$ results from dense and sparse retrievers, merges them via Weighted Reciprocal Rank Fusion ($k=60$) with weights $1.0$ for original queries and $1.2$ for counter queries, and selects the top‑$N_{\text{seed}}=20$ fused documents for the pre‑hop filter.

Multi‑hop graph traversal starts from $N_{\text{filter}}=10$ seed documents, explores up to $H=5$ hops, and expands $M=3$ neighboring documents at each hop. The post‑hop filter then chooses the final $N_{\text{final}}=10$ evidence documents for answer generation.

Model Output Analysis

Appendix D presents model output examples and evaluates embedding and judge choices.

We compare three dense‑retrieval embedding models—Qwen3‑Embedding‑8B, BGE‑M3, and NV‑Embed‑v2—to see how much the choice influences overall performance.

All three models achieve nearly identical scores across retrieval and downstream metrics, indicating that the quality of conflict‑aware reasoning and agentic filtering dominates the final outcome.

Consequently we adopt Qwen3‑Embedding‑8B for the open‑source experiments, as it yields the best overall numbers and aligns with the agent model family.

Across situation types, temporal queries consistently attain the lowest PASS Rate because they require aggregating evidence from multiple time‑related facts.

**Table 18.** Model output examples for temporal-type queries.

For completeness we also provide analogous examples for personal‑type (Table 19) and state‑type (Table 20) queries.

To test robustness of the evaluation pipeline we switch the judge model to Qwen3‑4B‑Instruct‑2507 while keeping the same set of queries.

**Table 8.** Evaluation results with different judge models.

The two judges agree on 86.40 % of cases, but Gemini tends to label conflict responses as WRONG and non‑conflict responses as PASS.

Computational Cost Analysis

Computational cost of PACEMAKER and baselines is broken down by indexing, retrieval, and LLM calls.

We evaluate computational cost by measuring latency and the number of LLM calls for offline indexing and online retrieval, using a randomly sampled instance containing 2,056 facts and 18 queries.

The original persona lists a fashion‑model‑actor with a California surf‑skate lifestyle; the expanded persona adds contextual details about habits, health, and upcoming relocation.

Tables 9, 10, and 11 illustrate the persona expansion and profile synthesis, showing demographic, geographic, socio‑economic, health, and device attributes in a structured format.

Tables 12 and 13 present conflict‑query and non‑conflict‑query examples from the PACE dataset, highlighting temporal, personal, and state constraints.

Tables 14 and 15 report retrieval performance under various hyper‑parameter settings and overall quality metrics, including PASS, WRONG, and FAIL rates.

Computational Cost Data

Key computational cost and performance numbers for the PACEMAKER system and baselines.

Tables 15–24 report the detailed computational and retrieval metrics that underpin the cost analysis of PACEMAKER and its competitors.

Across all tables, PACEMAKER consistently outperforms the baselines on both retrieval quality and computational efficiency, confirming the design choices described earlier.

Dataset Generation: Persona Expansion

Defines the prompts used to expand personas and generate conflict‑aware queries for the dataset.

The first prompt takes a brief, plain‑language persona description and rewrites it into a single, coherent narrative paragraph, adding neutral contextual details while avoiding explicit reasons or conflicts.

The second prompt similarly expands a sub‑character seed, grounding the resulting narrative in the main character’s environment and shared routines without stating the relationship explicitly.

The third prompt converts an ego persona seed into a highly granular JSON “Synthetic PII Profile,” enumerating concrete attributes such as health, tech ecosystem, and demographics while preserving internal consistency.

The fourth prompt generates a JSON entry describing a related alter, including relation type, minimal basic info, traits, conflict‑relevant hooks, and shared context facts that justify the relation.

The final prompt produces conflict‑aware and non‑conflict cases by pairing ego and alter personas with situational queries, contexts, and judgments, respecting temporal windows and ensuring each case hinges on concrete persona facts.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers