Bilevel Coordinated Reflection: a Game-Theoretic Approach to Multi-Agent LLM Systems

Yihang Chen, Yuxiang Chen, Yuxuan Huang, Meng Fang, Weilin Luo, Jun Wang

Multi-agent LLM systems improve via grounded, verifier-gated memory updates that guarantee convergence.

How can multi-agent LLM systems achieve stable, grounded performance by treating reflection as a bilevel game between an orchestrator and workers?

Multi-agent LLM systems often use "reflection" to improve performance, but unconditional appending of critiques leads to hallucination cascades and performance plateaus. The authors model this as a bilevel coordination game and introduce Stochastic Reflective Memory Ascent (SRMA), which only commits memory updates that strictly decrease a grounded verifier's risk. This approach replaces heuristic-based reflection with a formal protocol that converges geometrically or polynomially, resolving 72.2% of SWE-bench instances compared to 70.8% for standard references.

Paper Primer

The paper frames multi-agent LLM systems as a bilevel coordination game where an orchestrator partitions tasks and workers solve them. The core problem is that "free-form" reflection—where agents append critiques to memory without external validation—is prone to persistent harmful drift, as text-only judges cannot distinguish between corrective and hallucinated content in ambiguous environments.

SRMA is a verifier-gated update rule: it probes candidate memories against a grounded protocol (like a test harness or simulator) and commits only those that strictly reduce the verifier's risk. This mechanism acts like a filter on a noisy stream, ensuring that only verified improvements enter the shared memory, which forces the system to converge toward an optimal state.

Grounded verification is information-theoretically necessary for uniform improvement.

The authors prove an impossibility theorem showing that no text-only gate can improve uniformly across indistinguishable environments, whereas a grounded gate recovers geometric convergence. Grounded SRMA outperforms self-gating (text-only) by 14–30% in Overcooked coordination tasks.

SRMA achieves exact convergence with order-tight rates.

Under calibration and non-degenerate corrective mass, the system converges geometrically (if the error reduction is constant) or polynomially (if it scales with the error). On SWE-bench, the complete system resolves 72.2% of instances, exceeding the 70.8% baseline.

Why does "free-form" reflection fail to converge in practice?

Unconditional reflection mixes corrective information with hallucinated content, creating a "hallucination floor" where harmful commitments inject non-vanishing error that the system cannot self-correct.

What is the role of the orchestrator in this framework?

The orchestrator manages the decomposition quality; it solves a bilevel game to maximize global utility while minimizing the "coupling slack" between workers, which directly controls the stability of the followers' subgame.

Introduction: The Drift Problem

We expose why self‑contained reflection loops drift and how grounding can halt the failure.

Multi‑agent LLM systems repeatedly edit a shared textual memory, yet without an external anchor these reflection loops tend to drift: the memory evolves in directions that are not tied to the task environment, leading to hallucination and divergence.

The failure of self‑contained reflection loops is the core obstacle this work seeks to overcome.

Bilevel Coordination Game

We formalize multi‑agent query solving as a bilevel coordination game that guarantees convergent worker updates.

Naïve single‑agent generation dilutes context and degrades reasoning, especially for complex queries. Modern systems split the task: a leader proposes a high‑level decomposition, then workers iteratively refine their pieces. This division creates a coordination problem that we cast as a bilevel game.

The orchestrator picks a high‑level plan $\tau$, workers respond with local solutions $x_i$, and the resulting interaction forms an $\eta_{c}$‑approximate potential game that converges to a stable joint output.

How does this bilevel coordination game differ from standard hierarchical planning?

In hierarchical planning the leader’s plan is fixed and workers simply follow it. Here workers iteratively perform $\eta_{c}$‑better‑response updates, and the leader chooses $\tau$ while explicitly accounting for the equilibrium that those updates will reach, turning the interaction into an approximate potential game with a provable slack bound.

Initial actions: x₁⁰ = 0, x₂⁰ = 0. Local utilities: u₁(0 | $\tau$₁)=0.5, u₂(0 | $\tau$₂)=0.4.

Worker 1 evaluates switching to 1: u₁(1 | $\tau$₁)=0.8, interaction change $\Delta$$\psi$₁ \leq 2 \cdot $\kappa$ = 0.2, so total gain = 0.8 - 0.5 + $\Delta$$\psi$₁ ≥ 0.5 > $\eta_{c}$.

Because the gain exceeds $\eta_{c}$, worker 1 updates to x₁¹ = 1 ($\eta_{c}$‑better‑response).

Worker 2 now evaluates switching to 1: u₂(1 | $\tau$₂)=0.6, $\Delta$$\psi$₂ \leq 0.2, total gain = 0.6 - 0.4 + $\Delta$$\psi$₂ ≥ 0.4 > $\eta_{c}$, so x₂¹ = 1.

The process stops because no unilateral deviation can improve a worker’s utility by more than $\eta_{c}$; the profile (1,1) is an $\eta_{c}$‑approximate Nash equilibrium.

Even with tiny coupling ($\kappa$ = 0.1) the bound $\eta_{c}$ = 0.2 guarantees that a modest improvement suffices for each worker to settle into a stable joint solution.

**Figure 1.** Bilevel coordinated reflection. The orchestrator (leader) selects a decomposition $\tau$ and updates strategy memory $m_o$ on the slower timescale; workers (followers) update execution memory $m_e$ via $\eta_c$-better responses on the faster timescale. Under bounded coupling, the followers' subgame is an approximate potential game with slack $\eta_c \leq 2d_{\text{max}}\kappa$, while verifier-gated SRMA separately governs which memory proposals are committed.

Dual-Memory Drift Dynamics

We expose how dual‑memory drift dynamics govern reflection error and enable bounded convergence.

When reflections are written directly into memory, harmless updates mix with harmful hallucinations, causing the system’s error to drift upward.

Two separate buffers—execution memory $m_e$ for concrete actions and strategy memory $m_o$ for high‑level plans—drift independently, letting the orchestrator correct one without corrupting the other.

How does dual‑memory drift differ from using a single shared memory?

With a single buffer, corrective and harmful updates interfere, making it impossible to isolate the effect of a reflection. By splitting into $m_e$ and $m_o$, the system can apply drift to the strategy layer while preserving the concrete execution state, preventing error from propagating unchecked.

Step 1: $V_0 = |m_e - m_o| = 0.3$.

Step 2: Apply the drift recursion $V_1 = (1-\gamma)V_0 + \nu = 0.6\cdot0.3 + 0.1 = 0.28$.

Step 3: Repeat: $V_2 = (1-\gamma)V_1 + \nu = 0.6\cdot0.28 + 0.1 \approx 0.268$.

Step 4: The error settles near the fixed point $\nu/\gamma = 0.1/0.4 = 0.25$, never dropping to zero.

The residual term $\nu$ creates a non‑zero error floor, illustrating why unrestricted reflection can’t fully eliminate hallucination.

Theorem 2 (Finite‑Time Upper Bound) states that under Assumption 2, the expected error $e_T$ decays geometrically toward the floor $\nu/\gamma$, guaranteeing $e_T \le (1-\gamma)^T e_0 + \frac{\nu}{\gamma}[1-(1-\gamma)^T]$.

Proposition 1 (Worst‑Case Tightness) shows the bound $\nu/\gamma$ is tight: there exists a deterministic process with $V_{t+1} = (1-\gamma)V_t + \nu$ that exactly attains the limit $\nu/\gamma$.

Theorem 3 (Universal Lower Bound) proves that under Assumption 3 the error cannot fall below the same floor, i.e. $e_T \ge (1-\gamma)^T e_0 + \frac{\nu}{\gamma}[1-(1-\gamma)^T]$, so $\liminf_{T\to\infty} e_T \ge \nu/\gamma > 0$.

Corollary 2 (Two‑Sided Error Tube) combines the two drift conditions, yielding $\frac{\nu}{\gamma} \le \liminf_{T\to\infty} e_T \le \limsup_{T\to\infty} e_T \le \frac{\nu}{\gamma}$; when the drift rates match, the error converges exactly to $\nu/\gamma$.

The Necessity of Grounding

The section shows why a gate must observe an environment‑dependent signal to guarantee error reduction.

Self‑contained gates that rely only on the generated transcript cannot guarantee improvement because they lack any signal that distinguishes environments with different external states. This limitation motivates the need for a grounding signal that varies with the environment.

Grounding means the gate can look at a signal that comes from the outside world—e.g., a reward, a simulator state, or a formal‑checker result—so its acceptance decision can depend on facts that the text alone cannot reveal.

How does a grounded gate differ from a self‑contained gate that only sees the text?

A self‑contained gate decides solely from the proposal distribution $P(\cdot\mid m)$ and its internal randomness, so identical transcripts lead to identical decisions in both environments. A grounded gate also reads the environment‑dependent signal $V_{\varepsilon}(m)$, which can differ between $\varepsilon^{+}$ and $\varepsilon^{-}$ even when the transcript is the same, allowing it to accept corrective proposals in one environment and reject them in the other.

Step 1: A proposal from $C_{0}$ arrives. In $\varepsilon^{+}$ the gate applies $f_{0}(v)=(1-\kappa)v=0.7v$, so error becomes $0.7\times0.4=0.28$. In $\varepsilon^{-}$ the same proposal is treated as harmful and applies $f_{1}(v)=\kappa+(1-\kappa)v=0.3+0.7v$, yielding $0.3+0.7\times0.4=0.58$.

Step 2: A proposal from $C_{1}$ arrives. In $\varepsilon^{+}$ it applies $f_{1}$, giving $0.3+0.7\times0.28=0.496$. In $\varepsilon^{-}$ it applies $f_{0}$, giving $0.7\times0.58=0.406$.

Step 3: The gate accepts the $C_{0}$ proposal with probability $0.5$ (positive probability). After acceptance, the error in $\varepsilon^{+}$ is $0.28$, while in $\varepsilon^{-}$ it is $0.58$.

Because the gate can observe $V_{\varepsilon}(m)$, it knows which class ($C_{0}$ or $C_{1}$) is corrective in the current environment and can preferentially accept the beneficial one, driving error toward $0$ in both environments.

For every self‑contained gate and every horizon $T$, $\max\{e_{\varepsilon^{+}},\,e_{\varepsilon^{-}}\}\ge e_{0}$. Moreover, if $e_{0}<\tfrac12$ and the gate accepts at least one proposal from $C_{0}\cup C_{1}$ with positive probability by time $T$, then the inequality is strict. In contrast, the free‑form rule (accept everything) yields $e_{\varepsilon}=2$ in both environments, while a grounded gate that observes $V_{\varepsilon}$ accepts only the corrective class and satisfies $e_{\varepsilon}\to 1$ as $T\to\infty$.

Couple the two environments with shared proposal and gate randomness.

The reflection identity $f_{1}^{-a}(v)=1-f_{a}(1-v)$ yields $e_{\varepsilon^{+}}\ge 2e_{0}$.

The free‑form rule induces the affine recursion $e_{\varepsilon}^{(t+1)}=2-e_{\varepsilon}^{(t)}$, converging to $2$, while a grounded gate follows $e_{\varepsilon}^{(t+1)}=f_{a}(e_{\varepsilon}^{(t)})$ with $a$ chosen correctly, converging to $1$.

Because they cannot distinguish between environments that differ only in external state, any self‑contained gate can be forced into a scenario where its acceptance decision harms one environment while helping the other, guaranteeing a non‑trivial error lower bound.

Verifier-Gated SRMA Mechanism

Verifier‑gated SRMA enforces environment‑grounded reflection to guarantee convergence.

Recall that unchecked reflection can drift into hallucination; the core trick is to let a deterministic verifier gate each update, ensuring only environment‑grounded improvements survive.

Instead of letting every stochastic reflection proposal modify the memory, we first run a deterministic verifier that checks the proposal against a fixed, environment‑grounded evaluation; only proposals that improve the verifier’s risk are accepted.

Acceptance probability $p = c_1 R^{\beta}=0.5 \times 0.4^{0.5}=0.5 \times 0.632=0.316$.

Sample a uniform random number; suppose it is $0.2$, which is $<p$, so we accept the candidate.

On acceptance the risk drops by at least $c_2 R = 0.6 \times 0.4 = 0.24$, giving new risk $R' = 0.4 - 0.24 = 0.16$.

Repeat: new acceptance probability $p' = 0.5 \times 0.16^{0.5}=0.5 \times 0.4 = 0.20$.

Assume the next random draw is $0.25$ (reject), so risk stays $0.16$.

The gate guarantees a multiplicative reduction in risk whenever acceptance occurs, and the acceptance probability scales with the current risk, driving the process toward zero.

How does Verifier‑Gated SRMA differ from a plain SRMA that accepts any stochastic proposal?

Plain SRMA would accept a proposal based solely on a single stochastic sample, which can increase verifier risk due to variance. The gated version first evaluates the proposal with a deterministic verifier and only accepts if the risk strictly improves, providing a monotone drift guarantee.

Algorithm 1: Stochastic Reflective Memory Ascent for worker $i$

**Table 3.** Accepted proposals and final risk over five seeds. Rates are fractions of harmful/helpful proposals accepted.

Empirical Evaluation

We report the empirical gains of grounded SRMA across three benchmark suites.

Grounded SRMA resolves 72.2 % of SWE‑bench instances, beating free‑form multi‑agent reflection (58.4 %) and the public mini‑SWE‑agent v2 baseline (70.8 %).

Table 2 shows the highest success rate for the “Bilevel SRMA” row with the Kimi K2.5 backbone.

**Table 1.** Overcooked score over five seeds under matched interaction and model-call budgets. Score equals deliveries $\times$ 20; higher is better.

The table compares the performance of different systems using various backbones, measured by a success rate. The systems listed are mini-SWE v2, Bilevel SRMA, and Free-form MA, utilizing either DeepSeek or Kimi K2.5 backbones. The Bilevel SRMA system with the Kimi K2.5 backbone achieves the highest rate of 72.2%.

The table compares four methods ("No reflection", "Free-form", "Self-gate", and "Grounded SRMA") across three metrics: "Harmful", "Helpful", and "Risk".

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers