USERARENA: Benchmarking Interactive User Simulation for Long-Horizon Shopping Agents

Jujia Zhao, Jiakai Tang, Jing Yu, Wen Chen, Jian Wu, Han Zhu, Fei Sun, Suzan Verberne, Zhaochun Ren

USERARENA provides a long-horizon shopping benchmark and a distillation method to align user simulators with real-world behavioral patterns.

How can we build a user simulator that accurately models long-horizon, multi-session shopping behavior for evaluating recommendation agents?

Existing user simulators struggle to maintain behavioral consistency over long shopping sessions, often failing to reproduce the nuanced, multi-session patterns observed in real-world e-commerce data. The authors introduce USERARENA, a benchmark pairing real-user trajectories with a production-grounded environment, and propose On-Policy Trajectory Distillation (OPTD), which uses trajectory-level intent to guide student models toward coherent long-horizon behavior. OPTD achieves the highest overall fidelity across five evaluation levels and most closely reproduces the policy effects observed in controlled production A/B tests.

Paper Primer

USERARENA addresses the gap between isolated next-action prediction and long-horizon behavioral fidelity. By linking cross-session trajectories and providing a five-level evaluation protocol—ranging from environment compliance to aggregate outcome fidelity—it allows researchers to test whether a simulator preserves a user's preferences and tendencies across repeated visits.

OPTD improves simulator fidelity by replacing standard teacher forcing with on-policy distillation: it samples student-generated rollouts and uses a frozen teacher to provide dense, intent-conditioned supervision at every token. A trust-region constraint prevents the intent-based guidance from drifting too far from the student's observable state, ensuring the model remains grounded in the current interaction context.

OPTD achieves the strongest overall behavioral fidelity among evaluated simulators.

Across the USERARENA protocol, OPTD ranks first on all eight L1–L4 metrics in single-session settings and leads on five of eight metrics in cross-session settings. OPTD reaches an effect ratio of 0.713× in reproducing production A/B test outcomes, significantly outperforming baseline simulators that often overestimate or fail to capture the online policy effect.

Why is "next-action agreement" insufficient for evaluating these simulators?

Next-action agreement measures step-wise choices in isolation, which fails to capture whether a simulator preserves a user's behavioral tendencies, personalized preferences, or aggregate outcomes over long, multi-session horizons.

What is the primary advantage of OPTD's on-policy training over standard teacher forcing?

Standard teacher forcing conditions predictions on ground-truth prefixes, creating a mismatch with autoregressive inference where the model must condition on its own potentially erroneous outputs. OPTD trains on student-generated prefixes to reduce this exposure bias.

Researchers can now use USERARENA to validate simulators against production-grounded outcomes, while OPTD provides a robust training recipe for aligning LLM-based agents with long-term human shopping behavior.

Motivation and Problem Framing

Existing simulators miss long‑horizon, cross‑session shopping intent, prompting a new benchmark and training method.

Agentic recommendation requires a world model that simulates both the shopping environment and user responses over long interaction sequences. Existing resources lack long‑horizon, fine‑grained real‑user trajectories and cannot evaluate whether simulated behavior aligns with production A/B outcomes. Consequently, current simulators cannot faithfully capture cross‑session shopping intent.

We therefore introduce USERARENA, a benchmark that pairs real multi‑session trajectories with an interactive environment and supports offline validation against a controlled production A/B test. To improve simulator fidelity, we propose On‑Policy Trajectory Distillation (OPTD), which provides dense teacher targets on student‑generated rollouts using trajectory‑level intent. Experiments show OPTD yields the highest overall fidelity across a five‑level evaluation protocol.

The core gap is that existing simulators fall short of reproducing long‑horizon, cross‑session shopping behavior.

The USERARENA Benchmark

Defines USERARENA and its multi‑level evaluation protocol.

Existing user simulators cannot preserve long‑horizon, cross‑session shopping intent. USERARENA fills this gap by providing a production‑grounded environment together with fine‑grained real‑user trajectories and a five‑level fidelity protocol.

Think of USERARENA as a sandbox that mirrors the live store: it reproduces the exact feed layout, item‑detail pages, and the full set of shopper actions, letting agents walk through real‑world shopping histories.

Compute average sessions per user: 2 sessions / 5 users = 2.

Compute average actions per session: 4 actions / 2 sessions = 2.

Total feed‑scroll actions = 5 users × 2 sessions × 1 scroll = 10.

Total item‑click actions = 5 users × 2 sessions × 1 click = 10.

Overall action count = 20 feed‑scroll + 20 item‑click = 40, matching the dataset size.

This tiny instance already exhibits the same structural ratios (≈ 50 % scroll, 50 % click) that the full benchmark reports, making the scaling behavior transparent.

How does USERARENA differ from earlier interactive recommendation benchmarks?

Earlier benchmarks either omit visual context, restrict the action space, or lack long‑horizon cross‑session data. USERARENA uniquely combines a full‑action space, real step‑level trajectories, cross‑session histories, and production A/B validation in a single package.

**Figure 1.** Overview of USERARENA. **Left:** cross-session histories, step-aligned interaction records, production A/B validation, and dataset scale. **Right:** the five-level evaluation protocol, progressing from environment compliance (L0) to outcome fidelity (L4).

The benchmark’s evaluation protocol systematically escalates the fidelity demand, moving from simple parsing checks to full‑trajectory outcome matching.

Imagine grading a student’s essay by first checking spelling, then grammar, then logical flow, and finally whether the argument convinces a real audience; USERARENA’s five levels do the same for simulators.

Why provide both Replay and Rollout modes instead of a single evaluation?

Replay isolates the model’s immediate decision quality without contaminating the state, while Rollout reveals how small prediction errors accumulate over an entire session, exposing stability and long‑term realism issues.

**Table 1.** Comparison of representative interactive recommendation benchmarks. Full Action Space covers the complete executable interaction loop. Long-Horizon denotes temporally ordered user histories spanning multiple sessions. Production A/B Validation evaluates recommendation policies offline in simulation and compares their effects with outcomes observed in production A/B tests.

Benchmark design must preserve production‑grounded observations, expose the full action space, span long‑horizon cross‑session histories, and validate simulated policies against real A/B outcomes.

On-Policy Trajectory Distillation

OPTD injects trajectory‑level intent into on‑policy distillation to close the train‑test gap.

Standard simulators train on per‑step teacher forcing, which leaves them blind to the long‑range shopping intent that spans multiple sessions.

OPTD lets a frozen teacher watch the student’s own rollout and, using a summary of the whole session, give token‑by‑token guidance—like a coach who watches a player’s full game and offers move‑by‑move feedback based on the overall strategy.

How does OPTD differ from ordinary on‑policy distillation that simply copies the teacher’s logits?

Ordinary on‑policy distillation conditions the teacher only on the student’s observable state and prefix, so the teacher’s targets reflect what the student already knows. OPTD adds a trajectory‑level intent $z_u$, giving the teacher privileged context about the whole session; the trust region then limits how far this extra signal can pull the student away from its own information.

Trajectory‑level intent is a compact, abstract “shopping goal” extracted from an entire session—think of it as a high‑level itinerary (e.g., “compare phones, then buy the cheapest”) that guides token‑level decisions without spelling out each concrete click.

Why not just give the student the intent $z_u$ directly instead of routing it through a teacher?

Providing $z_u$ directly would expose the student to privileged information it could not infer at test time, breaking the on‑policy setting. By keeping the teacher frozen and only using $z_u$ to shape its token distribution, the student still learns from its own observable state while benefiting from intent‑guided supervision.

Sample a state $x_t$ from the training distribution $\mathcal{D}$.

Generate a full response $y_t$ from the current student policy $\pi_{\theta}$ (Eq 1).

Extract the session‑wide intent $z_u = E(\tau_u)$ (Eq 2).

Compute the intent‑conditioned teacher distribution $t_k(v)$ (Eq 3) for each token position.

Compute the unconditioned reference $q^{0}_{t,k}(v)$ (Eq 4).

Interpolate to obtain $q^{\lambda}_{t,k}(v)$ (Eq 5) and select $\lambda^{\star}_{t,k}$ via the KL trust region (Eq 6).

Use the resulting $q^{\lambda^{\star}}_{t,k}$ as the target for the student’s cross‑entropy loss.

Step 1: Sample $x_t$ = page 1 (phone list). Student policy generates $y_{t,1}$ = “I like the red phone”.

Step 2: Teacher computes $t_2(v)$ for the next token given $z_u$ and prefix “I like”. The intent pushes probability toward price‑related tokens.

Step 3: Unconditioned reference $q^{0}_{t,2}(v)$ assigns higher mass to generic continuation “the”.

Step 4: KL budget $\epsilon$ forces $\lambda^{\star}_{t,2}=0.6$, blending intent and reference.

Step 5: Student receives the blended target and updates its parameters to increase likelihood of “\$300” at position 2.

The trust region lets the intent guide the student when it aligns with the observable state, but pulls back when the intent would force an implausible token given the current page.

**Figure 2.** Supervision in SFT, GRPO, and OPTD. SFT learns from recorded responses, GRPO assigns a scalar reward to each generated response, and OPTD provides full-vocabulary teacher targets at every token of the student's own rollout.

**Figure C.3.2.** Trajectory-level intent example. (a) Cross-session action sequence. (b) Trajectory-level intent anchors.

OPTD Training Objective

A combined loss transfers trajectory intent while grounding each step with recorded responses.

Exact‑match objectives only reward a single token sequence, ignoring alternative phrasings a real shopper might use. This creates a gap between training signals and the diversity of actual behavior.

The loss blends a trajectory‑level distillation term with a step‑level grounding term, so the student receives both high‑level intent and concrete token supervision.

How does this differ from standard knowledge distillation?

Standard KD aligns the student’s output distribution to the teacher’s at each individual prediction, typically using KL divergence on a single softmax vector. Here the distillation term operates over the entire generated token sequence with a generalized Jensen–Shannon divergence, and it is explicitly combined with a separate teacher‑forced cross‑entropy term that grounds the student on recorded responses.

Compute the mixed distributions $m$₁=0.5·q₁+0.5·p₁=[0.7,0.3] and m₂=0.5·q₂+0.5·p₂=[0.4,0.6].

Evaluate JSD_$\beta$ for each token: JSD_$\beta$(q₁‖p₁)≈0.02, JSD_$\beta$(q₂‖p₂)≈0.01 (both bounded by 0.5).

Average over tokens: `L_dist` = (0.02+0.01)/2 = 0.015.

Assume the recorded response y★ matches the teacher exactly, so the cross‑entropy term yields `L_GT` = -(log 0.8 + log 0.5)/2 ≈ 0.46.

Combine with $\gamma$=1: total loss L = `L_dist` + $\gamma$·`L_GT` ≈ 0.015 + 0.46 = 0.475.

This toy calculation shows how the loss simultaneously nudges the student toward the teacher’s distribution (via the small JSD contribution) while still enforcing exact token matches through the larger grounding term.

Experimental Setup

OPTD attains the highest single‑session accuracy on USERARENA.

OPTD achieves the highest single‑session accuracy on USERARENA.

0.3204 accuracy, surpassing the next‑best STA (0.2899).

We evaluate two families of simulators distinguished by observation modality: text‑based models ingest raw page text, while GUI‑based models consume rendered screenshots. Using the same LLM eliminates scale confounds and isolates the effect of modality and the OPTD training strategy.

RecAgent is a text‑based user simulator that predicts the next user action from a textual representation of the current page.

Age encodes how many shopping sessions a user has completed, serving as a proxy for long‑term intent.

**Figure 4.** Behavioral composition of USERARENA. (a) Distribution of decision actions, grouped into feed, item-page, commitment, and session actions. (b) Distribution of clicked items across product categories, with Other aggregating the remaining 31 categories.

The evaluation spans a diverse set of baselines, covering both text‑ and GUI‑based simulators across single‑ and cross‑session settings.

Main Results

Key gains of OPTD on benchmark metrics and production A/B validation.

The paper’s central claim is that existing simulators miss long‑horizon, cross‑session intent, and that OPTD restores this fidelity.

We evaluate simulators in two modes: single‑session, where each user’s trajectory is confined to one visit, and cross‑session, which stitches a user’s chronologically ordered sessions to test long‑term behavior.

OPTD reproduces the online policy effect most closely, achieving an effect ratio of 0.713×.

Table 3 shows OPTD’s simulated $\Delta$PV‑CTR is 71.3 % of the true online change, the lowest deviation among all simulators.

**Table 2.** Single- and cross-session results on USERARENA across L0–L4. For L1–L4, bold and underlining denote the best and second-best results, respectively, within each setting.

For production validation we fix a pool of 10 000 users, split evenly between two ranking policies, and measure the simulated $\Delta$PV‑CTR against the recorded online $\Delta$PV‑CTR. The effect ratio is the simulated $\Delta$PV‑CTR divided by the online value; the effect error is the absolute deviation from 1. OPTD’s ratio of 0.713× (effect error 0.287) outperforms GPT‑6‑Sol (1.576×) and GPT‑6‑Luna (2.154×), which markedly overestimate the real impact.

Ablations and Human Evaluation

Component ablations and human evaluation reveal each part’s impact on simulator fidelity.

We run five ablations, each removing a single OPTD component while keeping all other settings fixed. Every removal harms at least one fidelity metric, confirming that each piece contributes to realistic behavior. The w/o SFT variant suffers the steepest drop, indicating that warm‑up pre‑training is essential for downstream trajectory distillation.

A/B Agent attains the highest human‑win rate (73.5 %) among the four simulators.

Figure 3(b) shows A/B Agent’s human‑win proportion at 73.5 %, exceeding the other agents.

**Figure 3.** Ablation and human-evaluation results. (a) Component performance relative to the full model. (b) Human win denotes correct identification of the simulated trajectory, Model win denotes misidentification of the real trajectory as simulated, and Unsure denotes abstention.

**Figure 9.** Human evaluation interface. Annotators compare two anonymized e-commerce interaction trajectories conditioned on the same user context. The interface presents recent interaction history, action timelines, and step-level interface observations. Annotators select the trajectory they believe is machine-generated or choose *Unsure*, and provide a supporting rationale.

Beyond OPTD, the field has progressed from static rating datasets (MovieLens, Amazon Reviews) to interactive benchmarks (RecBench+, AgentRecBench, OPeRA) and dedicated simulators (RecSim, Virtual‑Taobao, KuaiSim). Yet most resources lack long‑horizon, cross‑session user intent, which USERARENA explicitly captures.

We introduced USERARENA, a benchmark that records step‑aligned real shopping trajectories and validates simulator predictions against a production A/B test. OPTD, built on trajectory‑level intent, achieves the strongest fidelity across our five‑level protocol and best reproduces the production policy effect.

Future work will launch a leaderboard for USERARENA and explore simulators that maintain evolving preferences across sessions. The AI Use, Reproducibility, Ethics, and Acknowledgments statements follow the paper’s standard policies.

Production A/B Validation

We assess simulator fidelity to production A/B results, their inference cost, and human perception.

The central premise—that existing simulators miss long‑horizon, cross‑session intent and that USERARENA together with On‑Policy Trajectory Distillation (OPTD) aims to close this gap—has already been introduced.

It is an offline sanity check that asks: if we run a policy in a simulator, does the simulated lift in page‑views‑per‑click (PV‑CTR) match the lift observed in a real‑world A/B test?

For the validation we allocate 10 000 users (5 000 per policy) and a static product catalog to each simulator. Five task‑specific simulators (OPTD, STA, Customer‑R1, Shop‑R1, AlignUSER) and three foundation models (Qwen3.5‑2B, GPT‑6‑Sol, GPT‑6‑Luna) are evaluated against the production reference.

All other task‑specific simulators severely under‑estimate the policy response (effect ratios 0.020×–0.297×). The untuned Qwen3.5‑2B reaches only 0.004×. By contrast, the general‑purpose GPT‑6‑Sol and GPT‑6‑Luna over‑estimate, yielding ratios of 1.576× and 2.154× respectively, showing that raw model capacity does not guarantee realistic policy sensitivity.

**Figure 6.** Step-wise inference cost on single-session Rollout. Lines show the mean across simulated trajectories that reach each step, and shaded bands span the mean ±1 sample standard deviation. Before aggregation, we sum all model calls assigned to the same trajectory-step, including method-internal calls and invalid-action retries. Panels A–B report total tokens, Panels C–D report model-call latency, and Panels E–F report the number of contributing trajectories. No absent step is imputed, so curves end when no trajectory reaches a later step.

Efficiency is measured by aggregating token counts and latency for every model call made at a rollout step, including planning calls and retries. Means and sample standard deviations are reported across all trajectories that reach the step, isolating cost growth from interaction length.

Qualitative case studies illustrate USERARENA’s ability to preserve long‑term intent. Session 1 shows a user adding brow mascara and eye cream, Session 2 continues the beauty theme, and Session 3 returns to eye makeup, demonstrating continuity with modest shifts.

Representative trajectories (Section C.3.1) highlight how the same high‑level interest (beauty products) persists across sessions while specific product focus changes, confirming that the benchmark captures both stability and evolution of user goals.

The trajectory‑level intent example (Section C.3.2) extracts stable preferences (elegant design, branded goods) across four visits, abstracting away concrete items. This record conditions the teacher at every state, while the student receives only the current observation, enforcing on‑policy supervision.

Human evaluation (Section C.4) pairs real and simulated trajectories, randomizes their order, and asks annotators to label the machine‑generated one. OPTD attains the lowest machine‑win rate (38.7 %) and highest model‑win rate (26.3 %), with a Fleiss’ $\kappa$ of 0.528, indicating moderate inter‑annotator agreement and that OPTD trajectories are hardest to distinguish from real behavior.

Benchmark Capabilities and Statistics

Appendix A supplies extra benchmark data, user behavior stats, and full metric formulas for evaluation.

A.1 compares interaction horizons across three real‑user shopping benchmarks. Table 4 shows that USERARENA spans an order of magnitude more sessions and actions than OPeRA or USB, confirming its long‑horizon focus.

**Figure 5.** Additional statistics for USERARENA. (a) Empirical cumulative distributions of session length in the single- and cross-session settings. (b) Distribution of sessions per cross-session user. (c) User-profile coverage and click frequency across distinct items. (d) Row-normalized within-session transitions between semantic action groups. (e) Time gaps between adjacent cross-session pairs, excluding overlapping sessions.

A.2 provides deeper statistics on USERARENA. Feed‑page scrolling dominates (38.34 %), item clicks follow (12.76 %), and purchases are rare (1.02 %). Users average 10.29 sessions and 86.36 actions, while 99 %+ of profile fields are populated and 92 % of clicked items appear only once.

**Figure 7.** Ground-truth cross-session behavior over three consecutive visits by the same user. (a) Complete action sequence, with consecutive feed-scroll actions grouped as $Scroll \times N$ for readability. Colors distinguish browsing and navigation, product inspection, and commitment behavior. (b) Full feed observations immediately before four product clicks. Red outlines identify the selected products, and numbered markers link each observation to the corresponding click in panel (a).

A.3 enumerates the five‑level evaluation protocol. Below each level we give the exact mathematical definition used in Table 2.

Table 5 lists the prompt template used by On‑Policy Trajectory Distillation, separating shared student‑teacher context, teacher‑only intent, and the response contract.

Baseline Implementation Details

Appendix B details the baselines, data pipeline, training settings, and prompt design used in experiments.

We evaluate eight representative user simulators, preserving each method’s observation modality, context construction, memory mechanism, and training objective, all within the common USERARENA environment, action space, source test split, and output schema.

The baselines are RecAgent, Agent4Rec, SimUSER, Shop‑R1, Customer‑R1, AlignUSER, A/B Agent, and STA, each described briefly in the paper.

Interaction logs are normalized to the 13‑action USERARENA space and the pre‑action interface state is used to reconstruct visual and textual observations; users are split 8:1:1 with seed 42, keeping all sessions of a user together.

Training and validation examples are resampled for balanced action coverage, while each test trajectory appears exactly once and is never resampled; Replay reconstructs each recorded state independently, whereas Rollout starts from the initial context and proceeds with environment‑accepted actions.

All baselines and prompting‑only baselines share the Qwen3.5‑2B backbone (32 k‑token context) to isolate differences in prompting, memory, observation modality, and post‑training strategy from model scale.

Prompts are standardized across methods, and Qwen3.5‑397B generates a short first‑person rationale for each recorded action, which is paired with the structured action as the SFT target; GUI images are limited to 200,704 pixels.

For OPTD we perform an SFT warm‑up on the interaction format, freeze the checkpoint as the teacher, and initialize the student from it; training uses ROLL for distributed training, vLLM for on‑policy generation, and Megatron‑Core for student optimization and teacher inference.

Training runs in BF16 with tensor and sequence parallelism and activation recomputation; Replay uses greedy decoding and Rollout samples with temperature $1.0$, top‑p $0.95$, capped at 512 tokens per decision.

Rollout is limited to $31$ actions for single‑session evaluation and $42$ for cross‑session, one more than the maximum recorded count; trajectories reaching the limit are marked truncated.

All trainable baselines and OPTD use a peak learning rate of $2\times10^{-8}$; GRPO baselines employ a rollout batch of $64$ samples (eight responses per prompt) with KL‑controller coefficient $0.001$, KL loss $0.01$, temperature $0.99$, and top‑p $0.95$.

OPTD optimizes the student with Adam (zero weight decay), linear warm‑up and decay; it sets Jensen–Shannon mixing weight $β=0.5$, grounding weight $γ=0.1$, and a trust‑region with $λ_{\text{max}}=1$ and $ε=0.02$, approximating $λ^{\star}$ via eight bisection iterations (interval width ≤ $1/256$).

Table 5 defines the OPTD prompt: the student receives only information available at the current decision—user context, interaction history, current environment, and executable actions—while trajectory‑level intent is appended solely to the frozen teacher during training.

The shared prompt includes a system instruction, user context (profile, recent purchases, recent clicks up to 20 titles with duplicates collapsed), interaction history (last $10$ actions for single‑session, $20$ across visits), current screenshot and page description, and a JSON list of available actions with targets and scrolling range.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers