Realcompanion: Benchmarking Human Understanding from Reasoning Over Longitudinal Real-World Conversations

Arman Behnam, Sunglyoung Kim, Liangwei Yang

RealCompanion provides a longitudinal benchmark of real human-AI conversations to measure memory and persona reconstruction.

How can we reliably measure whether an AI companion actually remembers and understands a user over months of real-world conversation?

AI companions are evaluated on synthetic conversations that force memory usage, failing to reflect how rarely people actually reference the past in real, long-term interactions. RealCompanion releases ten real, anonymized human-AI relationships, providing a ground-truth dataset where every label is derived from the conversation and verified by a reasoning trace. The data reveals that memory is required in only 3.4% of messages, and when it is, the needed information is typically thousands of messages in the past.

Paper Primer

The authors define the task over two core challenges: identifying who the user is (persona reconstruction) and determining when the past is relevant (memory retrieval). Their method hinges on a multi-stage derivation pipeline—Verification, Classification, and Grounding—that forces every label to point to specific source messages, allowing for an auditable reasoning trace.

Memory demand is significantly lower and more distant than synthetic benchmarks assume.

In a random sample of real messages, only 3.4% required information from outside the current thread, with a median distance of 2,157 messages to the required information. Synthetic benchmarks often assume 100% memory demand, whereas real-world usage shows a demand rate near 3%.

Current AI systems struggle to distinguish between relevant and irrelevant past information.

When models are shown the same context labeled as "memories" versus "earlier messages," they retrieve past information 10–14 percentage points more often, even when nothing is needed. Retrieval detectors barely beat chance (AUROC ~0.53–0.55) at identifying when the past is actually required.

Why does this paper use real conversations instead of the generated ones common in the field?

Generated conversations are scripted to force memory usage and declare personas in advance, which masks the difficulty of identifying when the past is actually relevant and prevents testing how a persona emerges organically over time.

What is the scope of the reconstruction track?

It evaluates whether an AI can read a full history and accurately write a profile and persona file, with performance measured by F1 scores against the derived ground truth.

Researchers should shift from testing memory on synthetic, high-density prompts to evaluating systems on their ability to exercise restraint—knowing when to ignore the past—in long-term, sparse-memory environments.

Introduction: The Challenge of Longitudinal Memory

RealCompanion exposes the long‑term memory gap in AI companions by providing a true multi‑month conversation benchmark.

Current AI‑companion benchmarks rely on synthetic dialogs that embed the memory they test, obscuring how real users actually reference the past. RealCompanion fills this gap by releasing ten authentic, consented conversations spanning up to 120 days, together with exhaustive annotations of who the user is, what they have said, and when past utterances matter.

Think of the benchmark as a diary that records every exchange between a person and an AI companion, then tags each entry with who the person is, what facts were stated, and which earlier lines are needed for a correct reply.

Prior models typically attend only to the last 10 messages; they miss turn 2843 entirely.

The model generates a generic reply (“Good luck!”) without acknowledging the user’s lingering anxiety.

Human annotators label the turn as requiring the distant context, but the model’s response is marked incorrect.

The benchmark exposes a concrete failure: existing detectors cannot identify when a far‑back message is essential, leading to missed opportunities for truly personalized assistance.

**Figure 1.** A user’s messages to an AI companion, and what the companion built from them. Reply A uses a persona line the companion got wrong, so it congratulates the user on a job they were unsure about. Reply B uses only the profile, so it repeats what the user already said. Reply C uses a correct persona line and the profile, so it speaks to why the user applied.

The core gap is that short‑term chat alone cannot capture the long‑term understanding needed for consistent, personalized AI companions.

Background and Dataset

Describes prior memory benchmarks and introduces the RealCompanion dataset with its composition.

Prior work on conversational memory converges on a simple shape: construct a long history, pose a final question, and score the system on retrieving the required fact. Benchmarks such as LoCoMo, LongMemEval, CloneMem, CUPID, and HorizonBench expose the difficulty of selecting relevant context, often capping precision well below 50 %.

Companion‑focused evaluations (ANCHOR, CompanionBench, AlpsBench, PRISM, OmniBehavior, LUNAR, PersonaChat, PersonaMem) further highlight gaps: even with real‑world conversations, models struggle to maintain consistent persona traits or recall user state, often hovering near chance performance.

**Table 1.** Memory and personalization benchmarks. Filled circles mark yes, open circles no, and half-filled not applicable.

The RealCompanion dataset records ten months‑long AI companion conversations, preserving every message after anonymization. Each participant’s span, active days, and message counts are detailed in Table 2, showing that two relationships dominate the corpus with 73 % of all messages.

**Table 2.** Per-participant composition.

Constructing the Ground Truth

How the benchmark’s ground‑truth data are built and how experiments vary key factors.

The ground‑truth corpus is assembled by a fully auditable five‑stage pipeline, and the benchmark varies retrieval method, context condition, and evaluation metric to isolate memory effects.

Each label is produced by a deterministic chain of five stages that read only the output of the previous stage and write both a decision and its rationale.

Stage A: the referent is identified as “the product mentioned in M2”; locus = message 2.

Stage B: candidate records are the product description and a receipt; both exist in the source, so they are kept.

Stage C: rule #7 (“requires recent factual detail”) fires, assigning tier = intermediate.

Stage D: the reference reply is “You bought the X‑200 blender last week”; `gold_response` cites M2.

Stage E: verification confirms the reply mentions only information present in M2, so the item is retained.

The example shows how each stage’s output constrains the next, preventing invented evidence and guaranteeing that the final tier can be recomputed from the recorded lists.

How does this ground‑truth pipeline differ from a typical human annotation workflow?

Typical annotation asks humans to label items after looking at the whole conversation, which can introduce hindsight bias. Here each stage writes its decision and rationale before the next stage runs, and no later stage can alter earlier evidence, so every label is fully auditable and reproducible without human reinterpretation.

Define three tracks (Reconstruction, Chat, Question) and the corresponding evaluation metrics.

Select five retrieval methods (RANDOM, RECENCY, RECENCY‑USER, BM25, ORACLE) and three control variants (offset‑permuted, position‑randomized, broken‑oracle).

Run each method on each participant’s history, recording the top‑k retrieved messages for every probe.

Apply the context‑ablation conditions C0–C4, feeding the specified preceding messages to the generator.

Score retrieval by hit‑@k and MRR; score responses by content match, restraint, and evidence coverage.

Aggregate results per participant, compute macro‑averages, and bootstrap over participants for statistical significance.

**Figure 2.** How the chat ground truth is built.

Table 3 lists the three benchmark tracks, their input/output formats, and the number of scoreable items per track (e.g., the Chat track contains 1,533 items plus 434 cold‑open probes).

Benchmark Results

Supplying recorded messages boosts content match, while memory demand remains low and uneven.

Replacing the last three messages with recorded ones raises content match by +0.140 across both strata.

Measured on the full set of 1,477 chat probes.

A sequence of conversational turns that together form a coherent interaction thread, used to evaluate long‑term memory.

How does the Chat Track differ from a simple message‑pair evaluation?

Chat Tracks group messages into a longitudinal thread, so the system must retrieve information across many turns, whereas a pairwise evaluation only checks immediate adjacency.

**Table 5.** Content match by context condition.

**Table 6.** Persona reconstruction from the full history, pooled over the ten participants.

Performance varies widely across memory‑bearing strata, highlighting the need for targeted evaluation.

Evaluation Task Definitions

We detail the evaluation task, its data splits, scoring metrics, and validity conditions.

The central premise—that AI companions need longitudinal memory—is restated here: we evaluate how well a system retrieves information beyond a short recent window.

A subject record bundles a chronological message stream $M$ with a set of stores $S$ that a system may query; a probe is any person‑attributed message from that stream.

What exactly does the reference set $R(p)$ contain?

$R(p)$ lists the specific records (e.g., prior profile statements or earlier episode messages) that the gold response draws upon, and by definition none of those records appear in the recent window $W_k(p)$.

Four metrics quantify how well a system respects the annotation: restraint (false‑alarm rate), attribution, locus accuracy, and overall response consistency.

Why is the misfire rate used to evaluate restraint?

Restraint asks a binary “should we look beyond the recent window?”; the misfire rate captures the false‑alarm probability when the correct answer is “no,” directly reflecting unnecessary retrieval cost.

The demand rate $\pi = \Pr[\rho^{\star}(p)=1]$ is the probability that a probe truly needs information beyond its recent window, and it drives the difficulty of the restraint decision.

How does a low demand rate $\pi$ affect the interpretation of response scores?

Because $\pi$ is small, the overall response improvement $\Delta$ can be explained by either a large effect on the few retrieval‑required probes or by modest gains on the many non‑retrieval probes; without separating the two populations, $\Delta$ does not reveal the true memory benefit.

Collect a subject record $(M, S)$ for each participant.

Identify all person‑attributable messages as probes $p$ and compute their histories $H(p)$.

For each probe, extract the recent window $W_k(p)$ and construct the disjoint reference set $R(p)$.

Annotate the gold response $g(p)$ and verify the derivability axiom $g(p)$ from $(p, W_k(p), R(p))$.

Run the system to produce $\hat\rho(p)$, $\hat\lambda(p)$, and $\hat y(p)$.

Compute the four metrics (MR, AAR, LA, RC) on the appropriate probe subsets.

Aggregate results both pooled over items and macro‑averaged across subjects.

**Table.** Restraint is scored as a false-alarm rate, following the misfire convention of Yoon et al. (2026):

The table presents data across four rows categorized by "Stratum" (both, proportional) and "Reading" (recorded, strict). The columns include $\pi$, $\gamma_1$, $\gamma_0$, $\pi\gamma_1$, $(1 - \pi)\gamma_0$, and "From no-memory rows".

The table lists seven conditions (V1-V7) and their status in RealCompanion, along with the section where each is established.

**Figure 4.** Evidence coverage by context budget.

Related Work

Related work positions our benchmark among prior memory and personalization studies.

The companion locus is an episode reference limited to messages the companion sent, while a memory‑bearing probe is any turn that requires access to long‑term memory. The formal locus $\lambda$⋆ therefore takes a value in the two stores, and the released field refines it: thread and none carry $\rho$⋆ = 0, and companion refines episode. No item populates both reference lists, so |$\lambda$⋆(p)| = 1 on every memory‑bearing probe.

V5 requires that at least one released file be underived and that every derived claim cite that file; otherwise a trace cannot be checked. V6 mandates that refutation only demotes labels and that any demoted label resets all derived evidence, preventing contradictory probes. V7 obliges systems to report the rate at which evidence originates from the companion itself, because otherwise recall cannot be distinguished from repetition.

On the chat track, 22.0 % [18.3, 26.3] of the 404 memory‑bearing rows rest on memory evidence sent entirely by the companion, of which 65 rows have a verified companion locus. Episode‑level rows constitute 7.1 % [5.4, 9.2] of 339, and profile‑level rows 1.8 % [0.7, 4.5] of 223. On the question track the companion‑origin rate is 7.5 % [6.6, 8.5] of 3,165 items, or 5.5 % [4.8, 6.4] when restricted to cited turns.

A second reading counts grounding citations other than the probe, yielding 70.3 % [67.5, 73.0] of 1,045 rows that carry one. For thread rows this includes a gold that continues the companion’s own previous turn, i.e., the companion knows what it just said.

B.1 Three Substitutions describes how the benchmark replaces three natural components: the persona (a description of the user), the probe (the moment that needs memory), and the model’s judgment (the target). Personas originated in PersonaChat (Zhang et al., 2018) and were later extended to multi‑session settings (Xu et al., 2022). Industrial‑scale synthetic personas (Nemotron‑Personas‑USA, 2025) and studies showing demographic attributes explain only ~1.5 % of response variance motivate the substitution.

B.2 What the Substitutions Cost notes that authoring the probe removes three decisions, visible as gaps in the literature. Restraint means a benchmark built from questions cannot evaluate a system’s decision to look up information. Locus resolves where evidence lives, and Reach measures the distance between probe and evidence, which in real conversation spans a median of 2 messages (in‑thread) to 450 messages (out‑of‑thread).

B.3 Attempts to Remove Them surveys work that restores one substitution at a time. AlpsBench (Xiao et al., 2026) removes the first substitution while keeping synthetic probes; CompanionBench (Liu et al., 2026) removes the first substitution at collection but restores it at release; PRISM (Kirk et al., 2024) removes the third substitution by using human judgments on single‑session conversations.

B.4 Adjacent Work lists systems that implement the memory mechanisms our benchmark evaluates, such as MemGPT (Packer et al., 2023) which pages context, Generative Agents (Park et al., 2023) which maintains a recency‑importance stream, and AI Persona (Wang et al., 2024) which builds lifelong user profiles. These systems differ in whether they store raw messages (enabling identifier matching) or distilled statements (requiring a judged recall metric).

B.5 What Releasing the Alternative Requires discusses privacy and consent: releasing the full interaction preserves the exact provenance of each label, but it also creates a permanent record that cannot be regenerated if released incorrectly. Studies on user expectations and emotional dependence (e.g., Skjuve et al., 2021) highlight why such releases must be carefully controlled.

**Table 10.** Criteria behind each column.

B.6 The Comparison Table (Table 10) groups benchmarks by six criteria: how the underlying record was produced, whether it is longitudinal, how probes were generated, whether evidence is cited, whether abstention is scored, and whether derivations are recorded. These criteria help readers decide which benchmarks are comparable to our own and which measure distinct aspects of conversational memory.

Ground Truth Derivation Details

This appendix details the full ground‑truth derivation pipeline, its phases, sampling strata, labels, audits, and release checks.

This appendix expands the brief derivation sketch from Section 4, exposing every intermediate step, label, and quality check.

Phase A proposes a referent: record the raw user turn (`referent_text`) and the store (locus) it points to.

Phase B verifies each proposed reference against the source conversation; any evidence that fails is removed.

Phase C assigns a tier, a category, and a rule identifier (ruleC) based solely on the surviving context shape.

Phase D writes the gold response and the IDs of the turns that ground it (`grounding_dayids`).

Phase E re‑reads the probe, gold, and assigned category; if the item passes all checks it receives ok = true, otherwise it is withdrawn.

**Table 16.** What each phase writes.

Phase A is permissive—nothing has been checked yet—so its `referent_text` and locus may be wrong. Phase B is the first filter: it removes any reference that cannot be verified in the source.

**Table 17.** Sampling bucket against verified tier.

D.2 explains that probes are user turns drawn from three strata: proportional (unconditioned), enriched (memory‑bearing), and abstention (session openers). The bucket label is preserved throughout the pipeline for auditability.

Tier captures the shape of the context lists, Category refines the tier, and Locus states where the referent lives; only Locus depends on a judgment, the other two are deterministic.

**Table 18.** Chat categories by tier, scoreable items

**Table 19.** Verified locus of every scoreable chat item.

D.4 introduces on‑screen verdicts (clear, borderline, not on screen) that classify whether the recent window already contains the needed referent.

**Table 20.** On-screen verdicts by provenance.

**Table 21.** Memory-bearing rows under each reading.

D.5 details three failure modes—withdrawal, demotion, and repair—each recorded in the released files so that no error is silently hidden.

**Table 22.** Withdrawn rows by reason.

D.6 describes an independent audit performed by a model that never saw the data, followed by manual verification of flagged rows.

**Table 23.** Independent audit of the chat track.

D.7 lists 26 release gates (Table 24) that enforce internal consistency, plus five check families (Table 25) that validate referents, categories, structure, and arithmetic.

**Table 24.** Release gates, all passing.

**Table 25.** Check families.

D.8 introduces the question track (3 312 items) and quantifies the impact of three successive repairs on content‑defect rates.

**Table 26.** What each repair bought, same items, pooled percent.

**Table 27.** Category agreement between the classifier and the auditor.

Evaluation Protocol

Defines the experimental protocol, context ablation conditions, retrieval controls, baselines, judge, metrics, and models.

The appendix formalizes the evaluation pipeline that produced the numbers in the main experiments. It specifies every variable that was varied, held constant, and measured.

Context Ablation isolates the impact of each memory source by swapping the block of messages shown to the generator while keeping the generator, temperature, and prompt shape identical.

How is Context Ablation different from simply removing the context block?

Removing the block eliminates both the content and the heading, conflating two variables. Context Ablation swaps only the block’s content while preserving the heading (or vice‑versa) so the experiment can attribute performance changes to the specific memory source rather than to a generic “no context” effect.

The block is assembled: heading “RETRIEVED MEMORIES” followed by the three truncated messages.

The generator, at temperature 0, produces a reply using only the supplied block.

The response judge evaluates the reply, recording a high “`used_memory`” score because the reply mentions “July 5th”.

Even though the same three messages appear in C1 (heading “RECENT CONVERSATION”), the different heading changes the judge’s “`used_memory`” by roughly 0.12, demonstrating that headings themselves influence memory attribution.

Instantiate a single generator model with temperature 0 and a fixed prompt template.

For each probe, construct the five context blocks defined in Table 28 (C0–C4, C3m, C4m).

Apply each of the three retrieval controls from Table 29 (offset‑permuted, broken oracle, 20 pool messages) to the gold set of messages for the probe.

Run the four retrieval baselines (RANDOM, RECENCY, RECENCY‑USER, ORACLE) and the BM25 scorer on the same pool.

Feed the five candidate replies (one per context condition) to the response judge (temperature 0) and record the three fields: `content_match`, `used_memory`, restraint.

Compute the six quantities listed in Table 30 (misfire rate, appropriate application, `used_memory`, `content_match`, detectability, restraint) from the judge’s outputs.

Assign the generator, judge, and independent auditor roles to the models shown in Table 31, all at temperature 0, to ensure reproducibility.

**Table 28.** The five context conditions and the matched pair.

**Table 29.** Retrieval controls.

**Table 30.** How each quantity is computed.

**Table 31.** Models by role.

Additional Results and Demand Rate

Additional results quantify demand rate, retrieval, coverage, and system performance across multiple metrics.

$\pi$ measures how often a conversation turn needs long‑term memory access.

Demand‑rate estimates vary between 0.83 % and 3.4 % across the three procedures.

Table 32 reports 3.4 % for the natural‑rate stratum and 0.83 % for the sweep‑re‑read procedure.

All three procedures operate on the same corpus of 12,795 user turns, differing only in how they select candidate turns.

**Table 32.** Demand rate as a percentage of user turns.

Retrieval performance collapses to chance when probe positions are randomized.

Table 33 shows recency over user turns drops from 0.092 to 0.000 under the randomised gold set. 0.019 % drop between ordered and randomised conditions

The three retrieval methods (Random, Recency, BM25) are evaluated against the same set of substituted gold turns.

**Table 33.** Retrieval under substituted gold sets.

Window size strongly influences coverage: a 1 M‑token window achieves full coverage of memory‑bearing probes.

Table 34 reports a share of 1.000 for the 1 M‑token window across all three populations. 0.891 % increase from the smallest to the largest window

All windows are evaluated without any model or retriever; they simply check positional inclusion.

**Table 34.** Share of memory-bearing probes whose every required turn lies in the window. Populations are 404, 250 and 205. Tokens are counted with `cl100k_base`.

Exact context (C3) yields the highest response score on memory‑bearing strict probes.

Table 35 records a score of 1.822 for C3, surpassing all other conditions. 0.412 point improvement over the next best condition

All conditions share the same underlying conversation data; only the amount of surrounding context differs.

**Table 35.** Response score by condition.

Condition C2 improves over C1 by +0.149 on the whole‑corpus contrast.

Table 36 shows the C2 − C1 contrast as +0.149 [+0.107, +0.201].

The paired contrasts isolate the contribution of each condition by subtracting the preceding baseline.

**Table 36.** Paired contrasts, bootstrapped over subjects. Whole-corpus rows do not vary with the reading, which affects only the memory-bearing populations.

Selectivity improves: appropriate‑application rises to 59.2 % while misapplication drops to 50.8 % under condition C2.

Table 37 reports 59.2 % appropriate application and 50.8 % misapplication for C2.

All five conditions are evaluated on the same set of probes; only the context supplied to the model varies.

**Table 37.** Selectivity by supplied context.

BM25 achieves a macro recall of 0.424, far above the random baseline.

Table 38 shows BM25’s macro recall as 0.424 compared to 0.004 for Random.

Recall is computed as the proportion of required turns that appear within the top five retrieved results.

**Table 38.** Required turn in the top five, by subject, recorded reading.

Condition C2 reaches a perfect 1.000 score for some subjects, while C1 attains a macro of 0.361.

Table 39 lists a 1.000 value for C2 on subject U01 and a macro of 0.361 for C1.

The tables contrast the effect of supplying context versus withholding it across multiple subjects.

**Table 39.** Use of supplied context, by subject. C0 is zero in every cell.

Content‑match improves by +0.148 % on the macro average when moving from C1 to C2.

Table 40 reports a macro contrast of 0.148 [0.109, 0.194] for C2 − C1.

Both macro and pooled averages are reported to show consistency across subjects and across all judged rows.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers