The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads

Yushi Sun, Yanjie Zhang, Rui Sheng

Personalized LLMs fabricate nearly half of their user-specific claims, and their self-confidence is a misleading safety signal.

Why do personalized LLMs fabricate user attributes that were never provided, and how can we measure this "over-inference" behavior?

Personalized LLMs with persistent memory often assert facts about users that were never communicated, creating a "mirage" of personalization. This over-inference (OI) is distinct from factual hallucination, as it involves fabricating individual-level attributes rather than world knowledge. The authors introduce MirageBench, a benchmark that evaluates 12 models across 143,616 claims using an independent judge to distinguish grounded evidence from stereotypes and fabrications. Every tested model over-infers between 35% and 49% of its personalized claims, with models primarily inventing attributes rather than relying on stereotypes.

Paper Primer

The core mechanism of MirageBench is a four-way faithfulness taxonomy—Grounded, Reasonable, Stereotype, and Fabricated—that adjudicates claims based strictly on a sparse set of three revealed user facts. By fixing the evidence at this early-interaction threshold, the benchmark forces models to either admit ignorance or engage in unwarranted inference.

Over-inference is a universal failure mode across all tested model families.

Across 12 models and 7 families, the cross-model mean over-inference rate is 41.6%, with no model falling below 35%. Nearly 42% of all personalized claims are unsupported by user evidence.

Self-monitoring is inversely correlated with actual faithfulness at the model-selection level.

Models that report the lowest over-inference in their own self-audits are often the most prone to fabrication when measured by an external judge (Spearman $\rho$ = -0.60). Self-reported confidence is a misleading cross-model safety comparator.

In multi-turn interactions, this over-inference accumulates linearly (R² > 0.90 for 9 of 12 models), with top-performing models rarely revising their initial fabrications. This creates "silent memory pollution" where unsupported claims persist and compound over time.

Why does this problem matter if the models are just "guessing" to be helpful?

Because these systems treat inferred attributes as persistent memory, they build a "mirage" of the user that influences all future interactions. When models fail to revise these fabrications, they effectively lock the user into a profile based on invention rather than reality.

Is self-auditing completely useless for developers?

No; while self-audit is misleading for comparing different models, it remains a moderate-to-strong signal (AUROC 0.58–0.83) for ranking a single model's own claims. It is useful for internal filtering, but dangerous as a cross-model safety metric.

The Personalization Mirage

Personalized LLMs often fabricate user traits beyond the evidence they were given.

Personalized language models with persistent memory are deployed widely, yet they routinely generate user‑specific claims that were never supplied. This mismatch between provided evidence and model‑inferred persona is the core problem this work isolates.

The model invents personal details that were never mentioned, presenting them as if they were known.

Across 12 evaluated models, 35 %–49 % of personalized claims suffer from OI, and the models’ own self‑audit scores are inversely correlated with external judgments—a phenomenon we call the Self‑Monitoring Inversion.

**Figure 1.** Overview of over-inference in personalized LLMs. Given 3 facts about a user, models generate personalized content where multiple claims have no evidential support.

The critical gap is between what users actually tell the system and the persona the model infers on its own.

Defining Personalized Memory

Defines the evaluation setup and over‑inference metric for personalized LLMs.

We formalize personalized LLM evaluation by treating the user’s disclosed facts as evidence $E$ and the model’s output as a set of claims $C$, then measuring how many claims exceed what $E$ justifies.

The OI Rate is simply the fraction of a model’s user‑focused statements that are either stereotyped or outright fabricated, i.e., not backed by the user’s supplied evidence.

How does OI Rate differ from a plain error‑rate on factual statements?

Error‑rate counts any false statement, regardless of whether it contradicts user evidence, while OI Rate specifically isolates statements that over‑generalize or invent user attributes absent from $E$.

The Gap quantifies the discrepancy between what an external judge deems over‑inferred and what the model itself reports after a self‑audit.

The MirageBench Benchmark

MirageBench measures over‑inference with sparse evidence, realistic tasks, and a multi‑stage pipeline.

MirageBench is a large‑scale benchmark designed to surface over‑inference by providing sparse evidence, realistic personalization tasks, and a four‑stage evaluation pipeline.

It assembles a controlled set of user personas, a suite of tasks, and a pipeline that first asks the model what it can infer, then lets it act, and finally judges every claim against a four‑way taxonomy.

Why does MirageBench fix the number of revealed facts to exactly three?

Three facts are enough to ground obvious personalization (e.g., occupation) while leaving a substantial set of attributes unmentioned; this creates a clear gap where a model must either infer responsibly or fabricate, making over‑inference observable.

The taxonomy classifies every model‑generated claim according to how well it is supported by the revealed facts $E$ and whether it relies on demographic stereotypes.

How does a “Stereotype” claim differ from a “Fabricated” claim?

A Stereotype claim draws on population‑level associations (e.g., assuming a nurse likes yoga), whereas a Fabricated claim invents an attribute with no statistical or evidential basis at all (e.g., saying the user owns a yacht when no hint exists).

**Figure 2.** The four-way claim taxonomy. The bottom two categories jointly constitute over-inference.

**Figure 3.** The MirageBench evaluation pipeline. From the benchmark input (personas with profile $P$ and revealed facts $E$, and the six-task suite), the three instruments Probe, Task, and Accum elicit explicit, implicit, and continual inference, respectively, and an independent Judge classifies every resulting claim under the four-way taxonomy.

Probe: ask the model “list everything you can infer from the three facts” to obtain an explicit self‑aware inference baseline.

Task: run the six personalization tasks using only $E$, then have the model audit its own response by extracting and classifying each claim.

Judge: an independent Claude‑Opus‑4‑7 model classifies every extracted claim under the four‑way taxonomy; validation against human annotators yields 89.8 % agreement ($\kappa$ = 0.863, macro‑F1 = 0.896) and 95.0 % agreement on the binary over‑inference decision ($\kappa$ = 0.900).

Accum: repeat the interaction for eight rounds, updating a structured memory profile and measuring whether inferred attributes grow monotonically or are revised.

Over-Inference is Universal

Evaluation reveals pervasive over‑inference across models and tasks, with systematic patterns and unreliable self‑audit.

The evaluation quantifies over‑inference (OI) across twelve personalized LLMs using the MirageBench benchmark.

**Table 1.** MirageBench leaderboard: over-inference rates as assessed by JUDGE (Claude-Opus-4-7) on 150 personas × 6 tasks. Each model's rates are percentages of that model's total claims (a per-model micro-average). The Mean row is the unweighted arithmetic mean across the 12 models (a cross-model macro-average); the corresponding claim-weighted micro-average OI over all 143,616 claims is 41.8%. Models sorted by OI rate descending.

Every tested model over‑infers more than 35 % of its personalized claims, with a cross‑model mean OI of $41.6\%$ (macro‑average) and $41.8\%$ (claim‑weighted micro‑average).

Table 1 shows OI % values ranging from $39.4\%$ to $48.7\%$, yielding the reported means.

Fabricated claims dominate over stereotypes, averaging $31.1\%$ versus $10.5\%$ of a model’s output.

The “Fabric” and “Stereo” columns in Table 1 give mean values $31.1\%$ and $10.5\%$ respectively.

Counter‑stereotypical personas incur $7.8$ pp higher OI than stereotypical personas, a gap present in all twelve models.

Aggregating Table 1 across persona groups yields $44.8\%$ OI for stereotypical versus $37.0\%$ for counter‑stereotypical personas.

**Table.** Model performance comparison across Self%, Judge%, $\Delta$, and Pattern.

Self‑audit OI correlates negatively with external Judge OI (Spearman $ρ=-0.60$, $p=0.044$), indicating systematic inversion across models.

Table 2 reports the correlation coefficient and its significance.

Within‑model self‑audit serves as a moderate to strong signal: AUROC ranges from $0.58$ to $0.83$, with nine of twelve models exceeding $0.75$.

AUROC values are computed per model from the self‑audit OI versus Judge OI labels (see Table 2 analysis).

Task‑dependent OI varies widely, from $57.8\%$ on the “apartment/home” task down to $27.0\%$ on “birthday‑gift” recommendations.

Table 3 lists OI % per task, showing the highest and lowest values.

Attribute accumulation grows approximately linearly, with top models adding up to $15.2$ inferred attributes per round.

Growth columns in Table 4 show per‑round slopes ranging $5$–$15.2$ attributes.

Revision rates differ dramatically, from near‑zero for high‑accumulating models to $82\%$ for Qwen3‑8B, highlighting divergent memory behaviours.

Table 4 reports revision percentages per model, with the highest at $82\%$.

Why Models Over-Infer

Why models over‑infer and how to diagnose the causes.

Recall the paper’s core claim: personalized LLMs tend to fabricate user attributes that lack evidential support because they favor verbosity and stereotypical priors over factual grounding.

Longer replies automatically create more statements, so even with a fixed three‑fact knowledge base the chance of adding an unsupported claim rises with output length.

Generate a 5‑token response: “I love coffee.” → 1 claim, all grounded.

Generate a 12‑token response: “I love coffee, I often visit the downtown library, and I enjoy weekend hikes.” → 3 claims; the second and third extend beyond the three known facts, creating two over‑inferred statements.

Count over‑inferred claims: 0 / 1 (0 %) for the short reply vs 2 / 3 (≈ 67 %) for the longer reply.

The example shows how simply increasing token count multiplies the opportunity for ungrounded assertions, even when the underlying knowledge base is unchanged.

Beyond verbosity, two additional mechanisms drive OI.

When the model lacks concrete evidence, it falls back on statistical knowledge learned during pretraining, inserting plausible but unverified details.

Certain output genres (e.g., recommendation letters) implicitly require the model to produce content even when evidence is absent, pushing it toward invention.

A third, more subtle pattern emerges when we look at self‑audit signals.

Models that report higher internal OI tend to exhibit lower judged OI across different model families, yet within a single model the self‑audit ranking remains useful.

These findings have concrete design implications for personalized AI systems.

Self‑audit can be used to rank a model’s own statements for internal filtering, but raw self‑reported OI should never be used as a cross‑model safety metric.

Every stored user datum should carry a tag indicating whether it is directly stated, inferred with evidence, or generated without evidence.

Only about a quarter of personalized claims are grounded; the rest are inferential, so eliminating over‑inference would also erase most personalization.

Our three‑layer framework (Probe, Accum, Task) reveals a behavioral dishonesty: models hide their uncertainty during generation.

**Table 7.** JUDGE OI% by stereotype group, per model. $\Delta$ is stereotypical minus counter-stereotypical. All 12 models show $\Delta > 0$.

In sum, MirageBench exposes a pervasive over‑inference problem, and our analysis shows that while self‑audit offers a useful intra‑model filter, cross‑model safety requires external verification and robust provenance.

Limitations and Validation

We outline the evaluation constraints, scoring design, and open challenges of our over‑inference analysis.

Our OI metric is based on a single judge model, Claude‑Opus‑4‑7. We validate it against a blind human annotator on 400 stratified claims, achieving 89.8% / 95.0% agreement ($\kappa$ = 0.863 / 0.900). Because only one annotator is used, this measures agreement with a human judge rather than inter‑annotator reliability, and we do not report a confidence interval for $\kappa$.

The judge inspects the ground‑truth profile $P$ solely to flag contradictions, while the taxonomy label is derived from the revealed facts $E$ alone. Nonetheless, we cannot fully exclude subtle influence of $P$ on borderline decisions.

Our most consequential findings are relational: Self‑Monitoring inversion relies on cross‑model ranks, the imagination gradient on task‑level ordering, and accumulation trends on per‑turn counts. These patterns should transfer across judges applying the same rubric, but a cross‑judge robustness check using an independent frontier model and additional annotators is left for future work.

Both judge and annotator condition only on the evidence $E$, using the task context merely as framing, which makes the agreement a fair test of taxonomy reliability. A side effect is that task‑presupposed claims count as over‑inference, yielding an upper‑bound estimate for OI on presupposition‑heavy tasks.

The Self‑Monitoring inversion analysis is exploratory, based on only $n=12$ correlated model families and a bootstrap confidence interval that spans zero ([-0.90, +0.06]). Within‑model signals show AUROC between 0.58 and 0.83, using separate claim extractors so correlations are ordinal rather than exact matches.

Our setup uses two personas and a memory prompt that favors retention over pruning, biasing results toward accumulation. Establishing the true dynamics will require larger samples and prompts that permit pruning.

We characterize the phenomenon but do not evaluate specific mitigations or their impact on recommendation quality or user satisfaction.

**Table 11.** Judge–human agreement on 400 stratified claims.

Memory Systems and Benchmarks

We situate our work among memory systems, benchmarks, and prior analyses of personalization failures.

Commercial memory systems such as OpenAI’s 2024 release and open frameworks from Chhikara et al. 2025, Packer et al. 2023, Sun et al. 2026a, Yan et al. 2026, and Sarin et al. 2025 let large language models store and retrieve user information across sessions.

Benchmarks such as those introduced by Jiang et al. 2025b, Liu et al. 2026, Yang et al. 2024, Wu et al. 2024b, and Jiang et al. 2025a evaluate how accurately models remember, track, and apply stored user information.

MirageBench, introduced in §3, measures the orthogonal question of how faithful the stored information is to what the user has actually revealed.

Over‑personalization, faithfulness issues, and hallucination constitute a class of personalization‑induced failures that arise when models prioritize verbosity and stereotypical associations over factual grounding.

Hallucination benchmarks (Bang et al. 2025; Luo et al. 2024) target factual and dialogue‑level correctness with respect to world knowledge or grounding sources.

The retrieval‑and‑application side of memory research categorizes irrelevance, sycophancy, and repetition‑style failures (Hu et al. 2026) and shows that user history can distort factual QA and emotional reasoning (Sun et al. 2026b; Fang et al. 2026).

In contrast, we study the generation‑and‑storage side: the fabrication of user attributes that were never observed.

LLMs exhibit low explicit but high implicit bias; stereotype‑driven user inferences persist even against explicit disclosures (Zhao et al. 2025; Li et al. 2025; Neplenbroek et al. 2025; Kantharuban et al. 2025; Wang et al. 2026).

Introspection signals can be surfaced under audit or via adapters (Lindsey 2026; Pandey 2026; Shenoy et al. 2026), providing a partial view of model self‑knowledge.

Models are systematically overconfident, and confidence estimates are further biased by persona and RLHF‑amplified sycophancy (Chhikara 2025; Xu et al. 2025; Shapira et al. 2026).

Write‑time filtering and temporal‑validity checks have been proposed to curb error propagation across turns and domains (Xiong et al. 2026; Ma et al. 2026; Koch et al. 2025; Chao et al. 2026).

While STALE targets temporal validity of once‑observed facts, MirageBench targets evidential validity of never‑observed inferences, sharpening the evaluation line.

Our Self‑Monitoring Inversion analysis shows that self‑audit tracks over‑inference moderately well within a model, yet between models the ones reporting the least over‑inference commit the most.

Task-Specific Over-Inference

Breakdown of Over‑Inference rates across tasks, claim counts, and model‑level signals.

This section quantifies where Over‑Inference (OI) occurs: by task, by claim volume, by model‑self‑audit, and by persona stereotype alignment.

**Table 5.** Judged-claim counts per task, pooled across 12 models and 150 personas. Composition percentages and OI rates for these tasks are given in Table 3.

The table presents performance metrics for various models, including Spearman $\rho$, AUROC, and the number of samples ($n$).

Inference Accumulation Over Time

We quantify how inferred attributes accumulate across conversation rounds.

It fits a straight line to the number of attributes a model stores after each conversation turn, revealing how quickly memory grows and how often entries are overwritten.

How does Accumulation Regression differ from simply counting total attributes at the end of a conversation?

Counting only the final total ignores the dynamics of how attributes are added and removed each turn. Accumulation Regression captures both the rate of addition (slope) and the rate of attrition (removal‑rate), revealing whether memory growth is steady, bursty, or largely replaced.

**Table 8.** Per-model Accum regression (attributes vs. round, $n = 16$) and mean per-round removal rate. Removal rate is the fraction of unique attributes present at round $T$ that are absent at round $T+1$, averaged across the two personas.

Formal Metric Definitions

Defines the quantitative metrics used to evaluate over‑inference and related phenomena.

This section formalizes the metrics used throughout the study to quantify over‑inference, grounding, and the dynamics of attribute accumulation.

Aggregating OIJudge across models can be done in two ways: an unweighted mean of per‑model rates (macro‑average, reported as 41.6 %) and a claim‑weighted micro‑average (41.8 %). The two numbers coincide because each model contributes a similar number of claims.

The complementary Revision Rate measures how often attributes are removed or corrected between consecutive rounds; it is reported as a fraction of attributes revised per round (see Appendix F).

Experimental Prompts

All prompts that drive personalization, evaluation, and memory updates are presented.

This section enumerates the prompts that define the personalization task, the unified judge used to detect Over‑Inference, example personas, and the mechanisms for self‑audit and memory accumulation.

System prompt prefix used for all six personalization tasks

Six task prompts presented to the model

Unified Judge Prompt – extracts claims and classifies Over‑Inference

Persona examples used for evaluation

Over‑Inference examples by category

Probe Prompt – explicit inference elicitation

Task Self‑Audit Prompt – model audits its own personalized output

Accum Memory‑Update Prompt – updates structured memory each conversation round

Human Validation Details

How the study’s human judges measured and categorized model outputs.

Human annotators were presented with model‑generated persona responses and asked to label each utterance as Grounded, Reasonable, Stereotype, or Fabricated. The labeling interface displayed the original user prompt, the model output, and a brief definition of each category to keep judgments consistent across annotators.

**Table 9.** Model versions used in the study. Snapshots frozen at experiment time.

The aggregated judgments are summarized in a 4 × 4 confusion matrix (Table 10), where rows correspond to the ground‑truth category and columns to the annotator’s label. Diagonal entries indicate agreement, while off‑diagonal cells reveal systematic misclassifications such as Stereotype being labeled as Reasonable.

The table presents a confusion matrix comparing four categories: G (Grounded), R (Reasonable), S (Stereotype), and F (Fabricated). The rows represent the ground truth or primary classification, while the columns represent the secondary classification or judge's assessment. Each category has a total of 100 items, resulting in a grand total of 400 items.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers