LightMem-Ego: Your AI Memory for Everyday Life

Yijun Chen, Boyi Xiao, Yixian Zhao, Haoting Xia, Buqiang Xu, Jizhan Fang, Yanya Li, Yaqi Zheng, Xuehai Wang, Zirui Xue, Liuxin Zhang, Hui Li, Ningyu Zhang

A streaming multimodal memory system that organizes egocentric life-logs into a hierarchical store for retrospective AI assistance.

How can we build a lightweight, multimodal memory system that allows mobile and wearable AI assistants to accurately retrieve and reason about past daily experiences?

Personal AI assistants struggle to recall past experiences because they lack a persistent, organized way to store continuous visual and audio streams from wearable devices. LightMem-Ego segments these raw streams into events and routes them into a three-level hierarchy—current, short-term, and long-term memory—allowing the system to retrieve specific evidence based on the query's temporal scope. The system achieves a 74.1% recall rate on top-3 memory retrieval across diverse daily scenarios, enabling grounded responses for tasks like object finding and conversation recall.

Paper Primer

LightMem-Ego functions as a memory-augmented backend for mobile and wearable clients. It treats the user's day as a continuous stream of visual frames and audio chunks, which it partitions into discrete "micro-events" based on temporal continuity and visual change signals.

The core mechanism is a hierarchical memory router: it acts like a librarian who directs requests to different shelves—the "current" desk for immediate scene questions, the "short-term" bin for recent events, or the "long-term" archive for consolidated routines and episodic facts.

The system effectively retrieves relevant personal experiences from continuous streams.

Across object finding, conversation recall, and life summarization, the system achieved an overall Recall@3 of 74.1%.

The system provides usable memory-grounded answers for retrospective queries.

Human-judged accuracy for experience-based question answering reached 55.6% overall.

Why is a hierarchical memory structure necessary instead of just using a large context window?

A flat context window cannot scale to continuous, long-term egocentric streams without massive compute costs; the hierarchy allows the system to route queries to the cheapest sufficient memory level, keeping wearable interaction responsive.

What is the primary trade-off in the current system's performance?

While short-term memory queries are near-interactive (P50 latency ~6–7s), long-term retrospective queries require significant evidence aggregation, pushing P50 latency to ~15–20s.

The system's reliance on external APIs for vision and language models means that both latency and accuracy are currently tethered to the performance and rate limits of those upstream providers.

LightMem-Ego demonstrates that personal AI can move beyond stateless interaction by explicitly managing a persistent, multimodal record of lived experience.

Introduction and Motivation

We expose why continuous egocentric streams demand a hierarchical memory for efficient retrieval.

Continuous egocentric perception produces a relentless stream of visual and audio observations, yet existing large‑language‑model context windows cannot ingest or reason over such unbounded multimodal data.

The gap is the mismatch between (a) an ever‑growing, unsegmented egocentric data flow and (b) the fixed‑size, single‑window memory that current LLMs can attend to.

**Figure 1.** Overview of LightMem-Ego’s motivating scenarios and memory hierarchy. The system supports everyday memory assistance across object finding, conversation recall, life summarization, and routine discovery by routing user queries to current, short-term, and long-term memory.

Hierarchical memory is essential for streaming egocentric data because it lets a lightweight assistant retrieve evidence at the appropriate temporal scope without exhausting a single context window.

Related Work and Baselines

We situate LightMem‑Ego among prior conversational, wearable, and multimodal memory systems.

Prior work splits into three strands: conversational agents that retain dialogue context, wearable AI that perceives from a first‑person view, and systems that archive multimodal experiences for later retrieval.

These are AI services offered to end users (e.g., ChatGPT) that provide on‑demand language capabilities but typically lack long‑term, multimodal memory.

These systems augment language models with an external store that can be queried by text, enabling retrieval of facts beyond the model’s context window.

These agents ingest a continuous stream of first‑person video, audio, and sensor data, building a personal archive that can be queried later for context‑aware assistance.

**Table 4.** Capability comparison with representative commercial assistants, text-based memory systems, and egocentric multimodal assistants. The table compares publicly described system capabilities rather than experimentally measured performance. We mark a capability as supported only when it is implemented as an explicit first-class component. *Partial* indicates limited, implicit, offline, session-level, or modality-restricted support; — indicates that the capability is not explicitly supported or not publicly described.

LightMem-Ego System Architecture

LightMem‑Ego streams multimodal life data into a tiered memory that keeps on‑device interaction fast.

Continuous egocentric streams quickly outgrow a single LLM context, so a naïve “store‑everything‑in‑memory” approach stalls the user‑device.

LightMem‑Ego treats the phone or glasses as a thin sensor that timestamps raw video, audio, and auxiliary signals, then pushes them unchanged to a backend for later structuring.

How does this differ from a typical “record‑and‑store” video logger?

Standard loggers keep raw frames locally and later batch‑process them, which forces the device to buffer large amounts of data. LightMem‑Ego streams each sample immediately, discarding the raw payload after backend ingestion, so the device never accumulates a bulky buffer.

Each sample is paired with $\tau$: $(v_0,a_0,m_0,\tau=0)$, $(v_1,\tau=1)$, $(v_2,a_2,\tau=2)$.

The client compresses each frame to 50 KB and each audio clip to 5 KB, then uploads the three packets sequentially.

The backend receives the packets, stores them in a temporal buffer, and tags them with the same $\tau$ values for later alignment.

Because timestamps travel with the data, the backend can later fuse video, audio, and GPS without any extra synchronization step.

Memory is split into three layers—current, short‑term, and long‑term—so that cheap, fast updates stay near the device while expensive consolidation runs offline.

Why not store everything in a single flat buffer as some video‑log systems do?

A flat buffer forces every query to scan the entire raw stream, which is $O(T)$ in time and quickly exceeds LLM context limits. The hierarchical layout lets the router pick the smallest tier that can answer the query, reducing both latency and memory footprint.

$M_{\text{cur}}$ holds the last 3 seconds: $(v_7,v_8,v_9)$ and their audio.

After the walk ends, the first 7 seconds are grouped into one event $e_1$ and stored in $M_{\text{st}}$ with a representative frame $v_3$.

During nightly consolidation, $e_1$ is indexed, a textual description “walking in the park” is generated, and the entry is promoted to $M_{\text{lt}}$ as an episodic record.

The tiered design ensures that a query about “what did I see just now?” touches only $M_{\text{cur}}$, while a question about “where did I walk yesterday?” reaches $M_{\text{lt}}$ without scanning raw frames.

Parse the user query $q$ to extract temporal scope (e.g., “now”, “yesterday”) and semantic intent.

Choose the cheapest memory tier that can satisfy the scope: $M_{\text{cur}}$ → $M_{\text{st}}$ → $M_{\text{lt}}$.

Retrieve the top‑$k$ event records $R(q)$ from the selected tier using a similarity search over stored embeddings.

Fuse the retrieved multimodal evidence into a compact view $E_q$ (frames, transcripts, metadata).

Pass $q$ and $E_q$ to the language model $f(\cdot)$ to generate the final answer $\hat{y}$.

Routing and retrieval pipeline – high‑level pseudocode.

**Figure 2.** Interfaces of LightMem-Ego across web and wearable deployments. The web client visualizes live multimodal capture and retrieved evidence; the glasses client app provides a lightweight interaction surface; and the first-person overlay illustrates how memory-grounded responses are presented in the user's egocentric view.

By streaming raw multimodal samples, segmenting them into events, and organizing those events into a three‑tier memory, LightMem‑Ego keeps on‑device interaction snappy while still offering rich, long‑term recall.

Demonstration Scenarios

Shows LightMem‑Ego handling immediate, conversational, and long‑term memory tasks.

LightMem‑Ego supports three distinct memory needs—immediate assistance, conversation recall, and life summarization—within a single unified system.

Figure 3 illustrates each scenario with concrete user queries and retrieval outcomes.

**Figure 3.** Representative demonstration scenarios of LightMem-Ego. The system supports immediate assistance, conversation recall, life summarization, and routine discovery by retrieving evidence from hierarchical memory.

The system’s versatility lets it satisfy both instant assistance and long‑term habit reasoning without separate pipelines.

Quantitative Evaluation

Quantitative results of LightMem‑Ego on retrieval, QA accuracy, and latency.

Continuous egocentric perception overwhelms LLM context windows; LightMem‑Ego mitigates this by storing streams in a hierarchical memory for scope‑aware retrieval.

LightMem‑Ego retrieves relevant evidence within the top‑3 results for 74.1 % of queries and achieves an overall Mean Reciprocal Rank (MRR) of 0.627.

Table 1 reports retrieval results across three everyday scenarios, with overall R@3 = 74.1 % and MRR = 0.627.

**Table 1.** Memory retrieval accuracy across everyday-life scenarios. R@k denotes Recall@k, and MRR measures the reciprocal rank of the first relevant memory entry.

**Table 3.** Latency breakdown by memory scope and client. P50 / P90 are 50th / 90th percentile latencies.

Conclusion and Limitations

LightMem‑Ego organizes egocentric streams into hierarchical memory, enabling long‑horizon multimodal retrieval.

LightMem‑Ego bridges mobile and wearable multimodal capture with a hierarchical memory that supports question answering across time.

The system demonstrates four concrete capabilities: locating objects, recalling past conversations, summarizing daily life, and understanding routine patterns.

Our evaluation suggests that maintaining an explicit long‑horizon multimodal memory is a viable research direction for personal AI assistants.

LightMem‑Ego currently depends on external APIs for speech recognition, visual understanding, language generation, and retrieval‑based QA, so both accuracy and latency inherit the variability of those services.

Mis‑recognitions in transcripts, visual captions, or timestamp alignment can cascade into the memory store, degrading the faithfulness of downstream answers.

Continuously ingesting egocentric streams requires repeated segmentation, summarization, embedding, indexing, and storage, which adds noticeable compute and storage overhead.

The present memory‑update rule is rudimentary; it does not include a systematic policy for revising, merging, forgetting, or promoting memories as relevance changes.

Because the system records raw video and audio from a first‑person viewpoint, it inevitably captures private conversations, faces, documents, locations, and daily routines.

Persisting these streams—and their derived transcripts, summaries, embeddings, and long‑term semantic memories—creates privacy hazards if access is not tightly controlled.

The current prototype does not implement automatic redaction, by‑stander consent handling, fine‑grained access control, or robust retention and deletion policies.

LightMem‑Ego shows that hierarchical egocentric memory can power diverse personal‑assistant tasks, but its reliance on external services, lack of a mature memory‑lifecycle policy, and absent privacy safeguards constitute the primary failure modes to address.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers