LatentPress: Context Compression Beyond Text and Vision

Zhengze Zhou, Hejian Sang

LatentPress compresses long-context history into continuous soft tokens that a frozen LLM reads directly, bypassing text reconstruction.

How can we compress long conversational histories into compact soft-token memories that a frozen, unmodified language model can read directly?

Language models struggle to process long-running conversational histories or documents because the memory cost of raw text grows linearly, while existing compression methods rely on slow, error-prone text reconstruction or expensive model retraining. LatentPress maps text segments into compact continuous vectors using a small, reader-matched adapter, allowing a frozen decoder to ingest the compressed context directly through its input-embedding interface. This approach achieves up to 7.7× compression while matching or exceeding raw-context accuracy, and it generates compressed tokens an order of magnitude faster than text summarization or OCR-based methods.

Paper Primer

LatentPress treats context compression as a direct-read interface problem: it separates the "write" process—mapping text to soft tokens—from the "read" process, where a frozen decoder consumes these tokens as if they were standard input embeddings. By training only a lightweight adapter (roughly 0.1% of the decoder's parameters) and keeping the downstream reader entirely frozen, the system avoids the overhead of decoding vectors back into human-readable text.

LatentPress matches or exceeds the accuracy of uncompressed raw context in long-document QA.

In-domain task adaptation on LongBench-QA shows Qwen2.5-14B reaching 57.99% accuracy at 4× compression, compared to 47.93% for the uncompressed baseline. Up to 10.06 percentage points improvement in accuracy over raw context.

The system significantly reduces inference latency compared to reconstruction-based compression.

LatentPress takes 0.43–0.49 seconds per example for warm-loaded inference, which is 5–9× faster than reading raw context and 5.5–9.4× faster than cached OCR-based reconstruction.

Why use a "reader-matched" writer instead of a universal compressor?

The soft tokens are tied to the specific embedding space of the reader; training a writer per reader ensures the compressed vectors are immediately interpretable by the frozen decoder without requiring additional alignment or reconstruction layers.

How does LatentPress handle the trade-off between compression and information loss?

The system uses a role-based schedule for conversations—keeping user turns lossless while pooling assistant turns—and a uniform rate for documents, allowing the model to prioritize answer-bearing facts while aggressively compressing less critical segments.

Introduction: The Context Compression Problem

We expose the need for a direct soft‑token interface that lets frozen language models read compressed long context.

Long‑running assistants accumulate extensive histories—user utterances, tool calls, environment feedback—that quickly exceed a model’s context window. Existing pipelines retrieve, summarize, or reconstruct text before feeding it to a language model, even though the model itself does not need human‑readable text.

LatentPress writes long text into a short sequence of continuous “soft‑tokens” that a frozen decoder can read directly, eliminating any intermediate text reconstruction.

On the LongMemEval benchmark, a LatentPress writer trained on UltraChat reaches 0.504 accuracy at 7.70× compression, surpassing the 0.490 baseline that reads raw evidence. Text‑summary baselines fall to 0.184, and OCR‑based compression degrades from 0.426 to 0.312, highlighting the advantage of direct soft‑token storage.

For long documents (LongBench‑QA), in‑domain writers match or exceed raw‑context performance at 4–8× compression, though performance drops at the extreme 16× rate. Writing each conversation costs only 43 ms, and reading from the compressed soft‑token memory is 5–9× faster than processing raw text or cached OCR.

**Figure.** LatentPress compresses a long conversation into a compact soft-token memory for direct reading

How does LatentPress differ from conventional text summarization?

Summarization first produces a human‑readable short text that the model must re‑tokenize, incurring reconstruction overhead. LatentPress skips the textual intermediate entirely: the writer emits continuous vectors that are fed straight into the decoder’s embedding layer, so no tokenization or decoding of text occurs.

The shift from human‑readable text to machine‑readable soft‑tokens enables efficient, direct consumption of long context by frozen language models.

The LatentPress Architecture

We introduce a direct‑read soft‑token interface that lets a frozen decoder read compressed context.

The decoder is the most expensive component in a language model; changing it requires costly fine‑tuning. LatentPress sidesteps this by keeping the decoder frozen and feeding it a compact soft‑token representation of the full context.

Instead of decoding a long text and then re‑encoding it, the writer collapses the context into a short list of continuous vectors that sit directly in the decoder’s input‑embedding space.

How does the soft‑token interface differ from a reconstruction‑based pipeline?

Reconstruction pipelines first decode the compressed vectors back to text, then re‑encode that text for the reader, incurring two forward passes and potential information loss. The soft‑token interface skips the decode‑encode step entirely, feeding the vectors directly into the decoder’s embedding layer.

Compute $h_1 = H(E_1, c_1)$, $h_2 = H(E_2, c_2)$, $h_3 = H(E_3, c_3)$ (each yields a 4‑dim vector).

Average: $m_1 = (h_1 + h_2 + h_3)/3$.

Feed $m_1$ together with $\text{emb}(q)$ into the frozen decoder to obtain the answer.

The writer reduces three token embeddings to one soft token, cutting the input length by 66 % while preserving a learned blend of literal and contextual information.

A frozen reader is a pretrained language model whose parameters are never updated; it only consumes the soft‑token memory produced by the writer.

Why does keeping the decoder frozen matter for deployment?

Fine‑tuning a multi‑billion‑parameter decoder requires large GPU memory and long training time. By freezing it, only the tiny writer (a few megabytes) needs to be updated, making adaptation fast, cheap, and safe for production systems.

The writer reads the 100 token embeddings, applies the fusion $H$, and pools them into 5 soft tokens.

Only 12.8 M parameters are back‑propagated during training, while the 7‑B decoder remains static.

At inference, the decoder receives 5 soft tokens + the question embedding, completing the forward pass in a fraction of the original compute.

Even with a massive frozen model, the total trainable parameter budget stays under 0.2 % of the model size, enabling rapid adaptation.

Compression rates are chosen per segment based on the segment’s role (e.g., user turn vs. assistant turn), allowing important parts to stay uncompressed while less critical parts are heavily pooled.

How does role‑aware compression avoid over‑compressing critical user inputs?

Because the rule forces $k_{\text{user}}=1$, each user token is kept as a separate soft token, guaranteeing that no information from the query is lost during pooling.

User turn: each of the 2 tokens becomes its own soft token (2 soft tokens).

First assistant turn: 6 tokens are pooled into $\lceil6/8\rceil=1$ soft token.

Second assistant turn: another 1 soft token.

Total soft tokens = 2 (user) + 1 + 1 = 4, achieving a 4× compression overall.

By preserving user tokens verbatim and heavily pooling assistant content, the method keeps the question intact while still achieving strong compression.

The writer is trained with a combination of reconstruction loss (to predict target tokens) and forward‑KL distillation (to match the frozen decoder’s full‑context behavior).

Why combine reconstruction loss with forward‑KL instead of using only one?

Reconstruction alone may over‑fit to surface text and ignore the underlying reasoning, while forward‑KL alone could ignore the exact target tokens. Their sum encourages the writer to both generate correct answers and preserve the full‑context distribution.

The writer’s parameter count is tiny compared to the frozen decoder, ranging from ~4 M to ~26 M depending on the backbone, which is less than 0.2 % of the decoder size.

LatentPress achieves a direct‑read soft‑token interface that lets a frozen decoder answer questions from a heavily compressed context, with only a tiny writer to train.

Evaluating Conversational Memory

Compressed soft‑token histories keep QA accuracy while cutting memory dramatically.

We evaluate whether a compressed soft‑token history can still supply the frozen reader with the facts needed to answer questions.

LongMemEval measures how well a model answers questions when only a compressed conversational history is available.

DeepSeek-OCR compresses text by rendering it as an image, then applying OCR to recover tokens.

LatentPress retains 0.504 accuracy at 7.70× compression, matching uncompressed oracle evidence while baselines collapse.

Figure 2 and Table 2 show the accuracy–compression curves for three readers.

**Figure 2.** LongMemEval accuracy–compression frontiers. Role-aware LatentPress (orange) stays stable across compression rates on all three readers. Its relationship to the uncompressed oracle-evidence baseline (gray diamond) is reader-dependent: role-aware matches raw on Qwen2.5-7B, exceeds it on the weaker Qwen3-1.7B, and stays below the stronger raw baseline on Qwen3-8B. DeepSeek-OCR (blue) is competitive on Qwen3-8B at low compression but degrades as compression increases, and text summarization (red) is the weakest point on every reader. All results use the same 500 questions.

**Table 2.** **Zero-shot LongMemEval** on 500 oracle-evidence questions (Llama-3.1-70B-Instruct judge). LatentPress is mean±std over five seeds; baselines are deterministic. Token-F1 is in Appendix C.3.

The table compares the performance of different "Reader" models (Qwen2.5-7B, Qwen2.5-14B, and Qwen3-8B) across three methods: "Raw context", "LatentPress $f8$", and "Cached OCR b640".

LatentPress keeps QA accuracy stable across high compression rates, whereas all baselines degrade sharply.

Long-Document QA Performance

Mild compression boosts QA accuracy, but aggressive compression erodes performance.

Qwen2.5‑14B reaches 57.99 % overall at 4× compression, surpassing its uncompressed 47.93 % baseline.

Table 13 reports 57.99 ± 2.35 % for the 4× in‑domain setting versus 47.93 % raw.

Across all three readers, mild (4×) compression consistently improves or matches the uncompressed baseline, while aggressive (8×, 16×) compression degrades performance, revealing a trade‑off between memory savings and answer fidelity.

**Figure 3.** **LongBench-QA accuracy–compression frontiers.** Overall score for Qwen2.5-7B, Qwen2.5-14B, and Qwen3-8B under cross-domain (green) and in-domain (orange) writer training. Gray diamonds mark uncompressed performance at 1×, and the blue DeepSeek-OCR curve spans `base_size` 1024/512 (~ 2.6/9.9×). The red text-summary baseline (one point per reader, at 14–20×) is the weakest on every reader. In-domain adaptation exceeds the uncompressed result at the milder rates but drops below it at 16× on all three readers.

**Table 13.** **LatentPress on LongBench-QA English** (frozen soft-token compressor, official overall scores in %). Compression factors $f4/f8/f16$ correspond to $3.999/7.997/15.985\times$. The compressor is trained on LongMemEval-derived QA (cross-domain transfer), and the best operating point is reader-dependent. Overall is reported as mean$\pm$std over 5 training seeds. Raw rows repeat the uncompressed reference from Table 12.

**Table.** Comparison of performance across different readers and compression settings.

Efficiency and Deployment Costs

LatentPress dramatically cuts both write and read latency compared to reconstruction‑based baselines.

Recall that LatentPress compresses long conversational histories into soft‑token memories that a frozen reader can consume without any fine‑tuning.

LatentPress writes soft tokens roughly 22× faster than the DeepSeek‑OCR pipeline.

LatentPress generates encoded tokens in 43 ms per conversation, while DeepSeek‑OCR requires 844–1056 ms (≈ 22× longer) on the same hardware.

**Encoder-Training Ablation**

**Table 15.** Cold-cache end-to-end wall-clock time on LongBench-QA English (minutes), measured once on one H100 80GB GPU. We report the total time to go from raw context to a scored prediction, broken into stages. Cache is the one-time cost of building the method’s intermediate artifact before any question is answered: for DeepSeek-OCR this is rendering the context to images and reconstructing text by optical decoding; LatentPress has no such stage (—), since it trains a small adapter instead of precomputing a cache. Pred.+eval is reader prediction plus official eval.py scoring. Cold total is the whole-job time from scratch: for LatentPress, training+prediction+evaluation; for DeepSeek-OCR, cache generation+prediction+evaluation (i.e. the cost when no cache exists yet, hence “cold”). Score is the official overall F1 (%) at that operating point; the LatentPress scores match Table 4, while these DeepSeek-OCR resolutions (b512/b1024) are reported only here. Qwen3-8B DeepSeek-OCR ran cache and prediction in one stage=both job, so its per-stage times are merged (incl.) and only the cold total is available. Points are nearest available compression settings, not exact matches.

Separating write‑time from read‑time latency yields a deployment that is both fast to prepare and quick to answer.

Role-Allocation Ablation

Ablation studies quantify the impact of each component on LongMemEval performance.

We isolate the contribution of each design choice by removing it and measuring the drop in LongMemEval accuracy.

Freezing the borrowed encoder layers improves accuracy across all compression rates.

Freezing beats fine‑tuning by +0.018 at $ka=8$, +0.018 at $ka=16$, and +0.066 at $ka=32$.

Freezing the encoder yields higher token‑level F1, up to a +0.050 absolute gain.

At $ka=8$, frozen encoder achieves 0.504 vs 0.454 for fine‑tuned; the largest gap appears at $ka=32$ (+0.066).

**Table 9.** Token-level F1 on LongMemEval (Qwen2.5-7B reader, same 500 runs as Table 2). Judge-free auxiliary metric; the ordering matches judge-accuracy.

The table presents performance metrics for different reader models (Qwen2.5-7B, Qwen2.5-14B, and Qwen3-8B) using two methods: in-domain LatentPress and DeepSeek-OCR. Metrics include compression ratios, cache and prediction/evaluation times, total cold time, and final scores.

Table 10 breaks down the text‑summary baseline by question type, revealing that summarization struggles on answer‑bearing categories because it discards precise user facts.

Sections D.1–D.3 extend the analysis to LongBench‑QA, showing raw scores, cross‑domain transfer behavior, and failure modes that become pronounced as compression increases.

Compression Method Comparison

Key ablations compare compression strategies on compression ratio and token‑F1.

Table 7 evaluates three compression strategies—role‑aware (ours), uniform soft‑token, and DeepSeek‑OCR—across compression ratio and token‑F1.

Six recurring failure patterns emerge as compression grows: (i) unanswerable collapse, (ii) format artifacts, (iii) blank output, (iv) repetition loops (most common at 16×), (v) reasoning‑tag leakage, and (vi) well‑formed but semantically wrong short answers. The first five stem from decoding or formatting pathologies, while the last reflects genuine information loss.

Wall‑clock measurements (Table 15) show LatentPress is 6.0–13.7× faster than the DeepSeek‑OCR visual baseline, and the amortized four‑reader scenario (Table 16) still favors LatentPress.

Text-Summary Baseline Analysis

Limits of OCR cache amortization and baseline performance across readers.

Recall that LatentPress stores conversational histories as compact soft‑tokens that a frozen reader can ingest without any fine‑tuning.

**Table 10.** Text-summary baseline, per-category accuracy on LongMemEval (Llama-3.1-70B judge, official per-type protocol, 500 questions). Overall and achieved compression are reported alongside the seven question-type accuracies. Qwen2.5-14B is shown for reference (a LongBench-QA reader).

Table 11 provides spot checks of the compressed‑history reader (ka = 8, Qwen3‑8B) on representative LongMemEval questions. Even with 4–5× compression, the model recovers the correct answer in categories such as temporal‑reasoning, knowledge‑update, and abstention.

Table 12 lists the uncompressed LongBench‑QA English baseline scores, establishing the upper‑bound performance that the compressed‑history experiments are compared against.

Dataset-Specific Performance

Performance and failure analysis of LatentPress on LongBench‑QA English.

The table presents performance metrics for three different models (Qwen2.5-7B, Qwen2.5-14B, and Qwen3-8B) across several datasets: narrativeqa, qasper, `multifieldqa_en`, hotpotqa, 2wikimqa, and musique, along with an "Overall" score.

Failure‑mode analysis (Table 14) reveals several systematic degradations: unanswerable collapse (the model outputs “of adds”), JSON format artifacts, repetition loops that leak content, blank outputs, and well‑formed but incorrect answers such as returning “Watt” for a physics question.

Cold‑cache wall‑clock timings (Table 15) show that LatentPress incurs no separate cache‑generation stage, unlike DeepSeek‑OCR which must render context to images and run OCR before answering. Consequently, LatentPress’s total time (adapter training + prediction + evaluation) is substantially lower than the OCR pipeline’s cache + prediction + evaluation time.

OCR Cache Amortization

Cache‑amortization numbers reveal the trade‑off between OCR compression and block size.

**Table 16.** DeepSeek-OCR cache amortization across four readers (minutes). Amortized time is reader prediction/evaluation plus one quarter of the reader-independent cache-generation time. Qwen3-8B is omitted because its stage times were not logged separately.

Related Work and Conclusion

We position LatentPress among prior context‑compression and memory approaches.

We first clarify the baseline used for text‑summary comparison, then review how prior work relates to our soft‑token approach.

A simple baseline extracts a short textual summary of the context and feeds that summary to the frozen reader.

Soft‑token context compression replaces textual context with a handful of continuous vectors that a frozen decoder can read directly.

Compresses retrieved passages into a few continuous vectors that are fed to a language model decoder.

Learns an adapter that maps text to a compact latent representation for a frozen encoder.

Implements an auto‑encoding scheme where a frozen decoder reconstructs compressed vectors back to text.

Projects a single retrieved passage into one continuous token for downstream answering.

Targets extreme compression ratios by mapping long contexts to a few hundred vectors.

Token pruning and visual compression aim to reduce context size by discarding tokens or by turning text into images.

Prunes low‑utility prompt tokens based on learned importance scores.

Applies token‑level selection and re‑ordering to create a compact prompt.

Shows that training value is highly non‑uniform across tokens, motivating selective retention.

Renders text as images, runs OCR to reconstruct tokens before feeding a language model.

Uses glyph‑based image encoding to compress long textual context.

Applies OCR to agent‑generated histories, enabling visual compression of long dialogues.

Conversational memory systems manage long histories with retrieval, reflection, or summarization pipelines.

Maintains a persistent internal state for simulated agents via retrieval‑augmented generation.

Provides a bank of stored passages that can be queried to augment generation.

Extends GPT with an external memory that can be read and written during inference.

Combines retrieval and summarization to maintain a compact representation of long dialogues.

Latent reasoning compresses a model’s own reasoning trace rather than the input context.

Encodes a model’s intermediate reasoning states into continuous vectors for later reuse.

Distills long reasoning chains into shorter ones via self‑distillation.

**Table 1.** Positioning among continuous-vector context compression methods. “FT” is full fine-tuning; “autoenc.” decodes memory vectors back to text before answering.

In conclusion, LatentPress demonstrates that soft‑token memories can replace text or image representations while keeping the decoder frozen, opening a practical path for scalable context compression.

Implementation and Training Details

Implementation and training details for the compression system.

The compressor builds on the bottom two transformer layers of the frozen Qwen2.5‑7B‑Instruct decoder, deep‑copied so that gradient updates never affect the reader.

A tiny trainable head sits atop this encoder; it is a linear adapter $A\\in\\mathbb{R}^{d\\times d}$ initialized to the identity, so it starts by reproducing raw token embeddings and only diverges as training progresses.

Only the adapter head is learned – 12.849 M parameters for Qwen2.5‑7B, 16.781 M for Qwen3‑8B, 4.196 M for Qwen3‑1.7B, and 26.220 M for Qwen2.5‑14B – while the borrowed encoder layers and all decoder weights remain frozen.

Training optimizes the objective in Equation 3, which averages a reconstruction loss and a forward‑KL loss over non‑padding target positions, masking out padding in both terms.

Table 6 (see FigureSpotlight) enumerates the full hyper‑parameter configuration, including learning rate $1\\times10^{-4}$, chunk length 2048 tokens, and the role‑aware pooling factors $k_{user}=1$, $k_{assistant}\\in\\{8,16,32\\}$.

All generation uses deterministic greedy decoding (temperature 0), limiting the soft‑token reader to 64 new tokens per answer and the visual‑baseline reader to 256 tokens; DeepSeek‑OCR similarly generates up to 2048 tokens per page.

The compression ratio $\\rho$ is defined as the original token count divided by the number of injected vectors, $\\rho=|C|/(\\sum_i\\lceil n_i/k_{ri}\\rceil)$; with lossless user turns ($k_{user}=1$) the ratio is driven by the assistant rate, yielding reported means of 4.62–7.70×.

**Table 6.** Training and model hyperparameters. The uniform baseline uses a single factor $k \in \{4, 8, 16\}$; the role-aware model uses $k_{user}=1$ (lossless) and $k_{assistant} \in \{8, 16, 32\}$.

Evaluation and Baseline Details

Evaluation setup and hyperparameter details for the LongBench‑QA experiments.

For the cross‑domain LongBench‑QA sweep we reuse the writer architecture and training recipe, but train reader‑specific weights on QA triples using the LongMemEval‑style supervision. All runs use configs/simple.json with `qa_train`=true and mean pooling.

Table 6 lists the training and model hyperparameters: the uniform baseline varies a single pooling factor $k\in\{4,8,16\}$, while the role‑aware model fixes $k_{\text{user}}=1$ and varies $k_{\text{assistant}}\in\{8,16,32\}$. The frozen decoder and encoder layers are shared across models, and a linear adapter of size $d\times d$ is identity‑initialized and trained. Model families include Qwen2.5‑7B, Qwen2.5‑14B, Qwen3‑8B, and Qwen3‑1.7B, with backbone‑dependent parameter counts ranging from 4.196 M to 26.220 M (12.849 M for Qwen2.5‑7B); additional settings cover optimizer, learning rate, training steps, chunking strategy, `max_len`, batch size, reconstruction loss, forward‑KL weight $\lambda$, and precision.

Judge Model Configuration

Additional experimental details and ablations for the LongMemEval and LongBench‑QA evaluations.

The judge model is a frozen Llama‑3.1‑70B‑Instruct decoder served via vLLM with FP8 precision and tensor‑parallel 2; it receives prompts without a system prompt and generates a single‑token verdict “yes” at temperature 0.

Training uses AdamW with a learning rate of 1×10⁻⁴, runs for 1000 updates, and processes short conversations padded in a mask‑aware fashion using a batch size of 400 and a maximum context length of 2048 tokens, completing a single epoch.

We optimize teacher‑forced cross‑entropy (pad‑masked) with a per‑token mean loss scaled to 1.0; the decoder runs in bf16 while the compressor head uses fp32 precision.

Compression is performed with factors $\\{4,8,16\\}$, yielding effective compression ratios of $1 / \\{8,16,32\\}$ relative to the original token count.

Evaluation data include UltraChat (2 000 zero‑shot conversations) and LongMemEval (500 questions); we shuffle with seed 0 and decode greedily (temperature 0).

The text‑summary baseline prompts a fixed system instruction to compress a conversation into at most $\text{budget}$ tokens, where $\text{budget}= \max(128,\lfloor n_{\text{tok}}/r_{\text{sum}}\rfloor)$, preserving all concrete personal facts while omitting small talk.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers