ENOKI: Efficient Multi-Level Hallucination Detection

Elisei Rykov, Timur Ionov, Nikolay Ivanov, Maksim Savkin, Maksim Makarenko, Alexander Panchenko, Vasily Konovalov, Julia Belikova

ENOKI unifies claim-level verification and span-level localization using text-anchored OpenIE facts.

How can we detect hallucinations at multiple granularities (span, entity, sentence) efficiently by decomposing text into verifiable, self-contained facts?

Hallucination detectors typically force a choice between claim-level verification, which is interpretable but lacks precise localization, and span-level detection, which identifies errors but lacks factual structure. ENOKI bridges this gap by extracting text-anchored relational facts that serve as a shared representation for both verification and span projection, eliminating the need for separate alignment modules. This approach achieves superior performance on fine-grained localization benchmarks while offering flexible extraction backends that trade off accuracy and inference cost.

Paper Primer

ENOKI decomposes generated answers into incremental relational triples (subject, predicate, object) that remain tied to the original text. By verifying these triples against evidence and projecting unsupported facts back to the specific answer spans, the framework provides both a factual audit trail and precise error localization.

The framework supports three extraction regimes: an LLM-based backend for high-capacity decomposition, a deterministic rule-based backend for low-latency inference, and an encoder-based backend that uses a permutation-invariant bipartite matching loss to handle incremental fact extraction.

ENOKI significantly improves fine-grained hallucination localization over existing detectors.

The framework achieves +15.3 AUPRC on HalluEntity and +8.0 Span Coverage F1 on MuSHROOM compared to the strongest prior baselines. Substantial gains in localization precision while maintaining competitive sentence-level factuality scores.

Why is a shared representation necessary for hallucination detection?

Existing pipelines often decouple decomposition and localization, which propagates errors and requires complex alignment modules. ENOKI’s text-anchored facts ensure that verification decisions are inherently linked to the specific spans responsible for the content, enabling unified multi-granular detection.

How does the framework handle the trade-off between accuracy and computational cost?

ENOKI decouples the extraction backend from the verification interface. Users can choose between high-accuracy LLM-based extraction or lightweight rule- and encoder-based backends, allowing the system to scale from high-stakes verification to real-time monitoring.

Researchers and practitioners can now use ENOKI to obtain both interpretable factual audits and precise span-level error localization from a single, modular pipeline, regardless of their available compute budget.

Introduction and Motivation

We expose the granularity‑interpretability trade‑off in hallucination detection and motivate a unified fact‑based approach.

Hallucination detection for large language models faces a core tension: coarse claim‑level verification is easy to interpret, while fine‑grained span‑level localization pinpoints exactly where the model went wrong.

Detecting hallucinations requires both interpretable verification units and precise error spans, yet existing pipelines treat these as separate stages that duplicate work and propagate mistakes.

ENOKI extracts two relational facts: (Tesnière | was born | in 1893) and (Tesnière | was born | in Montpellier).

Verification confirms the year fact against evidence, but flags the birthplace fact as unsupported.

Projection maps the unsupported birthplace back to the exact span “Montpellier” in the answer.

By keeping the supported year while isolating the false birthplace, ENOKI demonstrates how a shared fact representation can simultaneously satisfy claim‑level interpretability and span‑level precision.

**Figure 1.** ENOKI verifies text-anchored facts and projects unsupported facts back to localized answer spans.

The key challenge is balancing granularity and interpretability when detecting hallucinations.

Related Work

Survey of extraction, verification, and span-level approaches relevant to ENOKI.

Related work falls into three strands: OpenIE for fact extraction, claim‑level verification pipelines, and span‑ or entity‑level hallucination detectors. Each strand informs a different design choice in ENOKI.

OpenIE pulls relational triples from raw text without a fixed schema, giving a flexible fact representation.

FactCheck‑Bench is a sentence‑level benchmark that measures how well a model’s answer matches a reference fact, without requiring span annotations.

Claim‑level hallucination detection typically follows a decompose‑then‑verify workflow: generated text is split into atomic claims, each claim is checked against evidence, and the results are aggregated.

Extracts atomic facts from a generated answer and computes a factual precision score by matching each fact to retrieved evidence.

An LLM‑agent pipeline that searches for supporting evidence and verifies each extracted claim, then aggregates the verification outcomes.

Restricts evaluation to claims that can be directly verified against a knowledge base, ignoring unverifiable portions.

Extracts claim‑triples and provides a structured interface for reference‑grounded checking of each claim.

Focuses on extracting entity‑centered claims, simplifying verification to entity‑level entailment checks.

Optimizes claim coverage and de‑contextualization to improve verification recall.

Span‑ and entity‑level detectors annotate the exact words or mentions that are hallucinated, enabling direct training of localized models.

Provides span‑level annotations of unsupported tokens in retrieval‑augmented generation settings.

Multilingual span‑level hallucination dataset covering scientific texts.

Captures unsupported spans in caption generation for multimodal models.

Scientific QA dataset with fine‑grained span annotations for hallucination detection.

Entity‑level benchmark that flags hallucinated entity mentions in generated answers.

Fine‑tunes a ModernBERT encoder to predict unsupported spans in RAG‑style inputs.

Improves span detection by augmenting ModernBERT with a lightweight attention head.

ENOKI unifies these strands by using text‑anchored OpenIE facts as a common representation, preserving relational structure while keeping arguments aligned to answer spans for direct projection.

**Table 1.** Comparison of hallucination detection datasets. Lengths are in characters; statistics use labeled splits.

The ENOKI Framework

ENOKI decomposes responses into incremental facts, verifies them, and maps failures back to precise spans.

ENOKI tackles the core pain of hallucination detection: balancing interpretability with fine‑grained span localization. It does so by breaking a generated answer into self‑contained facts, checking each against the reference, and projecting any unsupported fact back onto the exact text fragment that introduced the error.

ENOKI turns a raw answer into a series of tiny, verifiable facts, then highlights the exact words that caused a mismatch.

OpenIE baselines (Stanford OpenIE, MinIE, OpenIE6) provide schema‑free triples as a strong, off‑the‑shelf reference.

ENOKI‑LLM uses a prompt derived from CycleOIE, extended with three guidelines that enforce incremental fact decomposition.

ENOKI‑RULE applies 35 deterministic dependency‑parse rules over spaCy parses to emit text‑anchored triples.

ENOKI‑ENCODER builds on the Iterative Grid Labeling (IGL) architecture, replacing the BERT‑base encoder with ModernBERT‑large and adding a permutation‑invariant matching loss.

Instead of extracting one big triple, ENOKI builds facts step by step, each adding a tiny piece of information, so the verifier can pinpoint which new word broke entailment.

G1: extract fact — Enoki | is | mushroom (entailed).

G2: extract fact — Enoki | cultivated in | China (entailed).

G3: incremental addition introduces “northern” → fact — Enoki | cultivated in | northern China (not entailed).

Verifier scores G3 as contradictory; the delta span “northern” is projected back as the hallucinated fragment.

Incremental decomposition isolates the exact token that caused a verification failure, avoiding over‑penalizing the surrounding context.

**Figure 2.** ENOKI pipeline. ENOKI-LLM, ENOKI-ENCODER, and ENOKI-RULE instantiate text-anchored Open IE fact extraction; extracted facts are verified against the reference context and projected back to spans when unsupported, yielding claim-level and span-level hallucination labels.

**Figure 3.** Incremental fact decomposition: ENOKI keeps intermediate facts self-contained and assigns unsupportedness to the newly introduced delta span.

Experimental Results

ENOKI delivers top entity and span detection performance while offering faster, cost‑effective verification.

ENOKI‑LLM attains the highest AUROC (79.70) on HalluEntity, surpassing all baselines.

Table 2 reports ENOKI‑LLM AUROC 79.70 versus lower scores for competing methods.

On MuSHROOM, ENOKI‑LLM achieves the best Span Coverage F1 (52.07), outperforming encoder‑based variants.

Table 3 lists ENOKI‑LLM F1 52.07, higher than ENOKI‑RULE and ENOKI‑ENCODER.

Replacing the verifier with Qwen3.5‑9B raises MuSHROOM F1 to 54.22, a 2.15‑point gain over the BERT verifier.

Table 8 shows the Qwen‑based verifier achieving 54.22 versus 52.07 for ModernBERT‑large‑nli.

**Figure 4.** Accuracy-efficiency trade-off on RAGTruth. Point size denotes FLOPs per sentence. ENOKI-ENCODER achieves the best efficiency–accuracy balance, running 4–10× faster than competitive baselines and up to roughly two orders of magnitude faster than multi-stage LLM pipelines.

Limitations

This section outlines ENOKI’s key limitations and the trade‑offs they expose.

ENOKI’s verification relies entirely on the facts extracted in the decomposition stage. If the extractor misses a proposition, merges distinct facts, or produces overly coarse spans, the verifier cannot recover the missing structure.

Because verification is explicit, the same representation that enables interpretable decisions also caps recall when extraction is incomplete.

Span localization uses incremental projection, comparing new facts to earlier ones in the same group. This works when refinements are nested, but becomes less precise if several parts of a fact are unsupported or errors interact non‑locally.

The current implementation decomposes answers primarily at the sentence level. Consequently, cross‑sentence phenomena such as coreference, ellipsis, and discourse‑level attribution are only partially addressed.

ENOKI’s modular verifier can tolerate some extraction noise, but a brittle or poorly calibrated verifier may fail even when extracted facts are well‑formed, making the overall pipeline sensitive to verifier quality.

Matching Verification Efficiency

We assess how permutation‑invariant Hungarian matching improves incremental IGL performance.

Hungarian matching finds the optimal pairing between predicted and gold rows by minimizing total cross‑entropy cost, making the training loss invariant to row order.

How does Hungarian matching differ from simply shuffling rows randomly?

Random shuffling does not consider the cost of each pairing; Hungarian matching solves an optimization problem that finds the globally minimal total cross‑entropy, guaranteeing the best possible alignment for the current batch.

Permutation‑invariant Hungarian matching improves span‑level performance across all three benchmarks.

Table 5 shows increases from 41.02 to 46.96 (MuSHROOM), 27.25 to 34.84 (RAGTruth), and 61.14 to 65.51 (PsiloQA).

**Table 5.** Effect of Hungarian matching on incremental IGL training. We compare the original row-wise cross-entropy objective against its permutation-invariant variant with Hungarian matching. Both models are trained in the same incremental setup. Hungarian matching improves span-level performance across all three benchmarks, suggesting that row-order ambiguity is a substantial source of supervision noise in incremental extraction.

The D NLI ablation compares a standard encoder‑based NLI verifier with an LLM‑based verifier; the latter yields clear gains on MuSHROOM and RAGTruth while only slightly lowering PsiloQA.

Table 6 shows that swapping the GPT‑OSS‑120B extractor for GPT‑5.4 improves AUROC by 1.9 points and AUPRC by 1.9 points, suggesting the decomposition stage is not highly sensitive to the frontier LLM.

Using GPT‑5.4 as the fact‑decomposition backend improves HalluEntity AUROC by 1.9 points.

AUROC rises from 79.70 to 81.55 and AUPRC from 55.09 to 56.99 (Table 6).

**Table 6.** HalluEntity performance of ENOKI-LLM with two LLM decomposition backends.

Two examples illustrate annotation noise: in MuSHROOM the gold span covers a broad fabricated clause while the actual error lies in a short sub‑span; in RAGTruth the gold annotation marks the whole sentence though only the subject is incorrect.

Table 9 reports sentence‑level performance on Factcheck‑Bench, ANAH‑250, and RAGTruth, showing that explicit verification pipelines generally outperform implicit ones.

Table 10 compares extraction and verification times as well as FLOPs, revealing the trade‑off between accuracy and computational cost for each pipeline.

**Table 7.** Training hyperparameters for ENOKI-ENCODER.

The table displays two rows comparing "Gold annotation (coarse)" and "Actual error (subject swap)" with their corresponding character ranges and text snippets.

Computational Efficiency and Data

Key efficiency numbers and the ENOKI‑RULE pipeline details for fact verification.

Table 10 reports per‑sentence latency and FLOPs for a range of verification pipelines on FactCheck‑Bench.

These numbers show that multi‑stage LLM pipelines (e.g., Claimify) are the slowest, while rule‑based ENOKI variants achieve two orders of magnitude lower latency.

ENOKI‑LLM concentrates most of its runtime in the extraction stage, confirming that decomposition is the primary bottleneck for LLM‑based pipelines.

ENOKI‑RULE and ENOKI‑ENCODER retain explicit fact decomposition but replace costly autoregressive verification with a single forward pass through a modest NLI encoder, explaining their dramatically reduced FLOPs and latency.

The pipeline applies a fixed sequence of linguistic rules to a sentence, each rule emitting a candidate triple that is later turned into a concrete span.

A Candidate bundles three head tokens (subject, predicate, argument) together with optional metadata describing how the span should be built.

The gate decides whether a newly proposed rule improves overall performance, balancing F1 gain against coverage.

All rules inherit from a common abstract interface, guaranteeing a uniform API for the pipeline.

The 35 rules are organized into seven semantic clusters, each covering a distinct linguistic phenomenon.

The worked example shows how the rule `be_attr_noun_prep` extracts “is home to” from a sentence and how incremental widening rules emit progressively larger argument spans.

**Table 13.** ENOKIQA statistics. Hallucination rates are computed over incremental triples extracted by ENOKI-LLM and verified with Qwen3.5-9B NLI at threshold 0.5.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers