PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews
Dingjie Song, Zhiling Yan, Ziqian Zhang, Yuxuan Zhang, Lichao Sun
PremiseCheck decomposes review comments into verifiable claims, enabling cheap, accurate fact-checking without reading the full paper for every assertion.
How can we efficiently and accurately detect factual errors in peer review comments by decomposing them into verifiable premises?
Peer review is under strain, and both human and AI reviewers frequently include factual errors in their comments. Checking these errors by hand is impossible at scale, and existing automated systems are either too expensive or too inaccurate to be practical. PremiseCheck solves this by splitting the task: an LLM rewrites a comment into explicit, self-contained premises without reading the paper, and a lightweight decision model verifies each premise against the paper. This division of labor avoids the high cost of having a frontier model read the entire paper for every single comment. On the ReviewFact benchmark, PremiseCheck matches the accuracy of a frontier LLM (GPT-5.5) at roughly half the cost, and on NLP reviews, it achieves this at a quarter of the cost and less than half the latency.
Paper Primer
The core mechanism hinges on decomposition: the system treats a review comment as a collection of atomic claims rather than a holistic block of text. By isolating these claims, the framework allows a specialized decision model to perform targeted verification against the paper, while the LLM's role is restricted to simple text transformation.
PremiseCheck achieves accuracy comparable to holistic GPT-5.5 while significantly reducing computational overhead.
Across NLP, machine learning, and natural science reviews, PremiseCheck stays within 0.3 AUROC points of the reference system. On NLP reviews, the framework reduces cost to 25% and latency to under 50% of the reference system's requirements.
The framework is model-agnostic regarding the verification step.
Four different decision models, including the open-weight Kev-27B, achieved performance within one point of the holistic GPT-5.5 reference when used as verifiers.
Why is a decision model used instead of just having an LLM check the premises?
Decision models are optimized to return the probability that a specific statement holds within a document at negligible cost. Unlike LLMs, which generate text and require expensive reasoning passes, decision models can process multiple phrasings of a verification question in a single call, making them far more efficient for repetitive fact-checking.
What is the scope of this system — does it catch all types of review errors?
No; PremiseCheck is strictly limited to factual errors, such as misreading the paper or claiming an omission. Errors of judgment—such as whether a concern is important, whether a request is reasonable, or whether the work is novel—are explicitly out of scope.
For researchers and area chairs, PremiseCheck provides a scalable way to flag potentially erroneous review comments for human review, shifting the burden from manual verification to targeted inspection of specific, refuted claims.
Introduction
Introduce the fact‑checking challenge for peer‑review comments and the PremiseCheck solution.
The surge in research submissions has overloaded reviewers, and both human‑written and AI‑generated review comments frequently contain factual errors. Detecting these errors is essential because they can mislead authors and distort decisions, yet no dedicated fact‑checking task exists. Existing fact‑checking pipelines either read the full paper for each comment—incurring high cost—or rely on cheap decision models that fail on multi‑premise comments.
Fact‑checking review comments requires pinpointing statements that the paper does not support, without judging subjective concerns.
To study this problem we introduce ReviewFact, a benchmark that labels factual errors in review comments across NLP, machine learning, and the natural sciences, and reports accuracy together with cost and latency. A frontier LLM can judge comments accurately but must reread the entire paper for each comment, leading to prohibitive expense. In contrast, decision models such as Jev answer single‑claim checks in milliseconds at negligible cost but perform at chance on whole comments.
**Figure 1** Three ways to check a review comment, shown on a real comment from ReviewFact. AUROC, cost, and time are measured on the NLP reviews of ReviewFact, with cost relative to the LLM-only row, which is the reference in our experiments. The LLM is GPT-5.5 and the decision model is Jev.
PremiseCheck bridges the gap by using an LLM to rewrite each comment into a few explicit premises without accessing the paper, then passing those premises to a decision model that checks them against the document. The comment’s final score is the minimum of its premises’ scores, ensuring that any false premise flags the whole comment. This decomposition lets the expensive LLM operate once per comment while the cheap verifier runs once per premise, dramatically reducing overall compute.
The gap between holistic LLM capability and the need for precise, verifiable fact‑checking in peer review drives the need for efficient, premise‑level verification.
Related Work and Benchmark
We situate ReviewFact and Holistic LLM Judgment within prior work on review fact‑checking.
Prior work on factual errors in review comments has focused on datasets such as ReviewCritique, ReviewScore, PeerReview Bench, and ClaimCheck, which annotate comments against the underlying paper.
These datasets either apply inconsistent error definitions or omit cost and latency reporting, whereas ReviewFact adopts a single definition across three fields and explicitly measures computational overhead.
Fact‑verification pipelines typically decompose a text into atomic claims and verify each against a source, often using a small trained verifier; this approach multiplies expensive LLM calls when applied to review comments.
PremiseCheck avoids this blow‑up by letting an LLM generate premises without reading the paper and by using a decision model (e.g., Jev) to route and score them, contrasting with approaches that require the LLM to read the full paper for every premise.
Cost‑aware inference methods such as cascades and routers still forward the full paper to an expensive LLM for most comments; PremiseCheck’s design eliminates that cost by only sending the paper to a lightweight decision model.
It evaluates how well a system can spot factual errors in peer‑review comments by assigning a low score to erroneous comments and a high score to correct ones.
It lets a large language model directly assign a factuality score to a review comment without breaking the comment into premises.
**Figure 2.** (a) The task on a real ReviewFact comment that claims a baseline is missing although the paper reports it. (b) ReviewFact labels from the expert annotations of three datasets. For PeerReview Bench, two authors independently re-check which errors are factual on 178 comments and agree with the GPT-5.5 labels with $\kappa$ of 0.76 and 0.77.
**Figure 3** ReviewFact by part and reviewer type. The left panel shows the share of comments with a factual error, next to the counts of errors and comments. The right panel shows the types of these errors, where Other is Unstated statement or Inaccurate summary.
The PremiseCheck Framework
PremiseCheck breaks comments into premises, routes and verifies them, yielding cheap, accurate fact checks.
PremiseCheck processes a review comment in four stages: it first splits the comment into atomic premises, then decides which premises can be answered from the paper, verifies each premise, and finally aggregates the scores into a single comment score.
PremiseCheck treats a review comment as a tiny argument, extracts each claim, asks a specialised verifier for its truth, and then uses the weakest claim as the overall verdict.
LLM receives the 12‑token comment and outputs the two premises.
Each premise is a complete declarative sentence; no extra facts are added.
The decomposer prompt is 256 tokens long, independent of the paper size.
Decomposition isolates exactly what the reviewer is asserting, preventing the model from hallucinating additional facts.
How does this decomposition differ from simply extracting sentences from the comment?
Extraction would keep the original wording, which may contain fragments or implicit references. Decomposition forces each premise to be a self‑contained declarative statement and explicitly excludes any facts not asserted by the reviewer.
The decomposer rewrites the comment as a short list of independent claims, each phrased as a complete sentence that can be evaluated on its own.
The LLM receives the 9‑token comment.
It outputs one premise, preserving the exact claim.
No additional context (e.g., paper sections) is introduced.
Even a single‑sentence comment is handled uniformly, keeping the downstream pipeline simple.
Why not just feed the whole comment to a verifier without decomposition?
Without decomposition, the verifier would have to parse multiple intertwined claims, leading to ambiguous answers and higher compute because the full comment (and often the full paper) would be read for each claim.
Routing decides whether a premise can be answered solely from the paper or needs external knowledge, preventing the paper‑reading model from being penalised for knowledge‑type claims.
Routing model evaluates r₁ ≈ 0.78 → g₁ = 1 (paper‑grounded).
Routing model evaluates r₂ ≈ 0.32 → g₂ = 0 (knowledge).
Only q₁ will trigger a verification call that reads the paper.
Routing isolates the expensive paper read to the few premises that truly need it, saving compute.
Why not verify every premise directly against the paper?
Verifying a knowledge premise against the paper would almost always return false, unfairly penalising the comment; routing avoids this by sending such premises to a generic knowledge verifier instead.
Verification asks a decision model a yes/no question tailored to the premise’s route, either “according to the paper” or “in general,” and returns a probability that the premise holds.
Verifier receives the full paper D and the premise q₁.
It returns p₁ = 0.04, indicating the premise is likely false.
The low score reflects that the paper indeed contains results on dataset X.
Verification produces a calibrated probability, not a hard true/false, allowing the aggregation step to reflect uncertainty.
How does this verification differ from a standard entailment model?
Standard entailment models are trained on generic sentence pairs and output a binary label. Here the decision model is prompted with a specific question (“according to the paper”) and returns a probability, which can be directly compared across premises.
Aggregation collapses all premise scores into a single comment score by taking the minimum, because a single false premise makes the whole comment erroneous.
Compute s(c) = min(0.87, 0.22, 0.65) = 0.22.
The comment is flagged because premise 2 is likely false.
If the decomposer had returned no premises, the system would invoke the holistic model instead.
Using the minimum makes the system highly sensitive to any erroneous claim, matching the goal of fact‑checking.
Table 1 (in the Results section) shows that PremiseCheck achieves higher AUROC than the holistic baseline while incurring far lower compute and latency per comment.
**Figure 4.** PremiseCheck on a ReviewFact sample that experts labeled as a factual error, with premises shortened for display. Only the verifier reads the paper, and the lowest premise score, shown in red, is the comment score.
**Figure 8** Cost per comment on the NLP reviews (a) by GPT-5.5 token type and verifier and (b) by paper length at medium reasoning (Appendix D.3).
Main Results
PremiseCheck halves cost while matching holistic accuracy across ReviewFact domains.
We evaluate PremiseCheck on the ReviewFact benchmark, comparing per‑comment and batched settings against holistic LLM baselines and a cascade approach.
PremiseCheck matches holistic GPT‑5.5 AUROC while using roughly half the compute cost.
Table 1 and Figure 5a show cost ≈ $50\%$ of the reference and AUROC within $0.3$ points across all three ReviewFact parts.
Jev is a lightweight verifier that consumes only the input tokens of a comment, producing a binary plausibility score without accessing the full paper.
How does Jev differ from a traditional decision‑tree classifier?
Traditional trees split on discrete feature thresholds and require handcrafted features. Jev, by contrast, operates on dense token embeddings and learns its splits end‑to‑end, allowing it to capture nuanced linguistic patterns without manual feature engineering.
**Figure 5.** Cost and AUROC on the NLP reviews, (a) per comment and (b) batched per review (§8.1), where PremiseCheck uses 16 phrasings. Labels give the reasoning of the LLM, and the line and band mark the per-comment reference and ±3.
PremiseCheck consistently outperforms holistic baselines across domains while cutting compute cost by up to $75\%$.
Analysis of PremiseCheck
Ablation analysis shows each component’s impact on cost and accuracy.
PremiseCheck splits a review comment into atomic premises, routes each to a specialized verifier, and aggregates the scores, yielding higher accuracy than a holistic LLM while using less compute.
Removing the decomposition step forces GPT‑5.5 to act as verifier, inflating pipeline cost.
Cost rises to 3.1 × that of holistic judgment, with no accuracy gain over Jev.
Replacing Jev with the smaller GPT‑5.4‑mini verifier also raises cost.
Pipeline cost becomes 1.4 × holistic judgment, again without accuracy improvement.
Decision models (Jev, Kev‑27B, Microsoft Decision‑1, Perplexity Decider) score 50–57 as holistic judges but stay within one AUROC point of holistic GPT‑5.5 when used as verifiers.
AUROC difference ≤ 1 point across all four models.
Omitting the routing step reduces AUROC.
AUROC falls from 61.8 to 58.8 on a common set of machine‑learning premises.
Aggregating premises by mean instead of minimum harms performance.
AUROC drops to 59.4, a 2.4‑point loss compared with the minimum‑based aggregation.
Using 16 phrasings of the verification question improves AUROC.
AUROC rises from 63.1 to 65.7 when all 16 phrasings are used.
**Figure 6.** AUROC of PremiseCheck minus holistic GPT-5.5 on the NLP reviews, with 95% intervals.
**Figure 7.** AUROC on the NLP reviews of each model as a holistic judge and inside PremiseCheck. Cost is relative to holistic GPT-5.5 in (a) and to the same LLM in (b), where * marks API cost. The band marks holistic GPT-5.5 ±3 (Appendices D.2, D.4).
**Figure 9** AUROC on the NLP reviews by number of phrasings, as means over 50 random subsets of the 16 phrasings with 5–95% ranges. Squares are fixed sets.
**Figure 11** Flagging factual errors on the NLP reviews. Hit rate is the share of papers with an error that one of the three flags hits. PremiseCheck uses 16 phrasings, and ties are broken at random.
**Figure 12.** A flagged comment from the machine learning reviews with the authors’ OpenReview response, quoted verbatim.
Limitations and Ethics
We outline PremiseCheck’s current limits, ethical use, and future directions.
PremiseCheck is limited to checking factual statements that can be verified against the paper’s content. It does not assess subjective judgments such as the perceived importance or novelty of a claim, which the authors note as a direction for future work.
The main decision model, Jev, is accessed via an API; with the open‑weight Kev‑27B model it attains the same AUROC on the NLP reviews of ReviewFact. Extending evaluation to additional open decision models remains an open research avenue.
We evaluate only with ReviewCritique, ReviewScore, and PeerReview Bench under their research‑evaluation licenses; we do not train on these datasets. Our code is released under the MIT license, and the NLP and natural‑science portions of ReviewFact inherit the licenses of ReviewCritique and PeerReview Bench, which forbid model training. Because ReviewScore lacks an explicit license, we provide a reconstruction script instead of the raw data. A verifier that falsely flags a correct comment could unfairly damage a reviewer’s reputation, while a missed error could grant undue credibility to an incorrect claim. PremiseCheck is therefore intended as a human‑in‑the‑loop aid, not a replacement for reviewer, area‑chair, or author judgment.
**Figure 10** Stage that causes each sampled failure of PremiseCheck. False alarms are correct comments in the lowest 10% of scores, and misses are errors above the median.
The appendix provides extensive supplementary material: A.1 Source Annotations, A.2 Label Verification; B.1 Prompts, B.2 Decision‑Model Questions, B.3 Paper Input, Cost, and Latency; C.1 Full Results on the NLP Reviews, C.2 Subsets, C.3 Latency, C.4 Robustness, C.5 Machine Learning and Natural‑Science Reviews; D.1 Capability Map of the Decision Model, D.2 Verifiers, D.3 Cost Breakdown and Paper Length, D.4 Other Decision Models and Other LLMs, D.5 Design Choices, D.6 Error Types and Scope, D.7 Failure Analysis; E.1 Batching and Fusion; F.1 Scored Comments, F.2 Flagged Comments with Author Responses.
Practice and Conclusion
Practical deployment insights, batching, fusion, flagging policies for PremiseCheck and future directions.
Batching both approaches improves performance: batched holistic GPT‑5.5 without reasoning reaches 67.8 % at 7.7 % of the reference cost, while batched PremiseCheck reaches 69.0 % at 20 % cost. The two methods are not significantly different, making holistic GPT‑5.5 the cheaper choice when only scores are needed.
Fusing the batched scores yields the best result: the combined system attains 71.5 % at 28 % of the reference cost, significantly above batched holistic GPT‑5.5, and the two scores have a rank correlation of only 0.56.
In simulations of three flagging policies on NLP reviews (where 4.8 % of comments contain factual errors), PremiseCheck flags hit errors two to three times as often as random, whereas holistic GPT‑5.5, per comment or batched, performs only marginally better than random despite a comparable AUROC.
Reading the three most suspicious comments per paper lets an area chair detect an error in 40 % of papers that contain one with batched PremiseCheck, compared with 29 % using batched holistic GPT‑5.5.
Jev assigns a probability of 0.04 to the premise that the paper lacks a summary of its takeaways, and the authors respond that each experiment section ends with one.
Fact‑checking review comments does not require a frontier LLM; the ReviewFact benchmark defines factual errors across three fields, and PremiseCheck achieves accuracy comparable to GPT‑5.5 at a fraction of its cost.
Decision models from several providers can verify the premises, and on NLP reviews even cheaper LLMs than GPT‑5.5 can generate the premises.
PremiseCheck can support peer review both while a reviewer writes each comment and after the review is finished, where fusing it with holistic GPT‑5.5 gives the highest accuracy.
Future work includes improving the decomposer to reduce false alarms, handling errors of judgment that lie outside PremiseCheck, and extending ReviewFact to jointly measure accuracy, cost, and latency while testing open‑weight decision models.
Appendix: Benchmark Details
Benchmark datasets, label verification, and prompts used for PremiseCheck evaluation.
Source annotations come from three expert‑curated datasets. ReviewCritique (Du et al., 2024) contains sentence‑level reliability labels from 40 NLP researchers, each comment accompanied by an explanation of any deficiency. ReviewScore (Ryu et al., 2025) provides five‑point factuality ratings for premises reconstructed from machine‑learning reviews. PeerReview Bench (Kim et al., 2026) offers binary correctness judgments on comments from Nature‑family papers, with optional rationales.
Label verification is performed by two authors who re‑run the GPT‑5.5 decision on the 178 PeerReview Bench comments flagged as incorrect. Each annotator sees the comment, the expert rationale, and the paper text, but not the GPT‑5.5 label, and the order of comments is randomized per annotator. Agreement statistics are reported in Table 2.
**Table 2.** Human verification of the natural-science labels on 178 comments. Top: labels of the two annotators with agreement shaded. Bottom: Cohen's $\kappa$ for factual vs. other (binary) and for the three labels. GPT-5.5 labels 65 comments as factual, 92 as judgment, and 21 as unclear.
AUROC scores on the natural‑science reviews are summarized in Table 3. The “Holistic GPT‑5.5” column reports the baseline performance, while “Premise‑Check” shows results using the medium‑reasoning decomposer. The paired difference $\Delta$ (with 95% confidence intervals) varies from –3.7 to +2.7, and the only statistically significant advantage (+2.7) occurs on the 47 comments that both annotators label factual.
**Table 3.** AUROC on the natural-science reviews with factual labels from GPT-5.5 or from the annotators. “Both” and “Either” count a comment as a factual error when both or either annotator labels it factual. PremiseCheck uses the medium-reasoning decomposer, and $\Delta$ is the paired difference to holistic GPT-5.5.
The factual filter prompt for natural‑science labels asks the model to classify a review comment as “factual”, “judgment”, or “unclear”, requiring a subtype for factual errors and a one‑sentence justification. The prompt deliberately hides the paper text, forcing the model to rely solely on the comment and the expert rationale.
Decomposition prompts differ by domain. For NLP reviews, the system prompt requests a list of premises (1–6) that reconstruct the reviewer’s argument, each premise phrased to be understandable without the original review. For machine‑learning reviews, the prompt limits premises to 1–4 and restricts them to the reviewer’s explicit claims, allowing at most one general‑knowledge premise.
Implementation details (Section B) enumerate the prompts used for both the factual filter and the decomposition stages. These prompts are the only interface between the model and the raw review comments, and they encode the definitions of “factual”, “judgment”, and “unclear” as described in the annotation guidelines.
Appendix: Premise Examples
Logical premises and conclusion for the appendix.
The appendix enumerates three premises: (1) the paper contains information that contradicts the proposition; (2) the proposition is false according to the paper; (3) the proposition misrepresents the paper. Based on these, the conclusion is that the proposition is false.
Appendix: Additional Results
Provides detailed performance tables, figures, and cost analysis for Jev versus GPT‑5.5 on verification tasks.
D.1 Capability Map of the Decision Model shows AUROC results for Jev, GPT‑5.4‑mini, and GPT‑5.5 across individual verification operations and whole‑comment judgments.
On atomic operations Jev matches or exceeds GPT‑5.5, while on whole‑comment judgments Jev lags behind; all models hover near chance on “comment is correct”. The Jev verifier costs $0.00044$ per premise check and runs in $216$ ms, far cheaper than GPT‑5.5’s $5.1$ s.
The figure displays the AUROC performance of three models (Jev, GPT-5.4-mini, and GPT-5.5) across various atomic operations and whole-comment judgment tasks. Performance is measured on a scale from 50 to 100, with a vertical dashed line at 50 representing "chance" performance.
**Table 11.** AUROC of three models on verification operations with identical inputs. $n$ is the number of items and Neg the number of negative items. Bold marks the best model in each row.
**Figure 17** Two comments from the NLP reviews scored by PremiseCheck with the medium-reasoning decomposer. Route and $p$ are the decision and probability of Jev, red marks the premise with the lowest probability, which gives the comment score, and the comment, premises, and expert text are verbatim.
Appendix: Verifier Performance
This appendix details verifier performance, cost breakdowns, and design variations for PremiseCheck.
**Table 12.** AUROC on the NLP reviews of each model as a holistic judge and as the verifier of PremiseCheck. All verifiers receive the same premises from the medium-reasoning decomposer and the same routing. The reference is holistic GPT-5.5 at medium reasoning, and cost (%) is relative to it. Cost and total latency cover the whole pipeline, and verification latency covers only the verification step. Holistic GPT-5.4-mini uses low reasoning.
**Table 13** Off-the-shelf fact-checking models on the NLP reviews (6,466 comments, 308 factual errors). Reasoning refers to the GPT-5.5 call (for PremiseCheck, the decomposer). API cost is relative to the reference, and for Kev-27B and MiniCheck it covers only the decomposer. GPU is H200 GPU seconds per comment, an upper bound that includes idle time. The last column compares with PremiseCheck using Jev on the same premises.
**Table 14** Per-comment cost on the NLP reviews. Shares split the cost of each GPT-5.5 call by token type and the cost of PremiseCheck by stage. The Jev row includes routing, and the dimmed GPT-5.4-mini and GPT-5.5 rows give the approximate cost of checking with these models instead.