Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking

Kailin Jiang, Lei Liu, Jian Xi, Hui Xu, Junlin Liu, Baochen Fu, Bin Li, Vichwang, Yu Lu, Haibo Shi

A multi-dimensional evaluation framework and training-free selection method for optimizing document sets in RAG.

How can we evaluate document retrieval sets based on their utility for downstream generation rather than just individual document relevance?

Retrieval systems currently rank documents in isolation, ignoring how they interact within a set. This leads to redundant, conflicting, or incomplete context that degrades the performance of downstream large language models. The authors introduce SETWISEEVALKIT, a nine-dimension benchmark that evaluates document sets holistically, and RUBRIC4SETWISE, a training-free method that uses these evaluation rubrics as direct signals to select optimal document subsets. This approach achieves state-of-the-art generation performance while requiring fewer documents and search rounds than traditional relevance-centric rerankers.

Paper Primer

Traditional retrieval benchmarks like nDCG treat set quality as the sum of individual document relevance scores. This ignores critical inter-document dynamics: a set of five highly relevant but identical documents is useless for reasoning, yet receives a perfect score under current metrics.

RUBRIC4SETWISE converts evaluation criteria into selection guidance: it uses a reasoning backbone to model set utility through chain-of-thought prompting, adaptively selecting the document subset that best satisfies the query-specific rubric. It functions like a filter that reads the entire set to identify which documents provide unique, non-conflicting evidence, rather than just scoring them one by one.

Rubric-based evaluation scores are highly predictive of downstream generation quality.

Consistency analysis across 12 rerankers shows a strong positive correlation between rubric coverage and generation performance.

RUBRIC4SETWISE outperforms existing rerankers in both short-form and long-form scenarios.

In short-form QA, it improves EM and F1 scores over the best setwise baseline while using fewer documents; in long-form, it achieves the highest LLM-judge scores with fewer search turns. +0.97 EM and +0.62 F1 in short-form; +2.42 LLM-judge score in long-form.

Why do existing rerankers fail to optimize document sets effectively?

They suffer from two structural blind spots: they evaluate documents independently rather than as a set, and they focus exclusively on relevance, ignoring dimensions like complementarity, redundancy, and conflict.

What is the scope of the SETWISEEVALKIT benchmark?

It covers three levels (document, set, global) and nine dimensions (e.g., relevance, authenticity, complementarity, redundancy, conflict, completeness) across both single-turn and multi-turn agent scenarios.

Retrieval optimization must shift from maximizing individual document relevance to maximizing the utility of the document set as a whole. Practitioners should adopt multi-dimensional rubrics to diagnose and guide set selection for RAG pipelines.

Beyond Relevance-Centric Retrieval

We expose why evaluating document sets, not single documents, is essential for LLM generation.

The rise of Retrieval‑Augmented Generation means that LLMs consume entire retrieved document sets as context. When evaluation treats each document in isolation, it cannot detect harmful interactions such as redundancy, factual conflict, or missing evidence that degrade generated answers.

Traditional retrieval metrics score documents one‑by‑one, so they miss set‑level flaws that directly limit what a language model can generate.

Redundancy: Documents 1 and 2 contain identical sentences, consuming two of the five context slots without adding new information.

Conflict: Document 3 states “Event X occurred in 1998”, while Document 4 claims “Event X occurred in 2002”. The model receives contradictory facts.

Missing key information: The answer requires the statistic “42 % market share”, which appears in none of the five documents.

High nDCG does not guarantee a useful set; set‑level defects can cripple downstream generation despite perfect relevance scores.

**Figure 1.** Limitations of previous evaluation Although all retrieved documents are relevant (nDCG@5 = 100%), the set exhibits critical defects undetectable by traditional metrics: information conflict (Doc [11] contains factual errors), redundancy (Docs [11]&[12], [8]&[5] contain repeated content), and missing key information.

The key shift is moving from document‑level relevance to set‑level utility as the evaluation target.

Retrieval Evaluation Benchmarks

We situate our rubric framework among retrieval benchmarks and LLM‑based reranking.

Related work clusters around three themes: traditional retrieval benchmarks, the rise of LLM‑driven reranking, and rubric‑based evaluation. The former focus on per‑document relevance, the latter progressively consider set composition, while rubric approaches bring multi‑dimensional judgments from education into NLP.

**Table 1.** Comparison of existing benchmarks with our proposed framework.

Early benchmark that annotates relevance per document and aggregates scores with nDCG.

Broad‑coverage benchmark that also relies on per‑document relevance and nDCG aggregation.

Multitask benchmark that aggregates per‑document relevance via nDCG across diverse datasets.

Recent benchmark that adds LLM‑based judges but still reduces evaluation to binary per‑document relevance.

Evaluation suite that uses LLM judges to score individual documents, still aggregating with relevance‑only metrics.

Framework that applies LLM judges to documents but aggregates scores without considering document sets.

Proposes an order‑free set metric but still evaluates each document in isolation.

Our rubric‑based evaluation toolkit that scores document sets across nine dimensions and supports both single‑ and multi‑document rounds.

Chronology from pointwise scoring to setwise optimization, including reasoning‑enhanced and set‑oriented methods.

Adapts educational rubrics and LLM‑as‑Judge techniques (e.g., MT‑Bench, RubricEval, ResearchQA) for multi‑dimensional quality assessment.

A training‑free signal that uses the same rubrics to guide set selection, bridging evaluation and retrieval.

The Rubric-Oriented Evaluation Framework

We introduce a rubric-based framework and toolkit to evaluate and guide document‑set selection.

Standard evaluation treats each retrieved document independently and sums relevance scores, assuming set quality is merely the sum of its parts. This ignores redundancy, conflict, and complementary information that only emerge when documents are considered together. The resulting mismatch hurts downstream LLM generation that consumes the whole set as context.

Think of the evaluation as a structured checklist that verifies not only whether each document answers the query, but also whether the collection as a whole avoids duplication, resolves contradictions, and supplies all required facts.

How does Rubric‑Oriented Evaluation differ from traditional relevance aggregation?

Traditional aggregation scores each document in isolation and adds the numbers, implicitly assuming independence. Rubric‑Oriented Evaluation asks a set of nine holistic questions that explicitly probe redundancy, conflict, and complementarity, so the final score reflects the collective usefulness of the whole set rather than a simple sum.

Document A mentions tax burden and Enlightenment ideas.

Document B adds fiscal crisis but repeats the tax burden fact.

Document C contradicts Document A by claiming the tax burden was negligible.

The evaluator scores Complementarity = high (all three causes appear), Redundancy = moderate (tax burden repeated), Conflict = high (direct contradiction).

This toy example shows how the rubric surfaces issues that per‑document relevance scores would miss, such as a single fact being over‑emphasized or contradictory.

SETWISEEVALKIT bundles the data, rubric taxonomy, and generation pipeline so researchers can plug‑in any reranker and obtain rubric‑based set scores without building the infrastructure from scratch.

What does SETWISEEVALKIT actually provide beyond a dataset?

It supplies (1) a curated candidate‑pool retrieval baseline, (2) a nine‑dimension rubric taxonomy, (3) a multi‑model rubric generation and filtering pipeline, and (4) evaluation scripts that score any document set against the query‑specific rubrics, all ready to plug into existing reranking experiments.

**Figure 2.** Overview of SetwiseEvalKit’s evaluation. ❶ Three-level, nine-dimension rubric taxonomy for document set evaluation. ❷ Evaluation pipeline where a reranker selects documents from the candidate pool and a judge scores the selected set against query-specific rubrics on a 0–4 scale.

**Figure.** Rubric Quality analysis comparing three models (GPT 5.1, Gemini 3.1-Pro-Preview, and Deepseek-V4 Pro) across two scenarios (Short-form and Long-form) for two metrics: Discriminative Importance and Omission-based Change in Set Ranking.

Generate query‑specific rubrics R from the (q, a) pair using the SETWISEEVALKIT pipeline.

Feed R and the candidate pool C to the reasoning backbone (Qwen3‑8B) with a chain‑of‑thought prompt that asks the model to propose a subset S maximizing the rubric‑based utility f(S; q, R).

The model iteratively adds documents that improve the overall rubric score, stopping when adding another document no longer raises any dimension above its satisfaction threshold.

Return the selected set $S$* and its nine‑dimensional rubric score for downstream LLM consumption.

By replacing blind relevance summation with rubric‑driven selection, we obtain document sets that are less redundant, more complementary, and better aligned with the ultimate generation task.

Experimental Results and Correlation

SetR (8B) tops overall scores and rubric coverage predicts generation quality.

SetR (8B) attains the highest overall coverage score of 45.85 on SETWISEEVALKIT.

Table 2 reports an Overall score of 45.85 for SetR, exceeding the next best Rank4Gen (43.87).

SetR outperforms Rank4Gen by 1.98 overall points.

Overall scores: SetR 45.85 vs Rank4Gen 43.87 (Table 2).

Rerankers reorder a fixed pool of retrieved documents to improve downstream generation, using either pointwise scoring, sequence modeling, or joint set optimization.

How does setwise reranking differ from adhoc reranking?

Adhoc rerankers rank documents one‑by‑one and then pick the top‑k, whereas setwise rerankers evaluate the whole candidate set as a unit, explicitly optimizing for inter‑document properties such as redundancy, complementarity, and overall coverage.

**Figure 4.** Correlation between rubric coverage score and downstream generation performance. Each point represents a reranker method. The strong linear correlation (Pearson $r = 0.92$) demonstrates that SETWISEEVALKIT’s evaluation scores are highly predictive of actual generation quality.

**Table 2.** Performance comparison of different rankers across Doc-Level, Set-Level, and Global-Level dimensions.

Rubric coverage scores are highly predictive of downstream generation performance.

Implementation and Case Studies

Implementation specifics for datasets, models, rerankers, and rubric statistics.

Section A.1 lists the four short‑form multi‑hop QA benchmarks used to construct document sets: HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle.

HotpotQA provides over 112 K Wikipedia‑based questions that require reasoning across multiple supporting passages; 2WikiMultihopQA contributes ~192 K samples with explicit evidence triples; MuSiQue offers 2‑to‑4‑hop questions built from single‑hop seeds to avoid shortcut reasoning; Bamboogle is a curated set of 125 two‑hop queries where no single search can retrieve the answer directly.

Section A.1 also introduces the long‑form benchmark ResearchQA, a scholarly QA collection with 21 K queries and 160 K rubric items covering citation, explanation, and limitation criteria, validated by 31 Ph.D. annotators.

Section A.2 details the generators: the short‑form scenario uses Llama‑3.1‑8B‑Instruct to produce concise answers from the top‑k retrieved passages, while the long‑form scenario employs DR.Tulu‑8B, an autonomous multi‑turn search agent that alternates planning, tool invocation, and answer generation.

Section A.3 enumerates the reranker baselines integrated into the pipelines: BGE‑Reranker‑Large (cross‑encoder), MonoT5 (seq2seq pointwise), RankT5 (decoder‑based scalar scorer), RankLLaMA (LLama‑2‑7B pointwise), RankVicuna and RankZephyr (open‑source LLM listwise), Setwise (prompt‑based set selection), and Rank1 (chain‑of‑thought pointwise).

Section A.4 reports rubric count statistics from SETWISEEVALKIT (Table 5). Across three frontier LLMs, the short‑form scenario yields roughly 15, 9, 11 rubrics per query for GPT 5.1, Gemini 3.1‑Pro‑Preview, and DeepSeek‑V4 Pro respectively; the long‑form scenario produces about 21, 10, 20 rubrics per query, showing balanced coverage across Doc‑, Set‑, and Global‑level dimensions.

**Figure 6.** Case Study. Each rubric dimension pinpoints specific document set deficiencies, guiding targeted optimization.

**Table 5.** Rubric Count Statistics on SETWISEEVALKIT. We report the total number of rubrics generated per dimension (top) and the average number of rubrics per query (bottom), grouped by Doc-Level, Set-Level, and Global-Level dimensions.

Human Validation of Rubric Quality

Human validation of rubric quality, agreement metrics, and reproducibility.

The paper’s core claim is that evaluating document sets with rubric‑oriented metrics captures redundancy and conflict, which plain relevance scores miss. This appendix shows how we verified that the rubrics are reliable and reproducible.

We ask crowdworkers to judge each rubric item on two concrete questions, then check whether their scores are consistent and stable enough to trust the rubric.

Why do we report both Krippendorff’s $α$ and ICC(3,1) instead of a single agreement score?

Krippendorff’s $α$ accounts for the ordinal nature of the 6‑point scale, weighting disagreements by how far apart the ratings are. ICC treats the same data as continuous, measuring absolute consistency. Reporting both shows that the rubrics are robust under two complementary statistical views.

**Figure 7.** Annotation Interface for Human Study of Rubric Quality in the Short-form Scenario. For each rubric item, annotators answer two questions: (Q1) *Discriminative Importance*, rating how helpful the rubric is for distinguishing high- vs. low-quality document sets on a 6-point scale from Vacuous/Error to Critically discriminative; (Q2) *Omission-based Change in Set Ranking*, rating how much the ranking of document sets would change if this rubric were removed, from Unchanged to Significant change.

The image displays a research question evaluation interface. The top section provides a reference answer regarding the use of ground walnut shells as a deep-bed filter medium for oil-water separation, highlighting their oleophilic and hydrophobic properties, mechanical durability, and low density. The bottom section presents a rubric for evaluating document relevance, featuring two specific criteria: "Discriminative Importance" and "Omission-based Change in Set Ranking," each with a scale of six color-coded options ranging from "Vacuous/Error/Unclear" to "Critically" or "Significant change."

Rubric scoring reproduces across independent LLM‑judge passes with high ordering and magnitude agreement.

Doc‑Level dimensions achieve Kendall $τ$ ≥ 0.74 (short‑form) and $τ$ = 0.85 (long‑form) while Pearson $r$ reaches 1.00 (short‑form) and ≥ 0.95 (long‑form).

**Figure.** Scatter plots comparing 1st scoring vs 2nd scoring across nine different metrics: Relevance, Authenticity, Quality, Complementarity, Redundancy, Conflict, Completeness, Density, and Reachability. Each plot includes a dashed diagonal line representing perfect correlation, with Kendall's tau ($\tau$) and Pearson's correlation ($r$) coefficients displayed in the top-left corner of each panel. A legend at the bottom identifies the various models represented by distinct shapes and colors.

**Figure 10.** Reproducibility of rubric scoring across two independent passes (Long-form Scenario).

Granular Performance Analysis

Fine‑grained per‑round results and rubric‑dimension correlations.

Table 7 breaks down each reranker’s scores at the first search round, the last round, and the average across all rounds, revealing systematic trends as the retrieval process progresses.

Across rounds the gap widens especially for Doc‑Level and Global‑Level scores, while Set‑Level Redundancy stays high and Complementarity drops, indicating that later rounds favor internally consistent documents but lose diverse information.

**Figure 11.** Pearson correlation between individual rubric dimension scores and downstream generation quality (Downstream Avg). Each subplot corresponds to one of the nine evaluation dimensions across three granularity levels (Doc-Level, Set-Level, Global-Level). Each point represents a reranking method. We report the Pearson correlation coefficient (r) and p-value for each dimension. Dimensions with p < 0.05 are considered statistically significant.

The analysis underscores two bottlenecks: (1) later‑round retrieval struggles to maintain document‑level quality, and (2) existing rerankers cannot reliably assemble sets that support full reasoning chains, motivating rubric‑driven, gap‑aware selection.

Qualitative Case Studies

Case studies illustrate how the rubric reveals deficiencies missed by relevance‑only evaluation.

Tables 8 and 9 break the benchmark into fine‑grained slices, exposing where each reranker gains or loses ground.

The table presents performance metrics for various ranking methods across four domains: Health, Life Sciences, Physical Sciences, and Engineering. Each domain is further subdivided into specific categories (e.g., PH, EM, EP, NS for Health). The methods are grouped into "Adhoc Reranking," "Reasoning-Enhanced Reranking," and "Setwise Reranking," compared against an "Only Search" baseline. Cells are highlighted in yellow or pink to indicate performance variations.

The image displays a table header row containing hierarchical categories for evaluation metrics. The top-level categories are "Doc-Level", "Set-Level", and "Global-Level". Under "Doc-Level", there are columns for "Rel.", "Aut.", and "Qua.". Under "Set-Level", there are columns for "Cmp.", "Red.", and "Con.". Under "Global-Level", there are columns for "Cov.", "Den.", and "Rea.".

Two key patterns emerge. (i) Ranker separation is slice‑dependent; easy slices amplify differences while hard slices compress them. (ii) No single ranker dominates every slice, underscoring the need for multi‑slice evaluation.

To illustrate the diagnostic power of the nine‑dimensional rubric, we present four representative cases drawn from HotpotQA, TriviaQA, and ResearchQA.

**Figure 12.** Case 1 for Short-form Scenario.

**Figure 13.** Case 2 for Short-form Scenario.

**Figure 14.** Case 1 for Long-form Scenario.

Model Prompt Templates

Prompt templates for rubric‑oriented evaluation across document‑set scenarios.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers