Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation
Koutian Wu, Junjie Zhou, Ergan Shang, Jiayu Wang, Pengqian Han, Junkai Wang, Wanghan Xu, Lin Shi
Benchmark Radar is a living search engine and database that aggregates AI benchmarks, model scores, and evaluation evidence.
How can we systematically discover, catalog, and search the fragmented landscape of AI benchmark evaluations and their associated model scores?
Researchers evaluating AI models must manually search across disparate paper servers, code repositories, and vendor blogs to find relevant benchmarks and their specific evaluation settings. Benchmark Radar automates this discovery by collecting benchmark artifacts and scores into a unified, searchable catalog that retains source identities and citations. The system provides a web dashboard and command-line interface (CLI) for querying 1,283 benchmark records and over 12,000 numeric score observations.
Paper Primer
The system functions as a living index: it combines daily automated discovery of new benchmark releases with a curated catalog of existing evaluations. By normalizing identifiers across sources like arXiv, GitHub, and Hugging Face, it allows users to trace a benchmark from its initial paper to its subsequent use in model reports.
Benchmark Radar provides comprehensive coverage of the current evaluation landscape.
The catalog contains 1,283 source records, including 790 records with 12,916 numeric score observations and 493 records that link to task materials despite lacking scores.
The system separates candidate retrieval from suitability judgment. Users can query the catalog to find benchmarks matching specific modalities or tasks, then inspect the linked evidence—such as repositories or datasets—to determine if the evaluation is appropriate for their specific research needs.
Why is a centralized search engine necessary for AI benchmarks?
Existing benchmark registries and leaderboards are fragmented, often requiring researchers to consult multiple separate resources to connect a benchmark to its task materials, code, and reported scores.
What is the scope of the data included in the catalog?
The catalog covers 11 top-level domains including LLM evaluation, agentic and tool-use benchmarks, coding, reasoning, and domain-specific evaluations, with records preserved even when scores or citations are absent.
Introduction: The Fragmentation of AI Benchmarks
We expose the fragmented benchmark landscape and introduce Benchmark Radar as a unified searchable catalog.
Benchmark data for LLMs and AI systems is scattered across model cards, research reports, code repositories, and dataset hubs, making it hard to locate relevant evaluations or reproduce baselines. This fragmentation forces researchers to query multiple disparate resources, slowing progress and risking inconsistent comparisons. Benchmark Radar addresses this gap by continuously harvesting benchmark artifacts and exposing them through a unified, searchable catalog.
A living database that continuously gathers benchmark artifacts—from papers, repositories, and model cards—and makes them instantly searchable.
**Figure 1.** Benchmark Radar connects discovery, retrieval, and source inspection.
The shift from centralized benchmarks to fragmented, source‑specific reporting hampers reproducibility and discovery.
System Architecture and Data Collection
We unify fragmented benchmark data via source records and discovery snapshots, enabling consistent daily collection and retrieval.
Because benchmark data lives in disparate places—model cards, reports, repository releases—the system must constantly reconcile new mentions and updates without inflating the benchmark total.
A source record is the canonical entry that captures a single benchmark observation from any contributing source, normalizing its identifiers, scores, and provenance.
Both entries are parsed; the model name “GPT‑Mini” is normalized to the canonical identifier $m_{GPT\_Mini}$.
Entry 1 creates a source record $r_1$ storing the BLEU score 27.5 and a link to the model card URL.
Entry 2 creates a source record $r_2$ storing only the release URL; the score field is left empty.
Identity linking recognizes that $r_1$ and $r_2$ refer to the same model $m_{GPT\_Mini}$ and adds a bidirectional link.
Even without a score, a record contributes provenance and can later be enriched when a score becomes available.
How does a source record differ from a traditional benchmark entry?
A source record is a unified, schema‑normalized representation that may lack any score, whereas a traditional entry often assumes a complete set of metrics; the record’s purpose is to preserve whatever evidence is available, not to force a full evaluation.
A discovery snapshot captures the raw daily observations of mentions, releases, and updates before they are folded into source records.
The discovery engine records the arXiv preprint as observation $o_1$ with identifier “2101.00123”.
The repository URL is recorded as observation $o_2$ with identifier “github.com/example/model”.
Both observations are bundled into a snapshot $s_{2023-03-01}$ and stored with a UTC timestamp.
Later, $o_1$ may become a source record once a score is reported; $o_2$ may remain a raw artifact if no benchmark is attached.
Snapshots preserve the exact moment of discovery, allowing auditors to see what evidence existed before any downstream processing.
Why does a discovery snapshot not increase the benchmark catalog total?
Because snapshots only store raw observations without a quantified score; the catalog total counts only scored benchmark entries, so snapshots remain auxiliary evidence.
Define a rolling 48‑hour window and schedule a collection run.
Query each core source (arXiv, Hugging Face Hub, GitHub Search) and the 24 auxiliary feeds.
Parse incoming artifacts into raw observations and group them into a discovery snapshot.
Normalize observations into source records, linking identities where possible.
Remove any rows dated in the future to avoid premature publication.
Verify that all core sources have reported health metrics; abort if any core source fails.
Publish the updated catalog and generate a daily brief for offline consumption.
**Table 1.** Catalog sources in the v0.11.0 release. Rows count source-specific benchmark records, including records without scores.
**Figure 2.** Two input paths support publication: dated discovery snapshots preserve mention and release evidence, while benchmark registries and model reports populate the shared catalog. Clients use the catalog's stable source-record IDs.
**Table 2.** Reader questions and evidence scope.
CLI workflow for initializing, syncing, and serving the catalog.
The full‑population audit enumerates every source record, counts distinct model IDs and cited documents, and respects the original observation timestamps without applying any date or score filters.
Catalog Coverage and Empirical Findings
Catalog aggregates over a thousand benchmark records, revealing coverage gaps and source distribution.
The rebuilt catalog unifies 1,283 source records from four primary sources, offering a comprehensive benchmark view.
Table 3 details the breakdown of records, scored entries, and numeric scores per source.
Catalog Coverage measures how many benchmark records the catalog captures and how completely each record is annotated with scores and links.
**Table 3.** Score coverage across the full catalog: 790 records have numeric observations and 493 do not. Scores count observations, not distinct models.
**Figure.** A 3D visualization showing the progression of LLM performance over time. The horizontal axis represents the release date (from 2024 to 2026), the depth axis represents the reported score (0-100), and the vertical axis represents the number of models with reported scores. Specific models such as "Humanity's Last Exam," "CritPt," "APEX-Agents," and "HorizonMath" are highlighted with circles.
**Figure 4.** Inspecting Humanity’s Last Exam in Benchmark Radar. This Artificial Analysis record contains 577 reported scores; the highest displayed score is 55.47. Marks identify model organizations, and the line connects successive best scores ordered by model release date. Click the image to open the interactive view.
**Figure 5.** Capability taxonomy over the full catalog. Ribbon height is one unit per source record, so both columns carry all 1,283 records. Level 1 is one primary label per record; interaction paradigm and modality are recorded as separate facets and are not drawn here.
**Figure A.** Release year of all 1,283 records, by Level 1
**Figure 6.** Release-year composition of the classified catalog. Panel A holds every record: 615 on the year axis and 668 in the undated column, which carries its own scale. Panel B reports shares only for years with at least 30 dated records and a source mix close to the pooled one; the excluded years hold 108 dated records between them and remain counted in Panel A. Series marked facet are cross-cutting properties rather than Level 1 classes, so the lines in Panel B overlap and do not sum to 100%.
**Figure 7.** Discovery observations by source through 2026-09-07. Repeat sightings count as observations.
**Figure 8.** Discovery trends through September 7, 2026. Category cards show newly surfaced releases and track updates separately; the bars show daily evidence and attention volume. Category tags overlap. The visible coverage notice reports failed sources. Click the image to open Trends.
The figure displays a breakdown of 1,283 records, categorized into four groups: LLM Stats (687 records), OpenCompass Hub (461 records), Artificial Analysis (25 records), and Model reports (110 records). Each group is represented by a grid of squares, where blue squares indicate "Numeric score recorded" and grey squares indicate "No numeric score recorded". The total count is summarized as 790 records with scores and 493 without, with a note stating that 464 of the 493 unscored records still link to a paper, repository, or dataset.
The distribution of benchmark records across the four primary sources shows LLM Stats dominates record count while Artificial Analysis provides the bulk of numeric scores.
Practical Use Case: Prior-Art Checking
Illustrates the Prior‑Art Check workflow and its catalog results with a concrete example.
The central premise—that fragmented AI evaluation data hampers reproducible benchmarking—remains true. This section shows how the Prior‑Art Check operates on a concrete workflow.
The Prior‑Art Check queries the unified catalog to see whether a proposed evaluation already exists, preventing redundant benchmark creation.
The client issues a similarity query; the catalog returns three candidate records (Figures 11–13).
For each record the contributor inspects the boolean fields “`has_paper`”, “`has_repo`”, and “`has_dataset`”.
Record 11 lacks a paper and dataset, Record 12 includes a paper and repo, Record 13 shows a high retrieval score (10.82) and matches the credit‑assignment focus.
Seeing that Record 12 already covers the desired mechanism, the contributor decides the new benchmark would be redundant and aborts creation.
The Prior‑Art Check saves weeks of work by surfacing existing implementations that would otherwise be recreated.
How does the Prior‑Art Check differ from a simple keyword search over titles?
Keyword search only matches literal strings, missing semantically similar works that use different terminology. The Prior‑Art Check leverages the catalog’s structured metadata and a learned similarity metric, so it can retrieve records that share the same credit‑assignment mechanism even if the wording differs.
The survey uncovered five recent credit‑assignment works that use Qwen models.
Table 4 lists five entries spanning SRPO, ContextPilot, SkillGate, CIPO, and MoRSE.
**Figure 11.** A retrieved discovery record with a repository link for inspecting the implementation. Paper and dataset links are missing from the record.
**Figure 12.** A second discovery record provides both paper and repository links. Its dataset link needs a follow-up check.
**Figure 13.** A candidate record with query-word matches and retrieval-score components for the agent to inspect when judging relevance.
Limitations and Future Directions
We recap why fragmented benchmark data matters, then outline the catalog’s current limits.
AI evaluation data is scattered across model cards, reports, and posts, making it hard to locate benchmarks or reproducible baselines; Benchmark Radar unifies these sources into a searchable catalog.
The current catalog is limited by collection constraints, failed requests, missing identifiers, and inconsistent snapshot dates, which together reduce coverage and obscure the true number of distinct tests.
Collection limits are the practical boundaries that prevent the catalog from capturing every benchmark record—missing data, request failures, and mismatched timestamps all truncate the observable universe of evaluations.
Despite these gaps, Benchmark Radar already offers a unified interface for daily benchmark discovery, searchable catalog queries, model‑report linking, and score‑history tracking, enabling users to locate evaluations, task materials, and citation trails efficiently.
Appendix: Registry Evidence
Provides detailed statistics on documents, models, scores, and dates across the catalog.
The catalog contains $1,208$ distinct cited documents linked to $1,278$ of the $1,283$ benchmark records; five records lack any document citation. Model identifiers are present for every numeric score in the $790$ scored records, yielding $868$ distinct model identities.
**Table 5.** One cited page can document hundreds of scored models. Models count distinct source model IDs within a record; documents count distinct citation IDs; scores count numeric observations. These selected examples illustrate the units. The census retains all source records.
Only $82$ of the $790$ scored records declare a percentage unit with values bounded between $0$ and $100$; the remaining $708$ use other or unverified scales, preventing a unified percentage‑headroom calculation.
**Table 6.** Only 82 of 790 scored records meet the percentage-scale rule in Section 3.5. All 1,283 records remain accounted for.
Documentation analysis finds $1,208$ distinct cited documents supporting $1,278$ records; the five records without a document illustrate incomplete coverage, and $1,171$ documents lack organization metadata.
**Table 7.** Documentation across the complete v0.11.0 catalog. “With docs” counts benchmark records; the last two columns count distinct documents. Missing organization metadata remains unknown.
Score analysis retains all $1,283$ records and $12,916$ numeric observations; $22$ benchmark records contain scores for at least $100$ distinct models, and the export preserves raw maxima, tied scores, units, and missing‑value flags.
**Table 8.** Reported scores across every source in the v0.11.0 release. Models count distinct source model IDs within each benchmark record. Medians and maxima use records with known model counts; unscored records are not assigned zero models. Model counts are not summed across benchmarks. The record-level score export contains every catalog record, including raw maxima, all tied reporting models and citations, units, settings, and explicit missing values.
Date evidence shows $615$ records with known release dates, while $668$ lack this information; score observations carry $12,594$ model‑announcement dates and $322$ document‑publication dates, which serve as proxies rather than true evaluation dates.
**Table 9.** Date evidence for the complete v0.11.0 catalog. The first two numeric columns count benchmark records; the final three count numeric score observations. Records with no scores remain in the Records column.
Appendix: Collection and Filtering
Details on collection settings, display filters, source inventory, and connector health.
The 2026‑09‑07 snapshot recorded 1,003 fetched rows, which after duplicate removal left 954 candidates; of these, 366 qualified for publication and 138 met the recommendation threshold.
Published evidence retains source URLs, retrieval times, parser versions, identifiers, and payload checksums, while raw responses and credentials remain outside the public artifact.
Benchmark Frontier on the Leaderboard tab requires a numeric score, excludes known pre‑2024 benchmarks, and applies an initial score cutoff of 70; records with unknown dates or counts are kept but excluded from Pareto calculations.
General catalog search and export operations retain the full population, and the census reported in this paper applies none of these display filters.
A daily‑recommendation priority score combines relevance (35 %), evidence (20 %), recency (20 %), and adoption (25 %) on a 0–100 scale, guiding agents to focus on high‑value discovery observations.
Benchmark Radar ingests from 37 sources: 13 direct connectors and 24 first‑party feeds, each monitored for health and eligibility.
**Table 10.** Healthy direct discovery connectors at the current cutoff.
A connector is considered healthy if its request succeeds and the response can be parsed, even when it yields zero eligible rows.
Core sources covering papers, model or dataset artifacts, and code must be healthy before any publication; failures in optional sources are recorded but do not block publication.
**Table 11.** The 24 first-party research and engineering feeds monitored by Benchmark Radar.
Appendix: Extended Use Case
Illustrates how a coding agent uses Benchmark Radar to verify a new evaluation against prior work.
The contributor tasked a coding agent with checking whether a proposed evaluation duplicated existing work, focusing on credit‑assignment research from August 2026 that used small Qwen‑series models as reproducible baselines. The agent installed the Benchmark Radar CLI, downloaded the local corpus, and ran offline queries to retrieve candidate records, then enriched the search with web look‑ups to capture related work described under alternative terminology. After inspecting the retrieved discovery records and their links, the agent assembled a comparison table (Table 12) that highlights design‑dimension overlaps across eleven benchmarks.
```bash $ benchmark-radar --help usage: benchmark-radar [-h] [--author-output AUTHOR_OUTPUT] [--author-contacts AUTHOR_CONTACTS] [--author-repo-limit AUTHOR_REPO_LIMIT] [--author-contributors AUTHOR_CONTRIBUTORS] [--author-emails AUTHOR_EMAILS] [--config CONFIG] [--output OUTPUT] [--json-output JSON_OUTPUT] [--social-output SOCIAL_OUTPUT] [--items ITEMS] [--repo REPO] [--channels CHANNELS] [--existing-body EXISTING_BODY] [--since SINCE] [--until UNTIL] [--snapshot-dir SNAPSHOT_DIR] ``` The CLI has two groups: Consumer (query) commands – for searching and inspecting local benchmark data: - search "<query>" --scope catalog|radar|all --json – discover benchmarks (catalog = verified records, radar = recent unverified evidence like papers/repos/datasets) - show "<key-or-slug>" --json – inspect one record in detail - recent --json – newest Radar evidence - status --json – local data health/provenance - init / sync – set up or refresh the local data - serve --host 127.0.0.1 --port 8765 – local HTTP interface for the same queries Maintainer commands – for operating the radar pipeline (not for ordinary use): run, rebuild, backfill, migrate, rescore, simulate-history, export, classify, authors, social, normalize-external, build-data-release. For your local benchmark search use case, you’ll mostly need search, show, recent, status, and serve.
**Table 12.** A prior-art comparison table assembled manually for an earlier benchmark-design effort. Each row is one related benchmark and each column one design dimension, all read from the source papers by hand.
Appendix: Reproducibility and Access
How to reproduce the catalog build and locate its permanent artifacts.
This revision builds on Benchmark Radar v0.11.0 (commit 8f46bbf) with a discovery cutoff of 2026‑09‑07. A fresh detached checkout passed the six CI steps in order: linting, formatting checks, catalog normalization, classification, release construction, and tests. The August snapshots in data/leaderboard snapshots.yml remain the registry inputs; the September cutoff does not trigger a recrawl.
The data build normalizes the registry and model‑report inputs into a shared catalog, then client tools validate and classify discovery tracks and package the catalog with checksums. The project publishes a DOI and a Citation File Format record for proper citation. Table 13 (see below) enumerates the permanent locations of all artifacts.
This table lists various project artifacts and their corresponding canonical or permanent locations.
The repository includes scripts/`audit_catalog`.py, which validates IDs, counts, and exports catalog-data.tex and evidence/catalog-audit.json. Its JSON output records every source‑record key, measurement state, per‑record model and document counts, citation IDs, and SHA‑256 hashes, and checks score‑observation uniqueness, record‑to‑shard identity, and document‑registry coverage.
To reproduce the results, start from a clean checkout at commit 8f46bbf, run the README’s six‑step CI sequence (including --check mode), then build with make arxiv to generate the PDF; the software exporter also writes figure-data.tex, and the findings audit produces findings-data.tex, evidence/catalog-findings.json, and a CSV export. All analyses retain the catalog’s source‑record IDs, and table cells and numeric claims are generated from these same inputs. The v0.9.0 DOI remains an earlier deposit, while live dashboard counts require a retrieval date.