WearableQA: A Benchmark for Health Reasoning Over Real-World Wearable Data

Ji Soo Lee, Xilun Chen, Pierce Chuang, Ashish Shenoy, Jason Wei, Dohwan Ko, Hyunwoo J. Kim, Benoit Corda

WearableQA is a diagnostic benchmark for evaluating LLM reasoning over real-world longitudinal health data.

Can current AI models accurately reason over longitudinal, real-world wearable health data to answer clinical and data-driven questions?

Current health benchmarks rely on static text or simulated signals, leaving it unclear if AI can actually interpret the noisy, multi-day physiological records of real users. The authors introduce WearableQA, a benchmark of 4,084 multiple-choice questions derived from the longitudinal wearable data, blood biomarkers, and demographics of 200 real individuals. Performance across 14 models ranges from 19.6% to 72.9%, revealing that most models struggle to derive insights directly from raw sensor measurements or integrate information across multiple signals.

Paper Primer

WearableQA uses a dual-grounding framework to ensure questions are both clinically meaningful and faithful to real-world data. The method is a discover-then-label pipeline: reasoning objectives are identified from peer-reviewed literature or statistically validated population patterns, then resolved into deterministic ground truth by running computation primitives over a user's specific longitudinal trajectory.

Most LLMs struggle with data-driven reasoning compared to health interpretation.

Except for the top-performing model, all tested LLMs achieved higher accuracy on health-reasoning tasks than on data-reasoning tasks, which require computing trends and anomalies directly from raw sensor logs. For GPT-4o, the gap is 28.2 percentage points (53.5% vs. 25.3%).

Cross-signal reasoning remains a significant bottleneck for model performance.

Models consistently perform worse on questions requiring the integration of two or more signals compared to single-signal questions, with accuracy for cross-signal prediction peaking at only 40.5% even among strong proprietary models. Proprietary models show performance drops of up to 13.4 percentage points when moving from single-signal to cross-signal tasks.

Why is this benchmark necessary if we already have medical QA datasets?

Existing medical benchmarks are primarily text-based and assess static knowledge, whereas WearableQA requires models to aggregate and interpret noisy, longitudinal physiological measurements that vary by individual.

How does the benchmark prevent models from simply memorizing answers?

The ground truth is not asserted a priori; it is deterministically computed from each user's specific data, and the benchmark uses rigorous statistical gates and audit loops to ensure questions are non-trivial and resistant to superficial shortcuts.

Introduction and Motivation

We expose the missing evaluation of AI reasoning over real longitudinal wearable records.

Current health benchmarks treat data as static snapshots, ignoring the noisy, multi‑modal, and longitudinal nature of real‑world wearable recordings.

AI systems need to be tested on the same kind of continuous, noisy physiological streams that users actually generate, not on synthetic or single‑timepoint data.

The key gap is that static medical benchmarks cannot assess models’ ability to reason over dynamic, longitudinal wearable data.

The WearableQA Benchmark

We detail the WearableQA benchmark, its construction, and the experimental protocol.

WearableQA evaluates models on reasoning over real‑user longitudinal wearable records, enriched with blood biomarkers and demographic data.

How does WearableQA differ from existing health‑related benchmarks?

Unlike text‑only or synthetic time‑series benchmarks, WearableQA uses real longitudinal wearable data together with blood and demographic information, and provides deterministic ground truth through a dual‑grounding process rather than relying on ambiguous or manually annotated answers.

Collect wearable time‑series, blood biomarkers, and demographic attributes for 200 users over several hundred days.

Apply the dual‑grounding framework to generate deterministic answers for each of the 4,084 multiple‑choice questions.

Organize the questions into the 16‑type taxonomy spanning data vs health reasoning and single‑ vs cross‑signal complexity.

Split the question set into standard evaluation splits (train/validation/test) while preserving the distribution of reasoning and signal categories.

Evaluate each model by computing accuracy on each split and compare against a 10 % chance baseline.

Analyze results by aggregating performance across the two axes to pinpoint specific reasoning or integration weaknesses.

Compute the linear regression slope of the series: slope ≈ (74 − 62) / 6 = 2 bpm/day.

Since the slope is positive and exceeds the predefined threshold of 0.5 bpm/day, classify the trend as “increasing”.

Assign the correct multiple‑choice answer label “increasing” as the deterministic ground truth.

This concrete computation shows how deterministic ground truth is derived from raw measurements, eliminating ambiguity that would arise from subjective labeling.

**Figure 3.** Overall taxonomy of WearableQA. Taxonomy for 4,084 questions of real-world user data that span blood, demographics, and time-series wearables. The benchmark covers 16 question types of data-reasoning and health-reasoning.

**Figure 4.** Overall statistics of 200 real users from WearableQA.

**Figure 5. Overview of WearableQA.** From real users' wearable time series, blood, and demographics, we generate questions under dual grounding (peer-reviewed literature findings and statistically verified population-grounded patterns), compute deterministic ground truth from the raw measurements, organize them by a (2 × 2) reasoning × signal taxonomy, and apply validation loops for quality control.

Data and Question Construction

We detail how WearableQA questions are built, from data collection to dual grounding and validation.

Constructing WearableQA required a disciplined pipeline that starts from real user data, defines a taxonomy, generates questions, labels answers deterministically, and validates each instance against strict statistical criteria.

Each question’s answer is derived either from a peer‑reviewed study (literature grounding) or from a statistically validated pattern mined from a large cohort (population grounding), guaranteeing both clinical relevance and data‑driven authenticity.

How does Dual Grounding differ from relying solely on literature references?

Literature alone provides clinically validated relationships but may miss patterns that only appear in large‑scale wearable data. Dual Grounding augments those findings with population‑derived patterns that have passed rigorous statistical gates, ensuring the benchmark covers both established knowledge and emergent real‑world behavior.

Collect longitudinal wearable records and 17‑biomarker blood panels from 200 real users.

Define the 2 × 2 taxonomy (reasoning type × signal complexity) and the orthogonal grounding axis.

Generate candidate questions from (i) literature findings and (ii) mined population patterns.

Apply Dual Grounding: assign each candidate a source label (literature or population) and enforce source‑specific validation criteria.

Run deterministic computation primitives (e.g., `lagged_correlation`, `threshold_flags`) to label the correct answer for each user trajectory.

Construct nine distractors per question, ensuring only the gold option satisfies the computation.

Validate each instance with effect‑size, robustness, and authenticity gates; discard any that fail.

Perform shortcut auditing by model‑generated review and human verification, then remove any leaked or trivial cues.

Compute `lagged_correlation`(`max_lag`=3) → correlation at lag 0 = 0.62, satisfying |$\rho$| ≥ 0.5.

Bootstrap 30 resamples of the 28‑day window; 27 of them retain a correlation ≥ 0.5, giving 90 % stability (≥ 80 % required).

Leave‑one‑out removal of any single day leaves the correlation above 0.55, confirming robustness.

Cross‑user null test on mismatched pairs yields an average correlation of 0.12; the false‑discovery rate is 0.12 / 0.62 ≈ 0.19 ≤ 0.20, passing the authenticity gate.

Deterministic program flags the question as “positive coupling” and assigns the corresponding answer option.

This example shows why multiple statistical gates are needed: a raw correlation alone could be spurious, but bootstrap stability, leave‑one‑out robustness, and a low FDR together certify a genuine physiological pattern.

**Figure 10.** Example question from WearableQA for data reasoning. For readability, the wearable trajectory is visualized as a plot in this figure; models receive the underlying measurements in serialized textual form.

**Figure 11.** Example question from WearableQA for health reasoning. For readability, the wearable trajectory is visualized as a plot in this figure; models receive the underlying measurements in serialized textual form.

Benchmark Performance Results

We evaluate models on WearableQA, revealing large gaps between proprietary and open‑source performance.

The WearableQA benchmark tests models on longitudinal, multi‑modal health data. Here we report how different model families perform under the default chain‑of‑thought protocol.

Proprietary models achieve substantially higher overall accuracy than open‑source models on WearableQA.

Figure 2 and Table 1 show Gemini‑3.1‑Pro at 72.9 % versus Llama‑3.2‑3B at 19.6 %.

CoT prompting asks the model to generate a step‑by‑step reasoning trace before producing the final answer, turning a single prediction task into a structured reasoning process.

Definitional pairs are signal relationships that are universally true (e.g., step count ↔ active energy), while empirical pairs depend on the individual’s measured data (e.g., resting heart rate ↔ stress level).

**Figure 1.** Radar plot for performance of Open-Source models (left) and proprietary models (right) on our WearableQA Benchmark.

**Figure 2.** Overall accuracy by model (CoT, 95% CI).

**Figure 6** Accuracy along the two taxonomy axes (CoT). Each point is a model (95% CI); blue = proprietary, green = open-sourced, dashed line is $y = x$. (a) Most of the models sit above $y = x$ (health > data): models reason better about clinical health than about underlying data. (b) Cross-signal questions remain challenging across model scales, while the challenge appears clearer for stronger models (models sit below $y = x$ (single > cross)), while weaker models perform substantially low for both subsets.

**Figure 8** Answer-letter pick-rate vs. uniform ground truth (DA left, CoT right). Cells show each model's pick-rate (%) per letter (red = over-picked, blue = under-picked, white = 10); side panels give total-variation distance from uniform. Solid rule = ground-truth row, dashed rule = proprietary vs. open-sourced. Bias grows from CoT to DA and from larger to smaller models; Llama-3.2-3B under DA collapses onto one letter (51%), while CoT restores near-uniform.

Proprietary models significantly outperform open‑source models, but all struggle with empirical physiological coupling.

Evaluation Protocol and Ablations

Details of evaluation protocol, prompt formats, and time‑series representations used in the ablations.

Evaluation uses exact‑match scoring: the model receives a user’s demographics, wearable time series, blood panel, and cohort reference, then must output a single letter (A–J) parsed from the response.

The prompt consists of a fixed system instruction followed by five identical context blocks and a trailing answer‑format directive that differs between the Chain‑of‑Thought (CoT) and direct‑answer conditions.

In the CoT condition the trailing directive reads “Think step by step… Answer: X”, while in the no‑CoT condition it reads “Choose the single best answer… Answer:”. The parser looks first for “Answer: X”, then falls back to a lone final letter.

**Figure 9.** The assembled evaluation prompt. Under the no-CoT (Direct Answer) condition, the final instruction is replaced by “Choose the single best answer. Respond with ONLY the letter (A-J). Do not explain or output anything else. Answer:”.

Time‑series data can be presented in four textual serializations (row, column, markdown, CSV) plus two hybrids (markdown+stats, csv+chart); all preserve identical numeric values and column labels.

Row format lists each day as a single line of key=value pairs, emphasizing same‑day cross‑signal reading; column format groups each metric into a dated series, highlighting within‑signal temporal trends.

Markdown and CSV formats render the data as a grid, differing only in markup syntax; the hybrid conditions augment the best‑performing text format with summary statistics or a rendered plot, respectively.

Supplementary Visualizations

Supplementary prompts, examples, and visualizations for the WearableQA benchmark.

This appendix gathers the concrete prompts, example queries, and visual tables that underpin the WearableQA evaluation suite.

The evaluation prompt follows a row‑format with an explicit chain‑of‑thought (CoT) instruction: the model receives a user profile, daily wearable sensor readings, a blood biomarker panel, cohort reference statistics, and a multiple‑choice question, then must reason step‑by‑step before outputting the final answer letter.

A “strongest coupled pair” query asks the model to identify, within a 28‑day window, which two metrics exhibit the highest day‑to‑day correlation (allowing up to a three‑day lag), using a population‑grounded reference.

The “differential diagnosis” query presents a user’s recent wearable record together with blood markers and asks which single clinical diagnosis best matches the data, again grounding the decision in population statistics.

The appendix also distinguishes two layout styles for visualizing signals: “image‑grid” shows each signal in its own subplot (5 × N arrangement), while “image‑grouped” aggregates signals into range‑grouped plots.

**Table 7.** Comparison of chain-of-thought (CoT) and direct-answer (DA) prompting on WearableQA. Accuracy (%) is reported for each model. Gap denotes CoT - DA in percentage points, where positive values indicate an improvement from CoT prompting.

This figure displays five time-series line charts representing physiological and activity metrics over time, from late 2023 to early 2025.

The figure displays three time-series line charts tracking health metrics from December 30, 2023, to May 27, 2025. The top panel compares "Ex Avg HR" (blue) and "Bedtime Std(min)" (orange). The middle panel shows "Active Cal" (blue) and "BMR Cal" (orange). The bottom panel tracks "Steps" (blue).

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers