A Common Measure of Communication for Speech Brain–Computer Interfaces

Dulhan Jayalath, Benjamin Ballyk, Oiwi Parker Jones

OVMI provides a common information-theoretic scale to compare heterogeneous speech BCI systems.

How can we compare the performance of speech BCIs that use different vocabularies and experimental settings on a single, unified scale?

Speech brain–computer interfaces (BCIs) are difficult to compare because they use different datasets, recording modalities, and vocabularies, making standard metrics like accuracy or word error rate (WER) misleading. The authors derive Open-Vocabulary Mutual Information (OVMI), an information-theoretic measure that weights a decoder's in-vocabulary performance by the probability that an intended word is actually supported by the system. This allows researchers to evaluate disparate systems against a common reference distribution, revealing that small-vocabulary systems often overstate their communicative capability relative to broad language targets.

Paper Primer

OVMI is a component of mutual information: it measures the information conveyed among supported words, weighted by the lexical coverage of the decoder relative to a target reference distribution. By separating the decoding fidelity from the vocabulary's representational reach, it exposes the trade-off between supporting frequent words and maintaining high decoding accuracy.

OVMI enables meaningful comparison of heterogeneous BCI systems on a common scale.

The authors evaluated invasive and non-invasive decoders against four reference distributions (broad English, conversation, AAC, and narrative), showing that system rankings shift depending on the intended communication domain. Small-vocabulary systems show high sensitivity to the reference distribution, with one system's information transfer increasing from 6.4% to 40.4% when switching from broad English to an AAC-specific target.

OVMI serves as an effective objective for vocabulary selection.

Using a contrastive decoder, the authors selected vocabularies of varying sizes by maximizing OVMI rather than frequency or validation accuracy. This approach yielded peak relative improvements in accuracy of up to 16.3% across three speech domains.

Why do standard metrics like accuracy or WER fail to capture the true communicative capability of a BCI?

These metrics are conditional on the decoder's supported vocabulary; they measure how well a system decodes words it already knows, but they ignore whether those words actually represent what the user intends to communicate.

Does a higher OVMI score imply a more clinically useful BCI?

No. OVMI measures lexical information transfer, but it does not account for experimental setting, latency, or the practical utility of the interface, such as whether the decoding task (e.g., perceived speech) is suitable for a real-world communication device.

The Problem of Incomparable Metrics

We expose why current BCI metrics cannot compare heterogeneous speech systems.

Speech BCIs promise restored communication, yet the field lacks a common metric because studies differ in vocabularies, datasets, and recording methods, making reported scores incomparable.

Because each study evaluates on its own vocabulary and dataset, reported accuracy or word‑error‑rate only reflect performance within that limited scope, preventing direct comparison across studies.

**Figure 1.** OVMI maps incomparable evaluations onto a common scale. Speech BCI studies differ in experimental setting, supported vocabulary, and test data, so their reported scores are not comparable. Each study provides a decoding score over its supported vocabulary $S$, from which we estimate the information conveyed within that vocabulary, $I$. We define a common reference distribution $p(x)$ over the words a user may wish to communicate, and let $C(S)$ denote the probability that a word from $p(x)$ is supported by the system. OVMI weights the decoded information $I$ by $C(S)$, yielding a score that measures performance against $p(x)$. Evaluating all systems against the same $p(x)$ therefore allows comparing heterogeneous speech BCI studies on a common communication scale.

Current BCI metrics fail to capture real‑world utility because they ignore lexical coverage and the user’s intended communication distribution.

Defining Open-Vocabulary Mutual Information

Introducing a metric that unifies decoding accuracy and vocabulary reach.

Standard BCI metrics such as ITR ignore whether a user’s intended word is even representable by the decoder, conflating decoding fidelity with vocabulary gaps.

ITR quantifies bits conveyed per trial under a uniform‑prior, symmetric‑error model.

Lexical coverage is the probability that a word a user wishes to say lies inside the decoder’s vocabulary.

OVMI measures how much lexical information is transferred per word the user actually wants to say, weighting mutual information by the chance the word is supported.

Compute coverage: $C(S)=0.4+0.3=0.7$.

Normalize the distribution restricted to $S$: $p_S(w_1)=0.4/0.7\approx0.571$, $p_S(w_2)=0.3/0.7\approx0.429$.

Entropy of $X$ conditioned on $X\in S$: $H(X\mid X\in S)= -0.571\log_2 0.571 -0.429\log_2 0.429 \approx 0.985$ bits.

OVMI = $C(S)\times H(X\mid X\in S)=0.7 \times 0.985 \approx 0.69$ bits.

Even with perfect decoding on the covered words, the overall information is limited by the 30 % of language that the decoder cannot represent.

Empirical Results and Benchmarking

OVMI‑guided vocabulary selection yields up to 16.3% relative improvement over frequency‑based methods.

OVMI‑guided vocabulary selection outperforms frequency‑based selection, achieving up to 16.3% relative OVMI gain at $V\!=\!250$ across domains.

Figure 6 reports peak relative improvements of 15.4 % (TIMIT), 16.3 % (Podcasts), and 8.4 % (Sherlock) when vocabularies are chosen by maximizing OVMI.

The estimator treats the decoder as a symmetric $V$‑ary channel with a single macro accuracy $P$, assuming all errors are uniformly distributed among the $V\!-\!1$ wrong words.

How does this scalar estimator differ from simply reporting macro accuracy?

Macro accuracy alone ignores the non‑uniform source distribution $p$ and the effect of lexical coverage $C(S)$. OVMI incorporates both by weighting $p$ through $C(S)H(q_S)$ and subtracting the channel entropy $h_V(P)$, thus measuring actual information transfer rather than raw correctness.

OVMI allows for consistent comparison across diverse BCI studies.

Mathematical Foundations

Formal proofs of OVMI for in‑vocabulary decoders and related corollaries.

The central premise—OVMI unifies decoding accuracy and lexical coverage—still underpins every result below.

Conditioning on the intended word being in the deployed vocabulary $S$, the in‑vocabulary OVMI is $$\text{IOVMI}(S)=C(S)\Bigl[H(q_S)-\sum_{x\in S}p_S(x)\,H\bigl(K_S(\cdot\mid x)\bigr)\Bigr],$$ where $p_S$, $K_S$, and $q_S$ are the renormalised input distribution, the decoder channel restricted to $S$, and the induced output distribution respectively.

Start from the definition $ \text{IOVMI}(S)=C(S)I(X;Y\mid X\in S)$.

Express the conditional input distribution.

Write the conditional channel.

Derive the output distribution.

Apply the mutual‑information identity.

Multiply by $C(S)$.

When the decoder channel is homogeneous and symmetric with correct‑decoding probability $P$, the in‑vocabulary OVMI simplifies to $$\text{IOVMI}(S)=C(S)\bigl[H(q_S)-h_V(P)\bigr],$$ where $h_V(P)=-P\log_2P-(1-P)\log_2\frac{1-P}{V-1}$ and $q_S(j)=P\,p_S(j)+\frac{1-P}{V-1}$.

Specify the homogeneous symmetric channel.

Compute the row entropy.

Insert into Proposition 2.

Allowing each word $x\in S$ its own correct‑decoding probability $P_c(x)$ yields $$\text{IOVMI}(S)=C(S)\Bigl[H(q_S)-\sum_{x\in S}p_S(x)\,h_V\bigl(P_c(x)\bigr)\Bigr],$$ with $q_S(j)=p_S(j)P_c(j)+\frac{1}{V-1}\sum_{x\neq j}p_S(x)\bigl(1-P_c(x)\bigr)$.

Define the word‑dependent channel.

Substitute into Proposition 2.

Three immediate facts follow from the rearranged output formula $q_S(j)=\frac{1-P}{V-1}+\frac{P}{V-1}V\,p_S(j)$: (1) uniform $p_S$ yields uniform $q_S$; (2) chance performance $P=1/V$ forces uniform $q_S$ regardless of $p_S$; (3) for informative decoders ($P>1/V$), any non‑uniformity in $p_S$ propagates to $q_S$, lowering its entropy below $\log_2V$.

The table presents a comparison of evaluation types and their associated costs in relation to specific research questions.

This table lists symbols used in the paper, their descriptions, and where they are introduced. The columns are "Symbol", "Description", and "Introduced in".

Experimental Setup and Data

Details on data splits, candidate vocabulary construction, and evaluation procedures.

We evaluate on subject 0 from LibriBrain100 and report matched‑domain results for TIMIT, Podcasts, and Sherlock, using reference word distributions estimated solely from each domain’s training split.

Podcasts sessions 1–28 form the training set, session 29 is used for validation, and session 30 for testing.

Sherlock uses all training runs for training, session 11 for validation, and session 12 for testing.

TIMIT follows the canonical LibriBrain100 policy: 50 development speakers provide validation data, while the 24 core‑test speakers constitute the test set.

Repeated SA utterances and any training speaker that shares an SX sentence with a core‑test speaker are excluded from both validation and test splits.

MOCHA‑TIMIT training sessions A and D contribute only to decoder training and to the availability of candidate words.

The candidate pool is built from word tokens that have usable neural training windows after discarding incomplete examples; a word must appear at least five times in the pooled neural training data.

Eligible words are sorted by decreasing training count and the pool is capped at the top 250 words, a pool that is shared across all domains, selection methods, vocabulary sizes, and random seeds.

We train five independently initialised instances of the contrastive brain‑to‑T5‑embedding decoder for 50 epochs with a batch size of 128, selecting the checkpoint that maximises pooled validation contrastive top‑1 accuracy.

The model has no vocabulary‑specific classification head; after training we create a normalised target embedding for each candidate word and cache the cosine similarity between every neural prediction (validation or test) and all 250 candidates.

Evaluating any vocabulary therefore reduces to restricting the cached score matrix to the desired columns and taking an argmax—no retraining of the neural model is required for different methods or vocabulary sizes.

For a domain $d$, the empirical probability $p_d(w)$ of word $w$ is computed over the full training‑text distribution; lexical coverage of a vocabulary $V$ is the sum of these probabilities for words in $V$.

When every word in the trial vocabulary has at least five validation examples we use the full confusion‑channel estimate for OVMI; otherwise we fall back to the per‑word symmetric‑error approximation.

At a vocabulary size of 150, the fallback was applied to all five TIMIT seeds and to Sherlock seeds 0, 3, and 5; all other domain‑size‑seed combinations employed the full validation channel.

For each domain and vocabulary size we compare OVMI against Frequency using a one‑sided paired randomisation test with $B = 100{,}000$ Monte Carlo permutations, computing the raw p‑value as $p_{\text{raw}} = \frac{1 + \sum_{b=1}^{B} \mathbf{1}[T_b \ge T_{\text{obs}}]}{B + 1}$.

Implementation Details

Reference implementation of the OVMI computation used in the experiments.

Holm’s step-down correction is applied across the nine vocabulary sizes within each domain, controlling the error rate separately for TIMIT, Podcasts, and Sherlock.

Reference implementation of the OVMI computation

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers