GigaAM Multilingual: Foundation Model for Underrepresented Languages

Andrei Kuzmenko, Alexandr Maximenko, Aleksandr Kutsakov, Georgii Gospodinov, Dmitrii Bolotov, Oleg Kutuzov, Pavel Bogomolov, Fyodor Minkin

GigaAM Multilingual improves ASR for underrepresented Central Asian languages via cluster-aware pre-training and domain-stratified fine-tuning.

How can we build robust multilingual ASR foundation models for underrepresented languages despite severe data scarcity?

Multilingual speech models often fail on low-resource languages because massive, imbalanced training datasets allow high-resource "head" languages to dominate the model's internal representations. The authors introduce GigaAM Multilingual, which uses cluster-level data balancing during self-supervised pre-training and domain-aware sampling during fine-tuning to force the model to learn from underrepresented languages. In controlled comparisons, this approach significantly outperforms larger open-source encoders like Whisper Large v3 and Omnilingual-1B on Kazakh, Kyrgyz, and Uzbek speech tasks.

Paper Primer

The core mechanism hinges on two explicit balancing strategies. During pre-training, the authors group languages into clusters based on co-occurrence patterns and reweight these clusters to prevent head-language dominance; during fine-tuning, they apply domain-stratified sampling to regulate the influence of synthetic and weakly-supervised data.

GigaAM Multilingual achieves superior performance on low-resource languages compared to significantly larger, state-of-the-art foundation models.

In a controlled CTC fine-tuning setup, the 600M-parameter GigaAM model consistently yields lower Word Error Rates (WER) on Kazakh, Kyrgyz, and Uzbek than Whisper Large v3 and Omnilingual-1B. On internal spontaneous speech datasets, GigaAM achieves WERs of 16.2% (Kazakh) and 11.9% (Kyrgyz), compared to 4.6% and 3.9% for Whisper, but significantly outperforms Whisper on standard benchmarks like FLEURS where Whisper struggles with language-specific tokenization.

The model demonstrates high parameter efficiency and rapid adaptation to languages with minimal pre-training coverage.

A compact 240M-parameter version of GigaAM outperformed both 1B-parameter baselines on average across all target languages, and achieved lower WER on Bashkir and Georgian than the larger models. On Bashkir, GigaAM achieved a 3.6% WER, less than half the 7.8% WER of the Whisper-based baseline.

Why is cluster-level balancing necessary instead of simply upsampling low-resource data?

Naive upsampling of low-resource data often leads to overfitting or degradation of performance on high-resource languages. By clustering languages based on co-occurrence, the authors stabilize the pre-training objective and ensure the model learns robust representations across the entire language distribution.

What is the primary bottleneck for these specific Central Asian languages?

The primary bottleneck is data scarcity and heterogeneity; transcribed speech is often noisy or limited, and high-resource models frequently fail to generalize to the spontaneous, conversational speech patterns found in these regions.

For researchers working in low-resource ASR, this paper demonstrates that data curation and weighting policies are as critical as model scale. The GigaAM recipe provides a proven, efficient path to multilingual adaptation that avoids the "head-language dominance" trap common in larger, monolithic foundation models.

Motivation and Problem Framing

We expose why multilingual ASR falters on low‑resource languages and outline our balancing solution.

Multilingual ASR systems achieve strong results on high‑resource languages but still exhibit large error gaps on low‑resource, long‑tail languages because training data is heavily skewed toward head languages.

**Figure 1.** Language duration distribution by cluster. Bars show hours per language; shaded “uncertain” segments denote low-confidence predictions assigned to the corresponding language. Languages in bold participate in the experiments.

The disparity in ASR performance for low‑resource languages remains a critical challenge.

Headline Performance Results

GigaAM Multilingual’s WER beats prior multilingual ASR systems on target languages.

The authors evaluate GigaAM Multilingual against three public multilingual ASR baselines on three test suites. Across most non‑English languages the new model consistently records the lowest Word Error Rate (WER).

Omnilingual 1B is a 1‑billion‑parameter language model fine‑tuned for ASR, aiming to leverage massive multilingual pre‑training to handle many languages with a single model.

Seamless M4T large v2 is a large‑scale multilingual speech‑to‑text system that combines a massive encoder‑decoder architecture with multilingual training data, targeting high‑quality transcription across many languages.

**Table 1.** WER (%) of GigaAM Multilingual against public multilingual ASR systems on Common Voice (CV), FLEURS, and our internal in-the-wild test sets (Sec. 4.2), best in bold.

GigaAM Multilingual attains the lowest WER on four of the five target languages across every test split.

Table 1 reports GigaAM’s WER as lower than Omnilingual 1B, Seamless M4T, and Whisper for Russian, Kazakh, Kyrgyz, and Uzbek.

GigaAM outperforms existing multilingual models on the target languages.

Context and Prior Work

Key prior work on multilingual ASR and data‑centric techniques that motivate GigaAM.

Large‑scale ASR models such as Whisper, USM, MMS, and Omnilingual have pushed multilingual robustness forward by scaling data and model capacity, yet benchmark analyses still show uneven gains across languages, especially for long‑tail languages.

Self‑supervised learning (SSL) enables training speech encoders from raw audio, but merely increasing data volume is insufficient; language‑ and dataset‑imbalance require careful sampling, which motivated the group‑aware reweighting and cluster‑level sampling weights used in GigaAM.

Prior multilingual SSL work, exemplified by mHuBERT‑147, demonstrates that scaling alone does not guarantee coverage; instead, multilingual up‑sampling and balanced batching are needed to prevent dominant languages from overwhelming the training signal.

For low‑resource languages, data scarcity is the primary bottleneck; pipelines such as GigaSpeech 2 combine Whisper transcription, MMS‑based forced alignment, and iterative refinement to generate pseudo‑labels, while OWSM v3.2 and OWSM v4 study how heterogeneous sources and scalable cleaning affect model performance.

Choosing a suitable multilingual encoder for adaptation is non‑trivial because pretrained models hide their data mixtures; our controlled comparison of two large‑scale encoders (≈3–5 M hours) under identical fine‑tuning recipes highlights the impact of source‑aware weighting on stable low‑resource gains.

We summarize our contributions: (1) GigaAM, a state‑of‑the‑art open‑weight multilingual ASR model for Kyrgyz, Kazakh, and Uzbek; (2) group‑aware reweighting during SSL pre‑training and its interaction with downstream fine‑tuning; (3) cross‑lingual transfer results on tail languages.

Pre-training Methodology

Pre-training GigaAM with masked unit prediction and cluster‑level language weighting.

The pre‑training stage must learn robust acoustic representations while handling a highly imbalanced multilingual corpus. The authors therefore combine a HuBERT‑style prediction task with a cluster‑based language balancing scheme.

Instead of predicting raw audio samples, the model learns to fill in missing acoustic “units” – discrete symbols derived from a teacher network – much like a crossword where only some squares are hidden.

Student encoder processes the visible frames (1,3,5) and produces hidden vectors $h_1, h_3, h_5$.

For masked frame 2, the model predicts a distribution over {A,B,C,D}; the correct teacher label is B.

Similarly, for masked frame 4 the correct label is D.

Cross‑entropy loss is computed for frames 2 and 4 against the teacher’s labels (B and D).

Back‑propagation updates $\theta$ so that future predictions for similar masked patterns improve.

Masking a small subset of frames forces the encoder to infer missing acoustic context, which directly benefits downstream speech recognition.

How does this masked unit prediction differ from the masked language modeling used in BERT?

BERT masks word tokens and predicts vocabulary items, whereas here we mask acoustic frames and predict discrete acoustic units derived from a teacher’s continuous representations. The target space is learned from audio, not a fixed text vocabulary, and the encoder operates on spectrograms rather than token embeddings.

Instead of trying to balance every language individually—a task that fails for low‑resource tongues—the method groups languages into clusters based on how often they co‑occur, then balances at the cluster level.

Compute normalized weights, e.g. $w_{12}=30/(100\cdot80)=0.00375$, $w_{13}=5/(100\cdot50)=0.001$, …

Apply a threshold of $0.002$; edges $w_{13}, w_{14}, w_{24}, w_{34}$ are pruned.

Remaining graph connects L1–L2 and L2–L3, forming a single cluster {L1,L2,L3} and an isolated low‑resource language L4.

Cluster‑level weighting assigns a higher sampling weight to the large cluster and a modest weight to L4, avoiding over‑representation of the scarce language.

By collapsing related languages into a cluster, the method preserves useful multilingual signal while preventing the rare language from dominating its own weight estimate.

Why not balance each language directly instead of clustering?

Direct per‑language weighting requires reliable frequency estimates for every language; low‑resource languages have noisy counts, leading to unstable weights and over‑fitting. Clustering aggregates similar languages, smoothing the estimates and reducing the number of parameters to tune.

Segment raw audio into one‑minute chunks and run voice‑activity detection to keep speech‑only fragments.

Apply MMS LID 4017 to each fragment; aggregate fragment predictions by majority vote to assign a language to the whole recording.

Build a language co‑occurrence graph from recordings, normalize edge weights, prune low‑weight edges, and cluster languages.

Sample recordings according to cluster‑level weights while respecting the two confidence thresholds.

Feed masked mel‑spectrograms into the student Conformer encoder; compute the masked unit prediction loss against teacher‑derived discrete units.

Back‑propagate gradients to update the student encoder parameters.

Fine-tuning and Data Strategy

Fine‑tune the encoder with CTC loss on a multi‑domain, multilingual dataset.

Fine‑tuning adapts the pretrained encoder to the target ASR task, but multilingual fine‑tuning suffers from scarce labeled audio for Central Asian languages.

CTC aligns predicted character sequences with the reference without needing frame‑level alignment, letting the model learn from raw audio‑text pairs across languages.

How does CTC fine‑tuning differ from standard cross‑entropy training on aligned transcripts?

Cross‑entropy requires a one‑to‑one mapping between each frame and a target label, so it needs precise frame‑level alignments. CTC instead sums over all possible alignments that reduce to the same transcript, using a blank token to absorb extra frames, thus working with only the overall transcript.

Logits → frame1: a=0.7, b=0.1, blank=0.2; frame2: a=0.4, b=0.5, blank=0.1; frame3: a=0.2, b=0.6, blank=0.2.

CTC enumerates all alignments that collapse to “ab”, e.g., (a,blank,b), (a,b,blank), (blank,a,b).

Compute probability of each alignment by multiplying the per‑frame softmax probabilities, then sum them.

Negative log of this sum is the CTC loss; gradients flow back to adjust logits.

CTC lets the model learn correct character ordering without explicit frame‑to‑character timing, which is crucial when fine‑tuning on heterogeneous, loosely aligned data.

**Table 2.** Fine-tuning data (hours) by language and source.

Sampling Strategy Experiments

Domain-aware sampling cuts WER on low‑resource languages.

Domain‑aware sampling reduces average WER on low‑resource languages by up to 1.2 absolute points versus unbalanced sampling.

Table 4 shows the lowest WER for Kyrgyz (8.5 vs 9.4), Kazakh (8.7 vs 9.1), and Uzbek (9.7 vs 9.4) under domain‑aware sampling.

Each training epoch rebuilds the data pool so that languages or domains appear with a target probability, ensuring scarce clusters get proportionally more examples.

How does this differ from simple language‑balanced sampling?

Language‑balanced sampling equalizes language counts but ignores domain diversity within a language; cluster‑level sampling also balances sub‑domains (e.g., broadcast vs conversational), giving extra exposure to rare domains.

Domain‑aware sampling markedly improves ASR performance on low‑resource languages.

Training Ablations

Fine‑tuning sampling strategies are evaluated to see how they affect low‑resource ASR performance.

Applying cluster‑level sampling weights during pre‑training reduces WER on low‑resource languages.

WER on Kyrgyz drops from 9.4 % to 8.5 % and on Uzbek from 10.5 % to 9.7 % when moving from E0 to E2. -0.9 % (Kyrgyz) / -0.8 % (Uzbek) absolute reduction

These weights have negligible impact on high‑resource languages.

Russian WER changes minimally (4.6 % → 4.7 %) and English WER increases only slightly (14.4 % → 15.4 %). +0.1 % (Russian) / +1.0 % (English) absolute change

During fine‑tuning we compare three sampling strategies: the original unbalanced distribution, a language‑balanced scheme, and a domain‑aware approach that weights sub‑domains within each language.

Domain‑aware sampling yields the largest WER improvements for Kazakh and Kyrgyz, especially on spontaneous speech, while English regresses only marginally.

Uzbek shows no clear benefit from domain‑aware sampling over language‑balanced sampling, reflecting its smaller, less synthetic dataset.

Because the primary goal is to boost low‑resource performance, the authors adopt the E2 pre‑trained encoder and domain‑aware fine‑tuning for the final released model.

Encoder Architecture Comparison

Ablation isolates encoder impact on CTC fine‑tuning across languages.

We evaluate how much of the performance gain comes from the pretrained encoder versus the fine‑tuning recipe by swapping components in a controlled CTC setting.

Whisper Large is a high‑capacity speech model pretrained on massive multilingual audio, later fine‑tuned with a CTC loss for transcription.

How does Whisper Large differ from the GigaAM encoders despite using the same CTC loss?

Whisper Large’s encoder is trained on a broader, less curated dataset and retains a generic acoustic front‑end, whereas GigaAM encoders are pretrained with cluster‑level sampling that emphasizes under‑represented languages, giving them a more language‑balanced representation.

GigaAM 600M achieves the lowest average WER (10.2 %) across all five languages, beating Whisper Large v3 (14.1 %) and Omni SSL 1B (16.6 %).

Table 5 shows 10.2 % avg WER for GigaAM 600M versus 14.1 % for Whisper and 16.6 % for Omni SSL.

GigaAM 240M attains an average WER of 12.2 %, outperforming Whisper Large v3 (14.1 %) and Omni SSL 1B (16.6 %) on all languages except English.

Table 5 reports 12.2 % avg WER for GigaAM 240M, with lower WER than Whisper on Russian, Kazakh, Kyrgyz, and Uzbek.

GigaAM encoder provides better initialization for low‑resource CTC fine‑tuning than Whisper.

Tail Language Adaptation and Summary

Long‑tail languages suffer from data scarcity; GigaAM uses cluster‑aware sampling to improve robustness.

We evaluate how well the models adapt to truly low‑resource languages. Bashkir and Georgian were chosen because each received less than 500 h of pre‑training audio.

GigaAM achieves the lowest WER on both tail languages.

Table 6 shows GigaAM 3.8 % WER on Georgian and 3.6 % on Bashkir, beating Whisper Large v3 (13.4 % / 7.8 %) and Omnilingual 1 B (11.1 % / 8.2 %).

**Table 6.** Adaptation to minimal-coverage tail languages: WER (%) on Common Voice (separate per-language CTC model)

Beyond tail‑language adaptation, the study shows that explicit data balancing during pre‑training and domain‑aware sampling during fine‑tuning prevent head‑language dominance, enabling a 240 M‑parameter GigaAM model to outperform much larger baselines.

Even the compact 240 M GigaAM model surpasses Whisper Large v3 and Omnilingual 1 B under matched CTC fine‑tuning.

Experiments on Kazakh, Kyrgyz, and Uzbek—especially on spontaneous speech—show lower WER than the larger encoders.

The model generalizes to languages not seen during pre‑training.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers