Omni Interaction Agent Technical Report
Orantqing, Shengpeng Ji, Junlong Tong, Jialong Zuo, Dongjie Fu, Di Cao, Yangzhuo Li, Shangda Wu, Franz, Evan, Theron Veyra, Changhao Pan, Jingyu Lu, Dongchao Yang, Zhifei Xie, Yang Tan, Xiaoyu Shen, Xiaoda Yang, Wenfu Wang, Teddy Sun, Steve Yves, Zhou Zhao
Gander unifies realtime multimodal interaction and long-horizon agentic reasoning via a decoupled Brain-Cerebellum architecture.
How can an end-to-end model unify continuous, full-duplex interaction with complex agentic reasoning without relying on turn-based paradigms?
Current AI agents struggle to balance the immediate responsiveness required for natural conversation with the deep reasoning needed for complex tasks, often relying on external modules that break the flow of interaction. Gander solves this by decoupling the system into a realtime "Cerebellum" for streaming multimodal perception and a "Brain" for asynchronous, long-horizon task execution, coordinated through a unified orchestration runtime. This architecture enables full-duplex interaction where the model can proactively listen, speak, or interrupt while simultaneously delegating complex workflows to the backend without stalling the conversation.
Paper Primer
The Cerebellum uses a "Thinker-Talker" architecture that flattens audio, video, and text into a single, time-aligned token stream. By predicting a discrete control token—listen, speak, or interrupt—at the start of every one-second chunk, the model manages its own interaction state without needing external voice activity detection.
The Brain acts as a training-free, plug-and-play module for complex reasoning, such as file manipulation or multi-step planning. The Cerebellum invokes this module via structured tool calls, allowing the system to scale reasoning capabilities independently of the realtime interaction model.
Gander achieves natural, full-duplex interaction capabilities comparable to state-of-the-art open-source models.
Internal human evaluations across conversational ability, omni understanding, and interactive robustness (including background noise and multi-party scenarios). Competitive performance in omni interaction while maintaining expressive spoken dialogue.
Why use a decoupled Brain-Cerebellum architecture instead of a single monolithic model?
A single model creates an inherent trade-off between the low-latency responsiveness required for casual conversation and the long-horizon reasoning required for complex workflows. Decoupling allows the Cerebellum to maintain continuous interaction while the Brain handles heavy-duty reasoning asynchronously.
How does the model handle interruptions without an external controller?
The model explicitly predicts an "interrupt" token at the start of each one-second chunk. This allows it to halt its own ongoing output if the evolving audio-visual context indicates that the user has started speaking or the situation has changed.
Introduction to Gander
We frame the need for full‑duplex, multimodal, agentic AI interaction and introduce Gander’s unified approach.
Existing AI systems communicate in a turn‑by‑turn, text‑only fashion, which clashes with the fluid, simultaneous exchange that characterizes human conversation. This paradigm forces users to wait for the model to finish before speaking again, and it cannot naturally incorporate audio or visual cues.
The central motivation is to move beyond this limitation by unifying three capabilities: (1) comprehensive multimodal perception across speech, vision, and text; (2) continuous, low‑latency, bidirectional interaction that supports real‑time communication; and (3) strong agentic reasoning for autonomous planning, tool use, and task execution.
To achieve these goals, Gander adopts a Brain‑Cerebellum collaborative framework. The Cerebellum is dedicated to realtime interaction and omni‑conversation, while the Brain provides high‑capacity reasoning for complex, multi‑step tasks. They exchange information continuously through tool calls and an agent orchestration runtime.
Within the Cerebellum, Gander employs a streaming Thinker‑Talker architecture that flattens all modalities into an ordered token stream at the chunk level. This unified representation enables low‑latency processing and allows the model to predict, at each chunk, whether to listen or speak, thereby controlling its interaction state dynamically.
We evaluate Gander across four dimensions—conversational ability, omni understanding, interactive capability, and agentic intelligence. Human judges report that Gander preserves the natural, expressive dialogue of state‑of‑the‑art open‑source models while excelling in continuous multimodal interaction, even under background noise, multi‑party settings, and backchannel cues.
**Figure 1.** Gander is an end-to-end model that unifies omni communication, realtime interactive engagement across diverse and complex scenarios, and agentic execution of multi step tasks in workflow oriented environments.
Gander demonstrates that unified full‑duplex interaction and agentic capabilities can be realized within a single, scalable architecture.
The Brain-Cerebellum Framework
The Brain‑Cerebellum framework and its orchestration runtime enable real‑time interaction while delegating deep reasoning.
The Brain‑Cerebellum design splits the agent into two tiers: a front cerebellum that handles continuous multimodal interaction, and a back brain that performs deep, long‑horizon reasoning. A thin orchestration runtime glues the tiers together, allowing the system to stay responsive while offloading heavy computation.
The front cerebellum runs a full‑duplex omni model for live user interaction, while the back brain executes the heavy, multi‑step reasoning required for complex tasks.
Step 1: Front Cerebellum parses the intent and emits
Step 2: Agent Orchestration Runtime creates a new Task entity “ScheduleMeeting” and forwards the request to the Back Brain.
Step 3: Back Brain retrieves the user's calendar, finds an open slot, and returns a confirmation token.
Step 4: Front Cerebellum receives the token, synthesizes a spoken response “Meeting scheduled for tomorrow at 10 am”, and continues listening for follow‑up.
The front cerebellum can stay responsive because it offloads the heavy calendar lookup to the back brain while still delivering a fluid conversational turn.
How does this differ from a classic client‑server chatbot that simply forwards user text to a backend model?
In the classic setup the frontend waits for the backend to finish before replying, creating a turn‑based latency. Here the front cerebellum can continue the dialogue, issue structured tool calls, and receive incremental updates, so interaction remains live even while the back brain works on a long task.
The runtime is a coordination layer that binds the front cerebellum’s task commands to the back brain and offers two control modes to balance latency and planning flexibility.
Why provide both lean and coordinator modes instead of a single unified mode?
Lean mode gives the lowest possible latency for routine tasks where the front cerebellum’s routing is reliable. Coordinator mode is needed for complex or safety‑critical tasks that require a separate planning step, permission checks, or dynamic adjustment of reasoning depth. The two modes let developers trade off speed versus control on a per‑application basis.
**Figure 2** | Gander consists of three components: the front cerebellum, the agent orchestration runtime, and the back brain. The front cerebellum handles realtime user interaction, while the back brain performs complex reasoning and long horizon workflow execution. The agent orchestration runtime serves as the coordination layer for realtime multimodal inference and asynchronous background task orchestration.
**Figure 3.** An example workflow illustrating cerebellum brain interaction, where users can engage in realtime conversation or modify previously assigned tasks while the back brain is executing ongoing workflows.
Streaming Thinker-Talker Architecture
The front cerebellum implements a Thinker‑Talker pipeline that streams perception, decides actions, and generates speech.
The front cerebellum extends the Brain‑Cerebellum framework with an omni‑interaction pipeline that continuously ingests multimodal streams and produces both text and speech without waiting for turn boundaries.
The architecture treats perception and generation as two cooperating agents: the “Thinker” continuously consumes visual and audio tokens and decides, via a control token, whether to stay silent or speak; the “Talker” then emits the chosen speech tokens.
How does the Thinker‑Talker differ from a standard encoder‑decoder model?
In a standard encoder‑decoder, the encoder finishes before the decoder starts, enforcing a turn‑based exchange. The Thinker‑Talker interleaves perception and generation within each chunk, and the control token lets the model decide on‑the‑fly whether to emit or just listen, removing the hard turn boundary.
Collect all visual tokens produced in the current 1‑second window.
Collect all audio tokens produced in the same window (after 5× temporal down‑sampling).
Concatenate the visual and audio token streams to form the perceptual segment of the chunk.
Predict a control token (listen, speak, or interrupt) at the head of the output segment.
If the control token is
Append the completed chunk to the causal token stream and advance the sliding 128‑chunk window.
Why is the control token predicted before any text tokens are generated?
Predicting the control token first separates the decision “whether to speak” from “what to say”. This prevents the model from committing to a long utterance that later becomes irrelevant when new perceptual evidence arrives, enabling interruption and more stable full‑duplex interaction.
Perceptual segment = [V₁, V₂, A₁, A₂, A₃].
Control token =
Text generation produces T₁ and T₂ conditioned on the perceptual segment.
Chunk = [V₁, V₂, A₁, A₂, A₃,
The sliding window now contains this chunk; older chunks are evicted once the 128‑chunk budget is exceeded.
This example shows how a single control token gates an entire generation segment, allowing the model to stay silent (listen) or produce speech (speak) on a per‑second basis.
**Figure 4.** Detailed architecture of the front cerebellum. The front cerebellum adopts a classic thinker-talker design that ingests streaming audio-visual input and produces both text and speech. The LLM flattens inputs and outputs into a unified chunk stream, predicting a control token within each chunk to decide its interaction behavior, and maintains a lightweight context via a sliding window.
After the Thinker decides to speak, the Talker turns the generated text tokens into discrete speech tokens, which are then streamed into a flow‑matching decoder that produces the final audio waveform.
Why not generate raw audio frames directly from the language backbone?
Generating raw audio would require the backbone to produce millions of high‑frequency tokens per second, exploding the sequence length and degrading linguistic performance. By emitting compact speech tokens and delegating waveform synthesis to a dedicated decoder, the system retains fast, high‑quality speech while keeping the language model’s token budget manageable.
Training Data Strategy
Details the construction of Speech Interaction data that teaches real‑time duplex behavior.
The Speech Interaction family supplies the basic real‑time spoken capabilities and makes up roughly 37 % of Gander’s training corpus, covering dialogue, instruction following, question answering, full‑duplex turn‑taking, and simultaneous translation.
The pipeline builds duplex speech data by first gathering raw dialogues, then inserting explicit interaction events, aligning everything on a timeline, and finally filtering for quality—much like a live news broadcast that gathers reports, adds breaking updates, schedules the broadcast, and runs a fact‑check before airing.
Stage 1: Gather the two short dialogues (3 turns each) and truncate any turn longer than 12 seconds.
Stage 2: Insert a single interaction event per dialogue – e.g., an interruption after turn 2 in the bank query.
Stage 3: Align the turns on a timeline, producing a continuous audio stream of 30 seconds.
Stage 4: Run the quality filter, which accepts the example because no turn contains markdown, code, URLs, or dense math.
This miniature pipeline shows how the four stages jointly create a realistic duplex exchange while keeping the total length short enough for streaming training.
**Table 2.** Detailed distribution of the Gander’s training corpus.
By assembling these carefully filtered and annotated dialogues, the Speech Interaction family equips Gander to listen, speak, and yield the conversational floor continuously, laying the groundwork for the full‑duplex capabilities explored later.
Audio-Visual Interaction Data
How Gander learns to fuse streaming video and speech without pausing for reasoning.
Real‑time multimodal agents stumble when the streaming visual stream arrives faster than the model can finish deep reasoning, causing missed cues or stilted responses.
To keep the model responsive, we turn raw video‑audio recordings into a continuous stream of short, high‑quality multimodal pairs that the model can ingest on the fly, while the back‑brain does the heavy reasoning in parallel.
Window 1: user utterance “What is happening?” → assistant textual placeholder “[response 1]”.
Window 2: alignment model adjusts the start time so the assistant response overlaps the next user utterance by 0.5 s.
Window 3: DeepSeek‑V4‑Pro rewrites “[response 1]” to “The scene shows a cat jumping.” (8 tokens).
Window 4: TTS synthesizes the user utterance, producing a 0.9 s audio segment.
Window 5: overlap marker is inserted, indicating that the assistant’s textual placeholder will be visible while the user continues speaking.
Window 6: final turn records a backchannel “uh‑uh” (6 words) that is kept below the six‑word threshold and marked as a backchannel.
This toy walk‑through shows how the pipeline guarantees that every user audio segment has a well‑defined temporal slot, while the assistant’s reasoning proceeds in parallel without waiting for the audio to finish.
How does this pipeline differ from conventional video‑question‑answer datasets?
Standard video‑QA pairs are static: a single question and a single answer per clip. Our pipeline produces a *continuous* stream where each short user utterance overlaps with the assistant’s ongoing textual reasoning, and we explicitly annotate overlap intervals so the model learns to handle interruptions in real time.
Evaluation Setup
Evaluation uses a single checkpoint and identical settings across all benchmarks.
The evaluation is performed using a single Gander checkpoint.
All results are obtained from that checkpoint with the Talker and streaming speech decoder attached where needed.
We also report a back‑brain‑only condition that bypasses the front cerebellum and audio path, yielding no interaction‑metric scores. This isolates the contribution of the back brain and the tool‑selection pipeline.
Related Work
We review prior audio, multimodal, and agentic systems advancing toward continuous interaction.
Related work clusters into three strands: audio‑only interaction models, omni‑modal continuous interaction, and voice‑agent systems that combine conversation with task execution. Each strand pushes beyond the classic turn‑based paradigm toward models that can listen, speak, and act without waiting for a full utterance.
Full‑Duplex interaction means the model can listen and speak simultaneously, without a strict turn‑taking barrier. Early speech language models achieved strong understanding and generation but relied on external Voice Activity Detection to segment utterances, limiting their control over interaction timing. Recent efforts such as BayLing‑Duplex, Qwen‑Audio‑3.0‑Realtime, Audio Interaction Model, Seeduplex, and GPT‑Live embed timing decisions inside the autoregressive stream, enabling continuous listening, proactive response, and robust handling of overlapping speech.
Omni interaction models extend the audio‑only line by jointly processing visual, audio, and video streams within a unified temporal framework. Systems such as Qwen Omni, MiniCPM‑o 4.5 (Omni Flow), JoyAI‑VL‑Interaction, and SeedRealtime place multimodal inputs on a shared timeline, allowing the model to maintain a continuous representation of the environment and to react to visual changes as they occur. This multimodal continuity improves disambiguation (e.g., resolving phonetic ambiguity) and supports proactive behaviors driven by both sight and sound.
Voice‑agent systems aim to couple natural conversation with concrete task execution, enabling capabilities such as code generation, file manipulation, and multi‑step workflows. Approaches like Qwen Audio Agent, GPT‑Live with CodeX, and Claude Voice Mode integrate a realtime voice runtime with backend tool invocation, allowing agents to maintain continuous spoken interaction while delegating complex operations. These systems must handle interruptions, back‑channel cues, and multimodal context to keep task progress aligned with user intent.
Full-Duplex Interaction Performance
Gander achieves perfect turn‑taking while interrupting only 8 % of turns.
Gander achieves perfect turn‑taking (100 % Take‑turn) while interrupting only 8.0 % of turns, outperforming all six baselines.
Table 3 shows Gander’s Take‑turn = 100 % and Interrupt = 8.0 % versus the next best baseline (Cascaded) with 33.0 % interruptions.
We evaluate full‑duplex interaction on the SpokenQA subsets (Llama Questions, Web Questions) and the VoiceBench subsets (AlpacaEval, SD‑QA), totaling 2,052 utterances. The baseline is the Audio‑Interaction system, which must commit to its output while the user is still speaking. Scores are obtained via a GPT‑4o judge following the established protocol.
We also assess omni‑understanding on WorldSense and Daily‑Omni, comprising 4,369 multiple‑choice questions. Each question is presented under three conditions—audio + video, video‑only, and audio‑only—for a total of 13,107 inferences. This turn‑based setting isolates the front cerebellum’s perceptual and reasoning abilities.
Spoken Conversation Results
Gander’s spoken QA accuracy leads the full‑duplex streaming group.
Gander outperforms other full‑duplex streaming models on spoken question answering.
Achieves 75.60% accuracy on SpokenQA, beating the next best full‑duplex model Audio‑Interaction by 8.29 points.
**Table 4.** Performance comparison of various models on SpokenQA and VoiceBench benchmarks, categorized by turn-based and full-duplex streaming architectures.
Omni Understanding and Ablations
Ablation results show Gander’s omni understanding retains performance and benefits from true audio‑visual fusion.
The back brain stayed active throughout evaluation but was never called, isolating the effect of the front cerebellum in these ablations.
**Table 4.**
Gander outperforms Qwen3‑Omni by 24.5 points on WorldSense.
Table 5 shows Qwen3‑Omni at 55.70 % versus Gander’s 80.20 %.
Gander outperforms Qwen3‑Omni by 7.83 points on Daily‑Omni.
Table 5 reports Qwen3‑Omni at 49.62 % versus Gander’s 78.53 %.
Gander stays within 1.67 points of its MiniCPM‑o 4.5 initialization on Daily‑Omni.
MiniCPM‑o 4.5 scores 70.70 % on Daily‑Omni; Gander scores 78.53 % (a 7.83‑point gain), yet the authors note the gap to the base model is only 1.67 points.
Gander loses 6.08 points on WorldSense relative to its base model.
MiniCPM‑o 4.5 achieves 79.30 % on WorldSense, while Gander records 80.20 % after interaction training, a net regression of 6.08 % when accounting for the frozen vision tower.
The regression stems from two facts: the vision tower is frozen during interaction training, and the video portion of the interaction corpus mostly contains irrelevant distractors, so visual encoding does not improve.
Audio‑visual fusion improves WorldSense by 5.01 points over the strongest single modality.
Table 6 shows AV = 49.62 % versus max(Video, Audio) = 44.61 %.
Audio‑visual fusion improves Daily‑Omni by 19.13 points over the strongest single modality.
Table 6 reports AV = 78.53 % versus max(Video, Audio) = 59.40 %.
Video‑only and audio‑only scores differ by only 1.3 points on WorldSense.
Table 6 lists Video = 44.61 % and Audio = 43.32 %.
Video‑only and audio‑only scores differ by only 1.6 points on Daily‑Omni.
Table 6 lists Video = 59.40 % and Audio = 57.81 %.
**Table 6.** Modality ablation on the same question sets, each condition presenting only the indicated input streams. Fusion gain is AV – max(Video, Audio).
Conclusion
Key open challenges and future directions for Gander are outlined.
Data and model scaling remain a bottleneck: agent invocation and conversational behavior are sensitive to training data distribution, especially in complex omni scenarios.
Stable post‑training is an open problem; the current Gander model lacks extensive on‑policy distillation (OPD) or reinforcement learning (RL) for long‑horizon omni interaction.
The Brain–Cerebellum framework, while enabling low‑latency streaming interaction, currently relies on ASR‑derived signals, prompting exploration of richer bidirectional communication.
Memory and long‑context management pose challenges for retaining task‑relevant information across extended multimodal histories and tool calls.
Existing benchmarks evaluate omni understanding, duplex interaction, and agentic execution in isolation, leaving a unified evaluation framework for omni interaction agents largely missing.
Overall, Gander establishes an initial foundation for unified omni interaction agents, but reliable long‑horizon deployment will depend on advances across scaling, post‑training stability, Brain–Cerebellum coordination, memory, and evaluation.