Omni-IO Skills: Harnessing Your Agent Omni-Native
Yanlin Li, Mingyang Hao, Shengqiong Wu, Hao Fei, Mong-Li Lee, Wynne Hsu
Omni-IO Skills adds a plug-and-play harness to existing agents, enabling multi-modal workflows without retraining.
How can we unify fragmented multimodal agent capabilities into a single, persistent, and dependency-aware execution framework?
General-purpose agents excel at reasoning but struggle to coordinate complex, multi-modal production tasks like creating a coordinated video, slide deck, and landing page from raw source material. Omni-IO Skills acts as a middleware layer that organizes these capabilities into hierarchical, loadable "Skills" and manages artifact dependencies through a persistent registry, leaving the host agent's core reasoning model untouched. On the UniM-90 benchmark, this harness boosts input-support rates to 100% and significantly improves semantic quality and structural coherence across both GPT-5.6 Sol and Claude Sonnet 5.
Paper Primer
The system uses a four-layer architecture to decouple task knowledge from specific model backends. It represents workflows as Declare Execution Graphs (DEGs), which schedule tasks based on data dependencies and automatically register intermediate outputs for cross-turn reuse.
Omni-IO Skills enables full modality coverage for base agents that previously lacked it.
Input-support rates for GPT-5.6 Sol and Claude Sonnet 5 rose from 40.00% and 38.89% to 100%. A 60+ percentage point increase in task-handling capability.
The harness significantly improves the semantic and structural quality of multi-modal outputs.
Relative Semantic–Quality Coupled Score (SQCS) increased by ~48–50 points for both tested agents. Nearly doubling the relative performance score on the UniM-90 test set.
Why use a harness instead of just training a more capable multi-modal foundation model?
Training a single model to handle all modalities requires costly updates and reconciliation of disparate objectives; the harness approach allows developers to swap in specialized, evolving media engines as replaceable backends without retraining the host agent.
What is the role of the Asset Registry in this architecture?
The registry provides a persistent, unified data abstraction that decouples artifacts from specific file paths, allowing downstream tasks and future interaction turns to reference and reuse intermediate outputs reliably.
Developers can now extend general-purpose agents into omni-native systems by composing modular, dependency-aware skills rather than relying on monolithic model updates.
Introduction to Omni-IO Skills
We expose the fragmented multimodal gap and propose Omni‑IO Skills to orchestrate persistent asset workflows.
Current agents excel at planning and reasoning but their multimodal production pipelines are split across isolated tools, causing broken dependencies and lost intermediate assets. This fragmentation prevents reliable, end‑to‑end workflows that span text, images, audio, video, documents, 3D assets, and code.
General‑purpose agents can plan and reason, yet they lack a built‑in, persistent graph that tracks multimodal assets and their inter‑task dependencies across multiple turns.
**Figure 1.** Overview of Omni-IO Skills. Hierarchical Skills orchestrate multimodal understanding and generation, while generated assets are registered for downstream and cross-turn reuse.
**Figure 2.** Real-world task coverage supported by Omni-IO Skills, organized into cross-modal understanding, generation, reasoning, and retrieval. Parenthesized icons indicate representative input-to-output modality flows.
**Figure 11.** Cross-turn reuse represented as a JSON object. The second turn regenerates nothing: the referenced asset enters the new graph as a completed node, and only the video node issues a call.
The shift from fragmented multimodal tools to a persistent, orchestrated agent harness unlocks reliable, reusable workflows without altering the host’s reasoning core.
Related Work
Survey of prior Omni models and agent harnesses, framing Omni-IO Skills as a complementary orchestration layer.
Early multimodal research split into perception‑oriented models that map media to semantic embeddings and generative models that reconstruct visual or acoustic detail. Omni foundation models later merged these strands, offering a shared context for perception, reasoning, and generation.
Two dominant design routes emerged: unified autoregressive models that discretize all modalities into a single token stream, and hybrid discrete‑continuous models that keep autoregressive semantics while delegating fidelity to continuous decoders such as diffusion or flow. The former simplifies interleaving but suffers from tokenization overhead and limited perceptual detail, whereas the latter preserves specialization at the cost of coordinating multiple encoders and adapters. Recent systems add streaming and modality‑specific experts to reduce latency, and Omni‑IO Skills builds on this landscape by treating any model or media engine as a replaceable backend, unifying at the level of task execution and artifact flow.
Agent Harnesses supply the operational substrate for sustained execution, handling the action loop, tool access, context and state management, execution control, verification, and recovery. Mechanisms such as ReAct’s interleaved reasoning‑action‑observation loop and the Model Context Protocol (MCP) standardize tool exposure, but reliable long‑horizon execution also demands persistent state, dependency tracking, and robust failure handling. Omni‑IO Skills extends this harness with a dependency‑aware execution graph and persistent artifact state, while Agent Skills record procedural knowledge—inputs, tool invocations, and expected outputs—so that curated skills can be reused across tasks.
System Capabilities
Omni-IO Skills enables multimodal, multi‑asset tasks through hierarchical skill orchestration.
Existing agents stumble when a workflow must keep track of intermediate artifacts across turns. Without a persistent, dependency‑aware registry, assets are lost or duplicated, causing brittle pipelines.
Omni-IO Skills is a hierarchical layer that registers every intermediate artifact, records its dependencies, and exposes a uniform task interface so downstream skills can reuse assets without re‑computing them.
Understanding Skill parses the text and extracts key product features; it also tags the photo with visual attributes.
Generation Skill uses the extracted features to draft a layout image and a caption.
Retrieval Skill fetches the company brand‑guideline document (Asset 3) from the external knowledge base.
Reasoning Skill assembles the layout, caption, and brand guidelines into a final PDF brochure.
The completed brochure is stored in the Asset Registry, making it instantly available for any subsequent request.
This flow shows how assets are registered once and then reused across distinct Skills without recomputation.
Modality Coverage. Omni-IO Skills natively handles seven artifact types: text, image, video, audio, document, 3 D, and code.
Cross‑Modal Understanding. Understanding Skills ingest heterogeneous sources, turn them into structured evidence, and expose that evidence to the host agent for downstream use.
Cross‑Modal Generation. Generation Skills perform direct transformations (e.g., video → thumbnail), create single assets (e.g., code → executable script), or coordinate multi‑asset productions (e.g., images + 3 D → game asset pack).
Cross‑Modal Reasoning. Reasoning Skills combine extracted evidence with user constraints to plan, debug, or revise artifacts—such as generating a learning path or optimizing an experimental design.
Cross‑Modal Retrieval. Retrieval Skills link the current task context to external documents or web resources, enabling question answering, literature discovery, and news lookup.
Task Composition. Leaves are organized by application domain; higher‑level Skills can select multiple leaves, share intermediate assets, and assemble the final deliverables.
Layered Architecture
The section details the three‑tier skill hierarchy that turns high‑level requests into a dependency‑aware execution graph.
The core difficulty is coordinating many multimodal operations across turns without hard‑coding provider details. By separating knowledge, tool interfaces, implementations, and assets into distinct layers, the system can reuse components and remain agnostic to any specific service.
Organize capabilities as a three‑tier tree so the host agent can pick the right granularity and automatically reuse assets.
How does this hierarchy differ from a traditional monolithic workflow engine?
Traditional engines hard‑code a fixed sequence of API calls, tying each step to a concrete provider. The hierarchical architecture instead declares abstract Skills and lets the system expand them into a dependency‑aware graph, swapping providers or reusing assets without changing the high‑level description.
Scenario Skill S₁ selects Expert Skill E₁ (poster production) and Atomic Skill A₁ (caption generation).
E₁ expands into Atomic Skills A₂ (image generation) and A₃ (layout assembly).
Atomic Skills A₁, A₂, A₃ execute, each registering its output in the Asset Registry.
The DEG nodes for A₂ and A₃ are linked so the layout waits for the image before assembling the final poster.
All assets receive unique IDs; the final poster and caption are returned to the user.
The hierarchy ensures that shared intermediate assets (the generated image) are produced once and reused by both the poster layout and any other downstream deliverable.
Identify the Scenario Skill matching the user’s context.
Recursively replace the Scenario Skill with its constituent Expert and Atomic Skills.
For each Expert Skill, expand its workflow into the required Atomic Skills.
Resolve dependencies between Atomic Skills and register their outputs as assets.
Emit a Declare Execution Graph (DEG) where each node corresponds to an executable task.
Submit DEG nodes to the MCP Tool Service layer for concrete execution.
The MCP Tool Service layer translates the abstract task specifications into standardized executable calls, grouping capabilities into Understanding, Generation, and Utility functions.
The Provider and Configuration layer binds each semantic capability to a concrete model, credential set, and fallback policy, enabling seamless substitution without altering the Skill definitions.
Provide a stable, path‑independent reference for every intermediate and final artifact so later tasks can retrieve and reuse them reliably.
Why not store assets directly by file path instead of using a registry?
File paths are tied to a specific workspace and change when the underlying storage moves. The registry abstracts away those details, giving a stable identifier that survives renames, migrations, or provider swaps, and it also records provenance for reproducibility.
The audio generation Atomic Skill registers the clip as asset a₁ with
The registry stores
When the video montage skill runs, it queries the registry for the latest “speech” asset and receives
The montage skill reuses the existing audio file without re‑generating it, saving compute.
Asset reuse is achieved simply by referencing the stable `asset_id`, eliminating redundant calls to external tools.
**Figure 3.** Architecture overview of Omni-IO Skills, which presents the four-layer architecture.
**Figure 4.** Hierarchical organization of Omni-IO Skills. Higher-level Skills organize and invoke lower-level Skills. Scenario Skills coordinate application-level objectives, Expert Skills encapsulate reusable production workflows, and Atomic Skills provide executable capability primitives.
Consequently, a single high‑level request is automatically resolved into a dependency‑aware execution graph that can be executed, traced, and reused across turns.
Orchestration and Asset Lifecycle
Orchestrates multimodal tasks via a dependency‑aware execution graph and persistent asset registry.
Agents that need to generate images, audio, and code across multiple turns often stumble when tasks depend on each other’s outputs, leading to broken pipelines or duplicated work. The orchestration layer solves this by making dependencies explicit and managing assets throughout the workflow.
A DEG is a directed acyclic graph that spells out which generation tasks must wait for which others, turning a tangled set of multimodal requests into an ordered plan.
Create node $v_1$: $\langle\text{id}=1,\text{type}= \text{img},\dots\rangle$.
Create node $v_2$: $\langle\text{id}=2,\text{type}= \text{vid},\text{depends\_on}= \{1\}\rangle$.
Create node $v_3$: $\langle\text{id}=3,\text{type}= \text{code},\text{depends\_on}= \{1,2\}\rangle$.
Topologically sort: Wave 0 = $\{v_1\}$, Wave 1 = $\{v_2\}$, Wave 2 = $\{v_3\}$.
Execute $v_1$ → register image asset; $v_2$ reads that asset; $v_3$ reads both assets.
This tiny graph shows how a single intermediate result (the image) is reused without re‑generation, and how the DAG guarantees a safe execution order.
How does this DEG differ from a generic build‑system dependency graph?
Beyond pure control ordering, a DEG also records where each node’s output lives (via the Asset Registry) and is constructed on‑the‑fly from a user request, whereas a build graph is static and usually assumes files are already placed correctly.
The Asset Registry is a thread‑safe key‑value store that records every generated file’s path and metadata so later tasks can retrieve it by a stable identifier.
Task A writes “img1.png” and registers
Task B writes “sound1.wav” and registers
Task C queries the registry for “img1” and “audio1”, receives the two paths, and invokes the video tool with those inputs.
Task C’s output “promo.mp4” is then registered as “video1”.
By looking up assets instead of hard‑coding paths, tasks remain decoupled and can be retried or reordered without breaking the pipeline.
Why not pass file paths directly between tasks instead of using the registry?
Direct paths would require each producer to know the consumer’s filesystem layout and timing, creating brittle pipelines; the registry provides a uniform, atomic lookup that works even when tasks run on different machines or are retried later.
Receive user request and invoke Omni‑IO Skills.
Expand the request into hierarchical skills and finally into atomic tasks.
Instantiate each atomic task as a node and assemble the Declare Execution Graph (DEG).
Validate the DEG for resolvable references and acyclicity.
Schedule nodes into successive Waves based on dependency readiness.
For each Wave, dispatch all independent nodes concurrently.
Route each node to the appropriate tool via the MCP Tool Service, inject parameters, and execute.
Register the produced assets in the Asset Registry.
Expose registered assets to downstream nodes and to future turns.
Tool routing and parameter injection for a node
Before any execution the system checks that every edge points to an existing node or registered asset and that the graph contains no cycles; invalid graphs are rejected and rebuilt. Execution proceeds wave by wave, where each wave contains all nodes whose predecessors have finished, allowing parallelism across modalities, and any node that fails aborts only its dependent subtree.
Experiments and Results
Omni‑IO Skills boosts multimodal performance across two leading agents.
General‑purpose agents lack a persistent, dependency‑aware asset manager. Omni‑IO Skills adds a hierarchical orchestration layer that registers and tracks multimodal assets, enabling reliable multi‑asset execution.
Omni‑IO Skills raises the relative Semantic–Quality Coupled Score by up to 48 points for each base agent.
GPT‑5.6 Sol improves from 26.99 to 74.94 ($\Delta$ + 47.95) and Claude Sonnet 5 from 27.82 to 77.78 ($\Delta$ + 49.96).
UniM‑90 is a curated set of 90 multimodal instances drawn from the larger UniM suite, spanning text, image, audio, video, document, code, and 3D modalities in interleaved combinations.
GPT‑5.6 Sol is a large‑scale language model released by OpenAI, used here as a baseline general‑purpose agent.
Claude Sonnet 5 is Anthropic’s latest conversational model, serving as the second baseline agent.
**Table 3.** Main results on UniM-90. We report the input-support rate $\tau$ together with the absolute and relative variants of Semantic–Quality Coupled Score (SQCS), Interleaved Coherence Score (ICS), Strict Structure Score (StS), and Lenient Structure Score (LeS). Higher is better for all metrics.
The provided image contains a table comparing the performance of two base agents (GPT-5.6 Sol and Claude Sonnet 5) with and without "Omni-IO Skills" across various metrics (SQCS, ICS, StS, LeS) in both Absolute and Relative terms.
**Figure 6.** Qualitative case studies of Omni-IO Skills with GPT-5.6 Sol and Claude Sonnet 5.
Skill Catalog and Tooling
Appendix enumerates the full Omni-IO skill catalog and supporting specifications.
This appendix provides the exhaustive list of Omni-IO Skills and the detailed specifications that support their orchestration.
Fine‑grained capabilities that handle single‑modality perception or generation tasks.
Higher‑level capabilities that combine multiple atomic skills to produce polished, multi‑modal assets.
Top‑level orchestrations that coordinate several expert and atomic skills to achieve multi‑step objectives.
The table lists various skills categorized into Atomic Skills, Expert Skills, and Scenario Skills, each with a unique ID, skill name, and description.
Scenario Skill Coverage
This appendix details task coverage, tool inventory, provider bindings, defaults, and skill specification schema for Omni‑IO Skills.
This appendix enumerates the concrete task taxonomy, the MCP tool inventory, provider bindings, default generation settings, and the skill specification schema that underpin Omni‑IO Skills.
Collectively, these tables define the concrete taxonomy, tool bindings, and configuration defaults that enable hierarchical skill orchestration in Omni‑IO Skills.
Skill Implementation Details
Extension procedures for Atomic, Expert, and Scenario Skills and their orchestration.
This appendix details how developers add new Atomic, Expert, and Scenario Skills and how the system orchestrates them.
An Atomic Skill wraps a capability that the host agent cannot run natively, exposing it as an MCP tool.
An Expert Skill composes existing Atomic Skills into a production workflow that yields a concrete deliverable.
A Scenario Skill orchestrates multiple deliverables by selecting appropriate Expert or Atomic Skills for each.
The Image Generation Skill illustrates an Atomic Skill: it binds the `generate_image` MCP tool, resolves parameters, invokes the provider, writes the image, and registers the asset.
The table outlines two expert skills, "Poster Design" and "Complex Video Production," detailing their respective "Workflow stages" and "Inspection" requirements. The text below the table explains that an Expert Skill distinguishes generative tasks from production operations requiring precise control, delegating content generation to Atomic Skills while handling accuracy, structure, or timing through controllable processing or assembly steps.
This table outlines the "Responsible layer," "Symptom," and "Minimal rework" for various stages of a generative workflow. It includes four rows corresponding to Atomic asset, Written content, Assembly, and Planning layers.
```json { "tasks": [ { "id": "poster_bg", "type": "image", "prompt": "Contemporary music-festival poster background, deep indigo and fluorescent orange, abstract sound waves carrying the rhythm, subject weighted to the lower half, large clean negative space across the upper half, no letters, digits, logos, watermarks or pseudo-text of any kind", "params": { "aspect_ratio": "3:4" } }, { "id": "poster_logo", "type": "image", "asset_ref": "img_7c1e04" }, { "id": "poster_final", "type": "code", "prompt": "Typeset the locked copy over the background, place the logo in the lower right margin, export at 300 dpi as PNG and PDF", "depends_on": ["poster_bg", "poster_logo"] } ] } ```
The table outlines various "Scenario Skills" and their corresponding "Application boundary" and "Candidate deliverables". The rows include Social-Media Post, Office Documents, Job Application, Education Sharing, Event Material, and Game Asset.
The table categorizes asset types by their execution site, multi-dependency status, and auto-registration status.
Algorithm 1: Dependency‑aware orchestration in Omni‑IO Skills
DEG nodes are defined by the fields in Table 13; exactly one of prompt or `asset_ref` must be present.
Execution Constraints and Node Types
This section details node validation, failure handling, and asset‑registry semantics for reliable execution.
The system separates control state (node status) from asset state (the registry). Node status drives scheduling and cancellation, while the registry only records successfully materialised artifacts, allowing independent branches to survive local failures.
Each node carries a status flag – PENDING, COMPLETED, FAILED, or CANCELLED – which determines when the node becomes schedulable or is pruned. When a node fails, its pending descendants are cancelled but sibling branches continue, preserving any completed assets for later reuse.
Before any tool is invoked, the $DEG$ is validated against the rules in Table 15. These rules are decidable from the declaration alone; any violation triggers a planning error and forces a reconstruction of the graph.
Interruption handling follows the classes in Table 16. Only pending or in‑flight nodes are affected; completed assets remain registered, ensuring that partial work is not lost.
Cross‑turn reuse is enabled by the Asset Registry (Table 18). A previously generated asset can be referenced via its `asset_id`, appears as an asset‑source node that is complete on entry, and incurs no generation cost.
**Table 14.** Admissible node types and where they execute. Only the two natively executed types can consume multiple upstream assets, and only they require explicit registration.
This table categorizes different types of revisions and their handling procedures.
**Table 16.** Interruption classes and their handling. Completed assets remain registered; only pending and in-flight nodes are affected.
**Table 17.** Wave partition of Figure 10. Nodes in the same Wave are mutually independent and are dispatched in one concurrent round. The asset-source node is complete on entry and issues no call.
**Table 18.** Fields of an Asset Registry record. Records are keyed by `asset_id` and persisted as a single JSON file in the workspace.
```json { "scenario": "game_asset", "shared_brief": "Forest elf, emerald hair, silver light armour, stylised realism, consistent identity across outputs", "deliverables": [ { "id": "character_key_art", "invoke": "A6 Image Generation" }, { "id": "turntable_clip", "invoke": "A7 Video Generation", "reuse": ["character_key_art"] }, { "id": "character_model", "invoke": "A11 3D Generation", "reuse": ["character_key_art"] }, { "id": "background_track", "asset_ref": "mus_b4e8d2" }, { "id": "showcase_page", "invoke": "A16 Code Generation", "reuse": ["character_key_art", "turntable_clip", "character_model", "background_track"] } ] } ```
**Require:** user request $q$, Skill index $\mathcal{S}$, Asset Registry $\mathcal{R}$ **Ensure:** outputs and final execution states of all nodes 1: $\mathcal{T} \leftarrow \text{SKILLENTRY}(q, \mathcal{S})$ $\quad \triangleright$ selected Skills expanded into terminal tasks 2: $G = (V, E)$ with $V \leftarrow \mathcal{T}$ and $E$ induced by `depends_on` 3: **if** $\text{VALIDATE}(G, \mathcal{R})$ fails **then** 4: $\quad$ **return** planning error $\quad \triangleright$ unresolved reference or dependency cycle 5: **end if** 6: status$[v] \leftarrow \text{PENDING}$ for all $v \in V$ 7: **for all** $v \in V$ with `asset_ref` present **do** 8: $\quad o[v] \leftarrow \mathcal{R}.\text{GET}(v.\text{asset\_ref})$; status$[v] \leftarrow \text{COMPLETED}$ 9: **end for** 10: **while** some $v \in V$ has status$[v] = \text{PENDING}$ **do** 11: $\quad \mathcal{W} \leftarrow \{v \in V : \text{status}[v] = \text{PENDING}$ and all predecessors completed $\}$ 12: $\quad$ **for all** $v \in \mathcal{W}$ in parallel **do** 13: $\quad \quad \pi \leftarrow \text{RESOLVEPROVIDER}(v.\text{type})$ 14: $\quad \quad \theta \leftarrow \pi.\text{defaults} \oplus v.\text{params} \oplus \text{INJECT}(o[\text{pred}(v)])$ 15: $\quad \quad o[v] \leftarrow \text{EXECUTE}(\pi, v.\text{prompt}, \theta)$ 16: $\quad \quad$ **if** execution succeeded **then** 17: $\quad \quad \quad \mathcal{R}.\text{REGISTER}(o[v])$; status$[v] \leftarrow \text{COMPLETED}$ 18: $\quad \quad$ **else** 19: $\quad \quad \quad \text{status}[v] \leftarrow \text{FAILED}$ 20: $\quad \quad \quad$ mark all pending descendants cancelled $\quad \triangleright$ independent nodes continue 21: $\quad \quad$ **end if** 22: $\quad$ **end for** 23: **end while** 24: **return** $(o, \text{status})$