Beyond Simply Environment Scaling: Designing Effective Environment Distributions for Multimodal Agent Learning

Kejian Zhu, Zhuoran Jin, Dongqi Huang, Hongbang Yuan, Yupu Hao, Kang Liu, Jun Zhao

Effective multimodal agent training requires optimizing environment distributions for ability diversity and hierarchical difficulty, not just scaling.

How can we select and sequence multimodal environments to improve agent learning efficiency, rather than simply scaling the number of environments?

Simply increasing the number of multimodal environments often leads to performance degradation due to negative transfer and optimization conflicts between tasks. The authors propose a two-part framework: Ability-aware Environment Selection (AES) to curate a diverse, non-redundant environment subset, and Hierarchical Difficulty Curriculum (HDC) to manage training through scaffold removal and state-scale progression. This approach achieves a 143.2% average relative gain over base models, significantly outperforming naive environment scaling.

Paper Primer

Multimodal agents struggle with two primary bottlenecks: unreliable visual state extraction and flawed world modeling. The authors demonstrate that these issues are exacerbated by "mixed-training failure," where training on large, uncurated environment pools causes gradient conflicts and negative transfer, effectively stalling agent progress.

AES selects environments by decomposing agent trajectories into "meta-ability profiles" and maximizing coverage while minimizing redundancy and gradient-based optimization conflicts. HDC then organizes training into a two-level hierarchy: harness weakening (gradually removing textual scaffolds like hints or rule descriptions) and state-scale progression (increasing environment complexity within each harness stage).

AES and HDC significantly outperform naive environment scaling.

The combined framework achieved a 143.2% average relative gain across ID/OOD settings and model scales compared to the base model.

AES effectively identifies a high-quality subset that outperforms the full environment pool.

AES-selected environments achieved a 95.6% relative gain over the base model, compared to 43.4% for the full, uncurated pool.

Why is "naive scaling" of multimodal environments problematic?

Unlike text-symbolic tasks, multimodal environments exhibit stronger negative transfer and polarized gradient conflicts, meaning that adding more environments can introduce optimization interference that degrades overall performance.

What is the role of "harnesses" in the proposed curriculum?

Harnesses are textual scaffolds (e.g., rule descriptions or state hints) that assist the agent in overcoming visual extraction and world modeling bottlenecks; the curriculum gradually removes these to force the agent to rely on raw visual observations.

The authors assume that environment effectiveness is best measured by the underlying "meta-abilities" required to solve tasks, rather than surface-level task descriptions or environment code, which often fail to capture the actual learning dynamics of the agent.

Researchers should shift focus from simply increasing environment counts to curating distributions that balance broad ability coverage with a structured, multimodal-specific difficulty progression.

The Scaling Bottleneck in Multimodal Agents

We expose why more multimodal environments can hurt learning and outline our two‑dimensional solution.

Recent work builds ever‑larger multimodal environment pools, assuming that more environments automatically improve agent learning. Empirically we find the opposite: performance often fluctuates or even degrades as the pool grows, especially for multimodal tasks where gradient conflicts become severe.

Effective training therefore requires two complementary design axes. Diversity must cover a broad set of abilities while avoiding redundancy and conflict; we address this with Ability‑aware Environment Selection (AES). Difficulty must provide a meaningful progression, which we achieve via Hierarchical Difficulty Curriculum (HDC) that weakens auxiliary harnesses and scales state complexity.

**Figure 1.** Environment distribution needs to be evaluated from two dimensions: diversity and difficulty level.

**Figure 2.** An overview of our methods for evaluating and designing effective environment distribution.

**Figure 7.** Multimodal environment scaling under an expanded training budget. We fix the number of training samples per environment to 256.

**Figure 3.** Effect of simply environment scaling.

Scaling environments is not a linear path to performance; careful selection and curriculum are essential.

Environment Pool Construction

We characterize the multimodal environment pool and its training dynamics.

We collect and unify 200 multimodal environments, each instantiated in a text‑symbolic version (visual observations rendered as text) and a multimodal version (raw visual inputs). For every version we train separate agents on individual environments and a single agent on the mixed‑environment set.

Comparing mixed‑environment training to single‑environment baselines reveals that the multimodal version suffers a substantially larger performance drop than the text‑symbolic version, indicating stronger negative transfer when visual inputs are present.

We quantify training conflicts by computing gradients $g_i$ and $g_j$ for pairs of environments and measuring cosine similarity $\cos(g_i, g_j)$. The multimodal version exhibits more extreme positive and negative similarities, showing that visual richness amplifies optimization conflicts even when task logic is identical.

Manual error analysis of 200 failure trajectories of Qwen3‑VL‑4B highlights two multimodal‑specific bottlenecks: unreliable visual‑state extraction and inaccurate world modeling, both of which persist despite increasing the environment count.

Increasing the number of environment types while keeping compute fixed does not yield monotonic performance gains; at certain scales performance even declines, suggesting that naïve scaling introduces redundancy and conflict.

To expose the underlying ability requirements we collect 40 trajectories per environment from two models of differing capability, then use GPT‑5 to segment each trajectory into interpretable atomic abilities. These meta‑ability profiles feed the Ability‑aware Environment Selection (AES) pipeline, which also leverages gradient‑based conflict analysis to pick a compact, complementary subset of environments.

**Table 1.** Results of mixed-environment training. Avg. Single denotes the average performance of separately trained single-environment models, while Avg. Mixed denotes the average performance of one model trained on the mixed environment set.

**Figure 4.** Gradient cosine similarity analysis between environments under different versions.

**Figure 5.** Error analysis of multimodal agents.

Ability-aware Environment Selection

Select a compact, diverse set of environments by maximizing weighted ability coverage while penalizing redundancy and conflict.

Training on the full multimodal pool wastes compute because many environments repeat the same skills while others are mismatched to the agent’s current ability. The remedy is to pick a small subset that together spans the agent’s core capabilities, yet avoids redundant or conflicting examples.

AES greedily builds a set of environments whose combined core abilities cover the whole ability space, giving core abilities extra weight and penalizing environments that look alike or interfere during joint training.

Initial set $S=\emptyset$; compute $NewCoverage$ for each $e_i$: $e_1$ adds A+B (weight 4), $e_2$ adds B+C (4), $e_3$ adds A+C (4), $e_4$ adds A+S (3).

All $Redundancy$ values are 0 because $S$ is empty; all $Conflict$ values are 0.

With $\lambda_1=1,\lambda_2=\lambda_3=0.5$, $Gain(e_1)=4$, $Gain(e_2)=4$, $Gain(e_3)=4$, $Gain(e_4)=3$ → pick $e_1$ (tie broken arbitrarily) and add to $S$.

Update $C(S)$ to {A,B}. Re‑compute for remaining candidates: $e_2$ now contributes only C (weight 2), $Redundancy(e_2)=0.2$, $Conflict(e_2)=0.5\cdot0.3=0.15$ → $Gain(e_2)=2 -0.1 -0.075=1.825$.

$e_3$ contributes only C (2) with $Redundancy=0.2$, $Conflict=0$ → $Gain(e_3)=2-0.1=1.9$ (higher). $e_4$ contributes A (already covered) and S (1) with $Redundancy=0.2$, $Conflict=0$ → $Gain(e_4)=1-0.1=0.9$.

Select $e_3$, update $S=\{e_1,e_3\}$; now core abilities A,B,C are all covered, so the algorithm stops.

The greedy gain balances adding new core abilities against similarity and gradient conflict, so the final set contains the minimal number of environments that achieve full core coverage.

Compute meta‑ability profiles $P_e$ for all environments in the pool.

Initialize the selected set $S \leftarrow \emptyset$.

For every candidate $e$, evaluate $Gain(e \mid S)$ using the three formulas.

Select the environment with the highest gain and add it to $S$.

Update $C(S)$, $Redundancy$, and $Conflict$ for the remaining candidates.

Repeat steps 3–5 until all core abilities are covered or a budget (e.g., 30 environments) is reached.

How does AES differ from a naïve diversity filter that only looks at surface task descriptions?

Surface filters treat two environments as different if their textual descriptions differ, even if they require the same underlying abilities. AES operates on the *ability* level: it weights core abilities, removes low‑frequency skills, and explicitly penalizes similarity and gradient conflict, ensuring the selected set is diverse in what the agent actually has to learn, not just in wording.

**Figure 6.** Core ability coverage curve when selecting environments using AES.

**Figure 8.** The distribution of the 30 environments selected by AES across the profile space of 200 environments.

Hierarchical Difficulty Curriculum

We introduce a two‑dimensional curriculum that weakens textual scaffolds while scaling environment complexity.

Early training on raw visual inputs yields sparse rewards and unstable learning because agents cannot yet extract visual states or infer environment rules. To bridge this gap we introduce auxiliary textual scaffolds—called harnesses—that supply symbolic observations, state summaries, hints, or explicit rules. Weakening these scaffolds over time creates a curriculum that gradually forces the agent to rely on raw perception.

HDC treats difficulty as a two‑dimensional lattice: an outer axis that slowly removes textual scaffolds, and an inner axis that grows the raw state complexity within each scaffold level.

Sample $e_1$; draw harness $h\sim D_{e_1}$ → with probability 0.6 pick $h=0$, otherwise $h=0$ (only level 0 exists).

Sample state scale $s\sim\text{Uniform}\{0,1\}$ → suppose $s=1$ (larger grid).

Train on $(e_1, h=0, s=1)$; success rate rises to 0.78 > $\tau_s$, so increment $u_{e_1}\leftarrow2$ (window now $[1,2]$).

Later, success stays ≥ 0.80 for three consecutive updates; advance outer frontier $r_{e_1}\leftarrow1$ (remove one scaffold) and reset $u_{e_1}=1$, $\ell_{e_1}=0$.

Repeat for $e_2$; because its early successes are slower, it remains at $r_{e_2}=0$ while $e_1$ progresses to $H_1$.

HDC lets each environment climb its own difficulty ladder, preventing a single hard environment from stalling the whole pool.

Hierarchical Difficulty Curriculum (Algorithm 1)

How does HDC differ from a simple linear difficulty schedule that increases grid size over epochs?

A linear schedule treats difficulty as a single monotonic axis and applies the same progression to every environment. HDC decouples scaffold removal (outer axis) from state scaling (inner axis) and lets each environment advance independently, which avoids forcing easy environments to wait for the hardest ones.

Table 2 quantifies how each harness type boosts single‑turn success (ST) and multi‑turn return (MT); full scaffolds raise MT from 11.9 % to 30.5 %, demonstrating that auxiliary text dramatically stabilizes reward signals. Table 7 (Appendix) lists which scaffolds survive at each outer level $H_i$, confirming the gradual weakening design.

**Figure 9.** The training curve of HDC on Frozen Lake.

**Figure 10.** Corrected error analysis when learning algorithm using HDC.

Main Experimental Results

AES and HDC beat random baselines by large relative gains.

The central premise is that training on a massive, undifferentiated pool of multimodal environments wastes compute because many environments are redundant or mismatched to the agent’s current abilities. Selecting a diverse, ability‑aligned subset (AES) and ordering it by difficulty (HDC) therefore yields more efficient learning.

Pick a fixed number K of environments at random from the full pool, ignoring any notion of ability or diversity.

Why not simply increase K instead of using AES?

Increasing K adds more redundant or conflicting environments, which dilutes the learning signal and raises compute cost. AES deliberately picks the most informative subset, so a smaller K can outperform a larger, uncurated set.

AES + HDC achieves a 143.2 % relative gain over the base model, far surpassing Random‑K’s 73.7 % gain when paired with the curriculum.

Table 5 shows the combined method attaining the highest scores across all ID and OOD environment groups.

All experiments use Qwen3‑VL‑4B/8B‑Instruct as the backbone. Random‑K draws 30 environments from the 200‑environment pool, preserving the original category distribution (Wang et al., 2026b; Meng et al., 2026). Detailed hyper‑parameters are in Appendix B.

HDC’s ablations (Table 5) reveal that scaling‑only curricula add an 11.5 % relative boost, while harness‑only adds 18.1 %. Combining both axes yields the strongest improvement of 27.7 % relative gain.

AES and HDC consistently outperform random scaling, delivering large relative gains across both in‑distribution and out‑of‑distribution environments.

Related Work and Summary

We situate our work among recent multimodal environment research and discuss its broader implications.

Recent work has pushed multimodal environment and agent learning to the forefront, with studies exploring general multimodal settings and verifying environment quality at the executable and state‑transition levels.

While sample‑level quality has been addressed, systematic evaluation of environment distribution effectiveness—particularly diversity and difficulty structures—remains underexplored.

Our work therefore isolates diversity and difficulty, proposing Ability‑aware Environment Selection and Hierarchical Difficulty Curriculum, which consistently improve multimodal agent training.

Limitations include reliance on existing multimodal pools, a unified compute budget that may leave some environments under‑trained, and the extra offline cost of gradient‑based conflict estimation in AES.

Ethically, the methods are evaluated only in controlled research environments; deployment without safeguards could lead to unsafe decisions, and all artifacts respect original licenses.

Ability-aware Selection Implementation

Implementation details of Ability-aware Environment Selection and its components.

This section spells out how Ability-aware Environment Selection (AES) builds its environment pool, from raw trajectories to the utility score that drives selection.

We break each agent trajectory into short, reusable behavior units that can be recombined across environments.

How does GPT‑5 decide where one atomic ability ends and the next begins?

The prompt asks the model to list “distinct, self‑contained actions” in the trajectory. GPT‑5 leverages its internal world‑model to detect natural boundaries (e.g., a change in goal, a new interaction object). The subsequent manual inspection catches any over‑ or under‑segmentation.

We aggregate atomic abilities across many trajectories to form a concise, weighted description of each environment’s demands.

What criterion separates a core ability from a soft ability?

After frequency filtering, GPT‑5 is prompted to label each ability. An ability is marked “core” if it directly influences task success (e.g., moving to a goal). Anything that only affects output style or bookkeeping is labeled “soft”.

The selection utility balances three goals: covering many core abilities, avoiding redundant environments, and minimizing conflict.

Why are soft abilities weighted at only 0.3 × their frequency?

Soft abilities are auxiliary; without down‑weighting they could dominate the similarity computation (e.g., “answer formatting” appears in many environments). Scaling them ensures the redundancy estimate reflects truly task‑critical overlap.

Generate 20 trajectories per environment using Qwen3‑VL‑4B and Gemini‑3‑Flash.

Segment each trajectory into atomic abilities with GPT‑5 (prompt from Appendix F) and manually refine granularity.

Merge atomic abilities across 40 trajectories per environment, de‑duplicate synonyms via GPT‑5.

Filter out abilities whose occurrence frequency < 10 % (treated as noise).

Classify remaining abilities into core and soft categories; assign frequency‑based weights (soft × 0.3).

Construct ability‑graph edges from observed transitions between abilities.

Combine core abilities, soft abilities, graph edges, and frequencies into a meta‑ability profile for each environment.

Compute utility for each candidate environment: maximize core‑ability coverage, penalize redundancy (max cosine similarity), and penalize conflict (gradient overlap).

Select the top‑k environments according to utility to form the training pool.

**Table 7.** Harness levels used in the outer curriculum.

Curriculum Design Details

Ablation analysis quantifies the impact of removing the Hierarchical Difficulty Curriculum.

We evaluate how much the Hierarchical Difficulty Curriculum (HDC) contributes by ablating it and measuring the resulting performance drop.

Algorithm 1 – Hierarchical Difficulty Curriculum (nested harness‑annealed loop)

The harness level defines how much auxiliary information is available: H0 provides full textual observations, hints, and rule descriptions, while higher levels progressively strip away these cues, ending at H4 with only basic information.

The table lists various environments, their support status, and the corresponding scale parameters used for each.

Removing the ability‑aware selection (i.e., using representation‑based methods) reduces the relative OOD performance by 40.3 points.

Table 9 shows AES achieving a relative score of 40.3 versus 0.0 for the base model and far lower scores for description and code embeddings.

**Table 8.** Scale controllability of selected environments.

Training on all environments without the curriculum drops single‑turn success from 13.8 % to 9.5 %, a 4.3‑point decline.

Table 10 lists ST = 13.8 for AES versus ST = 9.5 for the “All Envs.” baseline.

**Table 9.** Comparison with representation-based environment selection. Description Embedding selects environments based on embeddings of textual environment descriptions, while Code Embedding selects environments based on embeddings of key environment code.

**Table 10.** Main results under the expanded-budget setting. Each training environment contributes 256 samples. ST denotes single-turn success rate and MT denotes normalized return in multi-turn environments.

These ablations collectively confirm that the Hierarchical Difficulty Curriculum is essential: removing it either by discarding ability‑aware selection or by forgoing the curriculum altogether leads to sizable performance degradations.

Robustness and Scaling Analysis

Key performance gains from AES and HDC across models and environment budgets.

We evaluate whether the Ability‑aware Environment Selection (AES) subset generalizes to a different model and examine scaling behavior when the training budget expands.

AES selected by Qwen3‑VL‑4B raises OOD‑ST from 13.4 to 18.3 when training InternVL3‑8B.

Table 13 shows the OOD‑ST improvement of +4.9 points over Random‑K.

With Hierarchical Difficulty Curriculum (HDC), the AES subset outperforms Random‑K by 3.1 OOD‑ST points.

Table 13 records a +3.1 advantage for OOD‑ST under HDC.

In expanded‑budget experiments, ID success peaks at 66.3 % with 40 environments.

Training each environment with 256 samples yields the highest ID success at 40 environments.

OOD success reaches its maximum of 36.4 % with 60 environments.

Expanded‑budget runs show the OOD success curve peaks at 60 environments.

**Table 11.** Transfer matrix in the text-symbolic setting. Each row denotes the training environment, and each column denotes the evaluation environment. All values are reported as accuracy in percentage. Colored values in parentheses denote the absolute accuracy change relative to the Base Model on the same evaluation environment. Diagonal entries correspond to single-environment training performance. The last row reports the change from mixed training to the corresponding single-environment result, i.e., $\Delta_j = M_j - T_{j,j}$.

This table lists various environments and their corresponding task descriptions.

Extended Related Work

Survey of prior work on multimodal environment construction, diversity, and curriculum.

Recent work has moved from static multimodal datasets toward dynamic, interactive environments, enabling agents to act and receive feedback over time.

Efforts to scale environment construction span Web and GUI platforms, game‑based reinforcement‑learning settings, and benchmark suites that standardize multimodal agent evaluation.

Systems such as Gym‑V and VisGym provide unified, multi‑turn interaction frameworks that serve as foundations for studying large‑scale multimodal training.

Sample‑level quality checks focus on executability and reliable reward signals, using sandbox execution, unit tests, golden trajectories, and interaction‑consistency metrics to verify individual environments.

Diversity is typically controlled either by removing semantically similar tasks via embedding similarity or by structuring environments through tool‑category analysis and scenario coverage.

These approaches measure surface‑level diversity (e.g., task descriptions) and may miss the underlying ability diversity that agents actually develop during training.

Difficulty grading often relies on structural proxies such as tool count or code size, or on performance estimates from strong models or human experts.

Curriculum‑learning methods—including RLVE, ADCL, VCRL, AdaCuRL, and WebRL—adapt difficulty dynamically, but most treat multimodal difficulty as a function of task scale rather than modality‑specific ability bottlenecks.

Unsupervised Environment Design (UED) generates or selects environments online, with approaches such as PAIRED (adversarial minimax‑regret), PLR (prioritized replay), REPAIRED, ACCEL, Diversity‑UED, and CENIE (curriculum‑aware novelty).

In contrast, Ability‑aware Environment Selection (AES) and Hierarchical Difficulty Curriculum (HDC) operate on a fixed, heterogeneous pool, focusing on selection and sequencing rather than online generation.

Experimental Configuration

Detailed configurations for training, evaluation, and experiment groups.

This section enumerates the concrete configurations used throughout the study. First we describe the settings common to all runs, then we detail the parameters for each experimental group.

Training uses Qwen3‑VL‑4B‑Instruct as the base model; a larger Qwen3‑VL‑8B‑Instruct is added in the main experiments to test scalability. All runs follow the RLVE protocol, consuming exactly 7,680 training samples and a rollout group size of 16 per prompt, executed on four A100 GPUs (80 GB each).

Evaluation measures both in‑distribution (ID) and out‑of‑distribution (OOD) performance. ID uses unseen instances from training‑time environment types, while OOD draws 30 held‑out environments from a pool of 200 that never appear during training; each environment type contributes 100 evaluation instances.

Preliminary experiments explore three questions. The environment‑scaling study (Section 2.1) fixes compute while varying the size of nested environment sets, using the smallest ID set of 20 environments for all models. Gradient‑conflict analysis (Section 2.2) samples ten batches per environment, each batch containing ten samples and sixteen rollouts, then computes per‑batch gradient directions. Error analysis (Section 2.3) relies on two annotators who label model errors, with disagreements resolved to produce the final label set.

Main experiments repeat the training setup with the larger Qwen3‑VL‑8B‑Instruct model and keep the compute budget identical across methods. The “All Envs” condition trains on 170 environments (the full pool after removing the 30 OOD environments). The AES‑selected set contains 22 single‑turn and 8 multi‑turn environments, while the OOD held‑out set comprises 20 single‑turn and 10 multi‑turn environments.

For ID evaluation, most methods treat the 30 AES‑chosen environments as the in‑distribution set; Random‑K instead uses 30 randomly selected training environments. Additionally, we assess general multimodal capability on three external benchmarks—MathVision, MMMU (validation split), and MMStar—using the lmms‑eval suite.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers