NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harness
NeoHorse Team, Guoliang Cao, Guohao Dai, Tianyu Guo, Kai Han, Hailin Hu, Zihan Jiang, Xiang Kuang, Boxun Li, Yulong Li, Zehua Pei, Yuchuan Tian, Jiamin Wang, Yu Wang, Yunhe Wang, Yihong Wu, Haiyang Xu, Shuo Zhang, Hang Zhou, Siyang Cheng, Jiayu Fan, Wei He, Qingrui Jiao, Hongguang Li, Zhiyuan Li, Runke Liu, Xi Liu, Xinchen Liu, Sinno Jialin Pan, Yi Ren, Liuyang Song, Chenyu Wang, Bei Yu, Quanlu Zhang, Xiangyu Zhang, Mengyu Zheng, Yingjie Zong
NeoHorse-1 uses routing-harness interaction records to create a feedback-driven, recursive training loop for agentic models.
How can we use the data generated by a deployed routing harness to create a feedback loop for recursive self-improvement in agentic models?
Agentic models often struggle to learn from their own failures because they lack a systematic way to convert execution trajectories into structured training feedback. NeoHorse-1 addresses this by using a deployment routing harness to record capability demand and interaction outcomes, which are then used to organize training data into a curriculum and guide on-policy distillation. This feedback-driven post-training lifts the macro-average score of a 4B model from 58.94 to 64.87, effectively narrowing the performance gap between smaller and larger model scales.
Paper Primer
The core mechanism is a routing-guided feedback loop: the system uses the harness's capability-demand predictions to stratify training data, ensuring the model learns from increasingly complex interactions while maintaining broad coverage. This process functions as a curriculum that orders both supervised fine-tuning (SFT) and on-policy distillation (OPD), where the model learns from its own generated prefixes under teacher supervision.
Agentic post-training significantly improves performance across diverse agentic and coding benchmarks.
Macro-average scores increased from 58.94 to 64.87 for the 4B model and from 65.60 to 69.04 for the 9B model. The 4B model's gains were sufficient to substantially narrow the aggregate performance gap to the 9B base model.
Routing-harness trajectories provide superior training signal compared to static public agent datasets.
Controlled experiments showed that routing-harness data outperformed public synthetic tool-agent data (Toucan) across all tested benchmarks, with an average improvement of 6.26 percentage points.
Why is a routing harness necessary for this training approach?
The harness provides the "capability demand" signal—a prediction of how difficult a task is—which allows the system to organize training examples into a curriculum rather than treating all interaction data as equally informative.
How does this approach differ from standard supervised fine-tuning?
Standard SFT typically imitates fixed teacher trajectories; NeoHorse-1 uses on-policy distillation to supervise the model on its own generated prefixes, ensuring the training distribution aligns with the model's evolving behavior in the execution environment.
By treating the deployment harness as a source of both experience and capability-demand feedback, NeoHorse-1 demonstrates that agentic models can be improved through a self-reinforcing loop that automates the transition from raw interaction to structured learning.
The Case for Routing-Guided RSI
Static instruction sets miss real‑world nuance; the routing harness supplies the feedback loop for RSI.
Training agentic models on static instruction sets fails to capture the evolving capabilities that arise during real‑world interaction. A concrete mechanism is needed: the system must observe its own performance and turn that evidence into the next learning round. The existing routing harness already records execution trajectories and capability signals, providing exactly such a feedback channel.
RSI is the process by which an AI system uses its own observed behavior to generate the data that will train its next version, creating a self‑reinforcing loop of capability growth.
The routing harness is the execution layer that routes user requests to the most suitable model in a heterogeneous pool, while logging the chosen model, the predicted capability demand, and the resulting interaction.
**Figure 2.** Towards RSI through routing-guided agentic training. Diverse tasks generate interaction experience through routing harnesses backed by heterogeneous model pools. This experience is organized into a training mixture for NeoHorse-1. Capability feedback guides the next training distribution, while updated models return to the harness for subsequent iterations. The agentic training stack summarizes the post-training methods described in this report.
The routing harness is an untapped data source for agentic improvement.
Agentic Post-Training Landscape
We survey prior work on post‑training, routing, and recursive improvement relevant to our approach.
Building on the earlier discussion of Recursive Self‑Improvement and the Routing Harness, we now review related work that informs our post‑training, routing, and curriculum strategies.
FireAct is a trajectory‑based fine‑tuning method that learns from logged execution traces to improve model behavior.
AgentTuning refines a pretrained model by directly optimizing on agentic interaction data rather than static text.
2.1 Agentic Model Post‑Training – Prior work explores how interaction data can be leveraged to improve models after their initial pretraining.
Extends FLAN by fine‑tuning on large‑scale instruction data, demonstrating that diverse task mixtures improve generalization.
A repository of agentic interaction logs used to train models on real‑world tool usage.
Introduces synthetic multi‑step tool‑use data and iterative SFT, improving reasoning over complex tasks.
Distills a student model by having it generate its own trajectories while a teacher provides token‑level supervision.
Uses a fixed teacher policy to generate trajectories that the student imitates.
Shows that the design of the interaction harness (tool selection, observation handling) strongly influences agent robustness.
Analyzes how post‑training methods interact with the underlying harness, affecting overall agent behavior.
Provides explicitly structured, environment‑grounded trajectories for terminal‑level decision making.
Replaces fixed behavioral labels with rewards derived from tool execution and task outcomes.
Applies the agentic routing paradigm to web‑search tasks, learning to route queries to specialized search models.
Investigates tool‑use in LLMs by routing requests to specialized tool modules.
Studies long‑horizon interaction by combining routing decisions with hierarchical planning.
Keeps the environment loop inside the deployment harness, enabling end‑to‑end training without external rollout.
Jointly updates the routing harness and the model, allowing mutual adaptation.
2.2 LLM Routing and Curriculum Learning – Recent work leverages routing signals to guide both inference cost and training difficulty.
Introduces cost‑aware model cascades that route queries to cheaper models when possible.
Learns a routing policy between strong and weak LLMs using preference data.
Extends routing to multi‑agent LLM systems, selecting the most suitable sub‑agent for each request.
Applies curriculum strategies to LLM post‑training, ordering examples by difficulty.
2.3 Recursive Self‑Improvement – A body of work studies how agents can iteratively improve themselves and their supporting infrastructure.
Evolves both task skills and the meta‑skill that governs their improvement.
Alternates evidence gathering with answer auditing to iteratively refine agent performance.
Uses failures and observability data to automatically update the interaction harness.
Systematically engineers the routing harness to improve agentic performance across tasks.
Optimizes the harness after deployment by analyzing past trajectories and adjusting routing policies.
Extends harness optimization to a reset‑free online setting, enabling continual adaptation.
Benchmarks agents’ ability to modify their own training algorithms for subsequent improvements.
Harvesting Agentic Trajectories
Describes the routing harness data pipeline, its composition, quality checks, subscene characterization, and capability‑guided allocation.
The routing harness records per‑turn capability predictions, policy adjustments, and served tiers, turning execution traces into a curriculum signal.
An Agentic Trajectory is the complete execution trace of a user request, encompassing model reasoning, tool actions, environment observations, and the final outcome.
A subscene groups consecutive user turns that share a local goal, forming a semantic unit for scene, goal, and outcome attributes.
The routing harness predicts the capability demand of each turn, optionally adjusts it via policy, and records the tier actually served, yielding a prediction–action–outcome triple.
Using routing‑derived signals, the training mixture is reweighted toward subscenes where the model underperforms, iteratively closing capability gaps.
Compute a base weight of 1 for every subscene.
Multiply the weight by 2 for any subscene with tier ≥ C2 (reflecting higher capability demand).
Further multiply by 3 for FAIL, by 2 for WARN, and leave unchanged for PASS (emphasizing deficiency).
Resulting weights: S₁ → 2 × 1 = 2, S₂ → 2 × 3 = 6, S₃ → 1 × 1 = 1, S₄ → 2 × 2 = 4.
Normalize the weights to form a sampling distribution: total = 13, so probabilities are S₁ = 15.4 %, S₂ = 46.2 %, S₃ = 7.7 %, S₄ = 30.8 %.
This simple weighting pushes the curriculum toward high‑tier, low‑quality subscenes, while still preserving some exposure to already‑good examples.
Collect raw execution trajectories from the deployment harness.
Deduplicate and decontaminate the corpus, removing any overlap with evaluation suites.
Validate structural integrity of each trajectory (request/response, tool call/result pairing, causal ordering).
Assign subscene boundaries and annotate Scene, Goal, and Outcome attributes.
Record routing signals per user turn: raw capability prediction, policy‑adjusted decision, and served tier.
Evaluate the current model on a disjoint validation set to build a deficiency profile.
Reweight the training mixture using the deficiency profile and routing tiers.
Feed the rebalanced data into the SFT curriculum and on‑policy distillation stages.
**Figure 3.** Subscene-level scenario characterization. A trajectory is represented as an ordered event stream of user queries and LLM calls; adjacent user turns that share a local goal form a subscene. The selected subscene is described through three complementary views—Scene, Goal, and Outcome—with representative attributes shown on the right.
The NeoHorse-1 Training Pipeline
Turn recorded interactions into a curriculum that teaches the model to act agentically.
Static instruction‑set training ignores the back‑and‑forth of real interactions, so models miss the nuanced decision‑making needed for agentic behavior. The trick is to harvest recorded agentic trajectories, turn each user turn into a supervised example, and then drive learning with a routing‑guided curriculum that respects capability demand.
Instead of feeding the model random examples, we let the routing harness score each interaction for “how demanding” it is and then feed easier examples first, gradually increasing the demand.
Hard ordering yields A → B → C.
Soft scores: $s_A=0\cdot0.9+1\cdot0.1=0.1$, $s_B=0\cdot0.2+1\cdot0.5+2\cdot0.3=1.1$, $s_C=1\cdot0.1+2\cdot0.6+3\cdot0.3=2.5$.
Sorting by $s_i$ gives the same order A → B → C, but if B had $\pi=[0.4,0.4,0.2,0]$ its soft score would be $0.8$, placing it closer to A.
Soft scores let the curriculum distinguish examples that share a hard tier, ensuring a smoother difficulty ramp.
**Figure 5 | Routing-guided curriculum.** Routing scores serve as a proxy for capability demand. The training mix shifts toward higher-scored examples over three approximately equal-sized stages, while some lower-scored examples are reserved for later stages. Color intensity schematically indicates the routing-score distribution within each stage.
After the model learns from recorded turns, we let it generate its own prefixes and then teach it to match a fixed teacher, again using routing scores to decide which prefixes appear early or late in training.
Stage 1: student generates short replies; teacher KL loss is modest because the prefixes are easy.
Stage 2: student sees longer, more structured prefixes; KL gradients become larger, pushing the model to handle tool calls.
Stage 3: student must continue a multi‑turn dialogue; the reverse KL penalizes any deviation from the teacher’s sophisticated token choices.
Gradually increasing context difficulty mirrors how humans learn: start with simple prompts, then practice on harder, multi‑step interactions.
**Figure 6 | Routing-guided on-policy distillation.** Routing scores schedule the recorded starting contexts over three stages, with lower-scored contexts reserved for later stages. The student generates responses from these contexts. At each generated prefix, the student and a fixed teacher provide next-token distributions for the reverse-KL objective in Equation 3. Both distributions use the same top-K candidate tokens and a bin for the remaining probability mass. Only the student is updated; refreshed student checkpoints generate subsequent responses. Rollout and scoring use the same student model, with parameters updated during training. Token strips and probability bars are schematic.
Identify a user turn: a user request plus all assistant responses and tool interactions up to the next user request.
Serialize the turn with the Qwen 3.5 chat template, keeping reasoning from the current turn and discarding reasoning from earlier turns.
Build a binary mask $m_{i,t}$ that marks tokens belonging to assistant targets (reasoning, tool calls, visible responses, EOS).
Apply the masked SFT objective (Eq 1) to maximize the likelihood of the marked tokens while ignoring all other tokens.
**Figure 4 | Agentic supervision within a user turn.** A recorded interaction (top) is converted into a training sequence for the current user turn (bottom). Earlier reasoning is omitted, while visible responses and tool interactions remain as historical context. In the current user turn, assistant target spans receive prediction loss; user messages and tool results do not. Reasoning is retained when present, and assistant targets include end-of-response tokens. The sequence uses causal attention. System instructions, tool specifications, and retained harness messages are omitted from the schematic; they receive no prediction loss. Block widths do not reflect token counts.
Benchmark Performance
NeoHorse‑1 delivers top‑line scores across agentic, coding, and instruction benchmarks.
Recall that NeoHorse‑1 leverages the Routing Harness to harvest and curriculum‑train on high‑quality Agentic Trajectories, turning static instruction sets into interactive expertise.
NeoHorse‑1‑4B achieves the highest average score (64.87) across all evaluated benchmarks, surpassing every 4B‑scale baseline.
Table 1 shows the average across agentic, coding, and instruction‑following tasks.
**Figure 1 | Comparison on six agentic benchmarks in the 4B and 9B tracks.** Orange bars denote NeoHorse-1-4B (top) and NeoHorse-1-9B (bottom); gray bars denote the comparison models in each track, with model sizes indicated in the legends. Full results are reported in Tables 1 and 2.
Ablations and Qualitative Analysis
Ablation results show larger models and routing‑harness data improve agentic tasks, while scaling supervision yields steady gains.
We evaluate three ablations: model scale, source of agentic trajectories, and amount of routing‑harness supervision.
NeoHorse‑1‑9B attains the highest average benchmark score of 69.04, surpassing all compared larger models.
Table 2 shows NeoHorse‑1‑9B leading in Agentic, Coding, and Instruction‑Following categories.
Routing‑harness trajectories improve average performance by 6.26 points over public synthetic agent data.
Table 3 reports a +6.26 average gain across five benchmarks when using routing‑harness data.
Increasing routing‑harness supervision to 10 M tokens raises the average score from 69.31 to 71.45 (+2.14).
Figure 7 plots the monotonic rise in performance as supervised token count grows.
The table presents a performance comparison of various models across three categories: Agentic, Coding, and Instruction Following, with an overall average score.
**Table 3.** Comparison of routing-harness trajectories with public tool-agent data under the same routing-guided training configuration. Higher is better; the final row reports absolute percentage-point differences, and Avg. is the unweighted mean across the five benchmarks.
**Figure 7.** Scaling routing-harness supervision.
**Figure 8.** Original pages after the same 26-click replay. Left: Qwen3.5-9B remains empty. Right: NeoHorse-1-9B displays 13 black and 13 white stones in an ongoing position.
Summary and Future Directions
We recap NeoHorse‑1’s harness‑driven self‑improvement loop, its current limits, and next steps.
Training static instruction‑following models overlooks real‑world interaction; NeoHorse‑1 instead taps the existing routing harness to harvest execution trajectories, routing signals, and recorded outcomes. These signals are turned into user‑turn training examples that retain interleaved reasoning, tool calls, and context, then validated via structural checks, six‑dimensional semantic evaluation, and subscene‑level labeling. The method organizes the data with routing scores into a three‑stage curriculum for supervised fine‑tuning and routing‑guided on‑policy distillation, while a capability‑guided allocation step closes the evaluation–selection–update loop.
Across ten benchmarks, post‑training consistently improves performance: the 4B model’s macro‑average rises from 58.94 to 64.87 and the 9B model’s from 65.60 to 69.04, with larger models still excelling on iterative debugging and long‑sequence tasks. The gains also demonstrate a preliminary closure of the recursive self‑improvement loop, as the 4B post‑trained model narrows the gap to the 9B base and feeds new trajectories back into the harness. Nonetheless, this is a single‑pass evaluation on a limited capability set; future work will iterate the loop across generations, broaden task coverage, and refine the routing signal into calibrated difficulty estimates.
Implementation Details and Contributions
Provides author contributions and detailed case-study evidence tables.
Authors are listed alphabetically by surname, with asterisks marking corresponding authors.
Core contributors include Guoliang Cao, Guohao Dai, Tianyu Guo, Kai Han, Hailin Hu, Zihan Jiang, Xiang Kuang, Boxun Li, Yulong Li, Zehua Pei, Yuchuan Tian, Jiamin Wang, Yu Wang, Yunhe Wang, Yihong Wu, Haiyang Xu, Shuo Zhang, and Hang Zhou.
Additional contributors are Siyang Cheng, Jiayu Fan, Wei He, Qingrui Jiao, Hongguang Li, Zhiyuan Li, Runke Liu, Xi Liu, Xinchen Liu, Sinno Jialin Pan, Yi Ren, Liuyang Song, Chenyu Wang, Bei Yu, Quanlu Zhang, Xiangyu Zhang, Mengyu Zheng, and Yingjie Zong.
Affiliations span TokenRhythm Technologies, Infinigence AI, Tsinghua University, Peking University, The Chinese University of Hong Kong, Visionplus Capital, WX Capital, and Alibaba Group.
The case‑study section evaluates NeoHorse‑1‑9B against Qwen3.5‑9B on three distinct tasks.
Case A asks the model to generate a Chinese Markdown ticket report for May 18, 2026, respecting temporal and audit constraints while merging aliases and selecting the latest eligible updates.
NeoHorse‑1‑9B correctly reports 50 eligible records and 19 tickets, matches an independent recomputation, and excludes prohibited next‑day updates, whereas Qwen3.5‑9B retains contradictory totals and mis‑assigns a next‑day state.
Case B requires implementing a time‑leakage auditor that respects a five‑minute tolerance, preserves input order, and outputs clean and rejected sample schemas.
NeoHorse‑1‑9B follows the five‑minute tolerance and maintains the required output schema, while Qwen3.5‑9B uses a two‑second tolerance, incorrectly rejects a valid sample, and alters the output structure.
Case C asks the model to produce HTML for a simple two‑player Gomoku game and execute a 26‑click sequence.
NeoHorse‑1‑9B preserves stone positions, colors, and turn order without runtime errors; Qwen3.5‑9B fails to handle moves, producing invalid board indices.
**Table 4.** Execution and artifact evidence for Case A.
**Table 5.** Repository requirements, implementation, and execution artifacts for Case B.
**Table 6 | Execution and artifact evidence for the Gomoku case.**