Learning from Fleet Experience: A Collective Learning Paradigm for Robots
Chao Yu, Shu'ang Yu, Hao Lin, Hongzhi Zang, Yuxiang Yan, Chenxi Jiang, Zhen Guo, Chuankang Li, Yihao Chen, Xin Xu, Xu Fu, Jiakai Zhou, Liangzhi Shi, Hongyi Ren, Yiwei Liu, Jing Wu, Yuanqing Wang, Chen Wu, Bo Dai, Chunyu Yang, Hongliang Huang, Shuoyu Yue, Weiqi Yin, Yinuo Chen, Ruihua Chen, Chunyang Zhu, Han Xiao, Yanghao Li, Ruipu Wu, Zhikai Cai, Xiu Chen, Xu Tan, Yuchen Yan, Jialin Zhang, Wen Zeng, Yiming Zhou, Zixiao Huang, Guanghui Ren, Quanlu Zhang, Boxun Li, Maoqing Yao, Yu Wang
Collective Experience Learning (CEL) improves robot policy reliability by pooling fleet-generated experience without requiring additional expert interventions.
How can a fleet of robots autonomously improve their policies using shared experience without requiring constant human teleoperation or corrective feedback?
As robot fleets grow, relying on human teleoperators to provide corrective actions during deployment becomes a bottleneck that prevents policies from scaling to reliable execution. The authors introduce Collective Experience Learning (CEL), a paradigm where robots share their execution experience to improve a common policy. They realize this through Failure-guided Experience Retrieval fine-tuning (FERT), which uses failure contexts to retrieve and prioritize relevant successful trajectory segments for supervised policy updates. Across five real-world manipulation tasks, this approach achieves success rates of 93.3–99.2% on held-out robots, with larger fleets consistently providing more experience to drive higher reliability.
Paper Primer
FERT functions as a retrieval-based supervised learning loop: it stores successful trajectories in a buffer and uses failure-context queries to identify and prioritize successful segments that resemble the policy's current failure modes. This mechanism effectively turns failures into guidance for selecting which successful behaviors to imitate, without needing a learned critic or value function.
Collective experience learning significantly improves task success rates in multi-stage manipulation.
In a 96-robot deployment for phone packaging, the policy success rate increased from 79.2% to 98.3%.
Failure-guided retrieval is essential for efficient policy improvement.
In a 16-robot experiment, removing the failure-guided retrieval mechanism reduced the success rate from 86.7% to 64.2%.
Why is this approach preferred over simply collecting more successful demonstrations?
Collecting more successes alone does not determine which behaviors are most relevant to the policy's current difficulties. FERT uses failure contexts to prioritize specific successful segments, ensuring the policy learns from the behaviors most likely to resolve its current errors.
How does the system handle the inevitable interruptions and varying speeds of a large robot fleet?
The Fault-tolerant Asynchronous Training System (FATS) decouples execution from policy optimization. It uses a data plane (SwarmLink) to pool experience asynchronously and a control plane (HotSwarm) to restart failed robot workers in place without stopping the rest of the fleet.
This paper demonstrates that robot fleets can self-improve through collective experience, shifting the burden of reliability from human teleoperation to automated, failure-guided data retrieval.
Introduction: Scaling Robot Learning
We expose the human‑teleoperation bottleneck and propose Collective Experience Learning to let robots improve policies from their own deployments.
Scaling robot learning to large fleets is hampered by the need for human teleoperators to monitor rollouts and intervene when policies fail, a requirement that does not grow with the number of robots.
Improving a robot’s decision‑making function by adjusting its parameters based on experience, so it behaves more reliably in new situations.
**Figure 1** Sources of experience for robot policy learning: human demonstrations, expert corrections and collective experience generated within a robot fleet. In our realization of collective experience learning (CEL), successful robot executions supply action supervision for further policy updates after expert-supervised initialization. FERT realizes this paradigm by using failure contexts to retrieve relevant successful experience. Episode outcomes can be labeled by human annotators or a learned reward model; our experiments use human-provided outcome labels and environment resets. The requirement for expert action supervision decreases from left to right.
Human teleoperation is the primary bottleneck for scaling robot learning.
Robot Policy Foundations
Defines the core learning mechanism that updates robot policies from fleet experience.
Scaling robot fleets hits a bottleneck: each new robot needs a human operator to correct failures, which is costly and slow.
Instead of waiting for a human to demonstrate the correct action, the robot tries, observes a binary success/failure signal, and nudges the policy toward actions that led to success — like a child learning to ride a bike by noting whether they reached the destination, without a parent showing the exact pedal motion.
Sample N=3 trajectories from the current policy $\pi$_$\theta$; outcomes are y₁=1, y₂=0, y₃=1.
Compute the empirical gradient estimate ∇_$\theta$ ≈ (1/3)·[∇_$\theta$ log $\pi$_$\theta$($\tau$₁)·y₁ + ∇_$\theta$ log $\pi$_$\theta$($\tau$₂)·y₂ + ∇_$\theta$ log $\pi$_$\theta$($\tau$₃)·y₃].
Only the successful trajectories ($\tau$₁ and $\tau$₃) contribute non‑zero terms because y₂=0 zeroes out its gradient.
Update $\theta$←$\theta$ + $\alpha$·∇_$\theta$ with step size $\alpha$=0.1, moving the policy toward actions seen in $\tau$₁ and $\tau$₃.
This procedure improves the policy using only the success/failure label; it never needs the exact expert action for each timestep.
How does this differ from standard reinforcement learning that also uses reward signals?
Standard RL typically learns a value function or uses multi‑step returns, whereas here the reward is a single binary success label and the update directly maximizes its expectation with a simple mean‑gradient over successful episodes, avoiding the need for a separate critic.
Collective Experience Learning Paradigm
Introducing the core CEL mechanism and its Failure‑guided Experience Retrieval Fine‑tuning algorithm.
Human teleoperators limit the scalability of large robot fleets because each correction requires a person to watch and intervene. The bottleneck grows as more robots are deployed, yet the policy must improve continuously. The method below eliminates the need for additional expert demonstrations by letting robots learn from their own successes and failures.
Robots continuously upload their rollout data to a shared buffer; the central policy is fine‑tuned on that buffer and the updated policy is pushed back to every robot, forming a closed learning loop.
How does CEL differ from naïvely sharing raw trajectories among robots?
In CEL the shared buffer is filtered to keep only successful trajectories and the policy is updated centrally; raw trajectories (including failures) are never used as supervision, so the fleet does not propagate bad behavior.
When a robot fails, the short window of observations before the failure is used as a query to pull similar successful snippets from the collective buffer; those snippets are then up‑sampled during fine‑tuning, steering learning toward behaviours that resolve the current difficulty.
Why does FERT retrieve from successful trajectories instead of directly correcting the failed actions?
Directly cloning failed actions would reinforce the error. By using the failure context only as a query, FERT extracts correct behaviour from other robots that faced a similar situation, guaranteeing that the supervision signal is always successful.
Asynchronously collect trajectories; label each with a binary outcome and store successful ones in the collective buffer 𝔇⁺.
Encode each frame into a normalized visual‑state vector $z_t$ using a frozen visual encoder and joint‑state normalization.
For each failed trajectory, build a short pre‑failure query $Q_{\tau}$, align it to buffers with slope‑constrained DTW, and select the $K$ lowest‑cost intervals.
Expand the selected intervals, update their sampling priorities $w_i$, and sample chunks proportionally to $w_i$ for likelihood‑based fine‑tuning.
From a failed rollout we extract the last $L=5$ frames before the failure offset $\delta=1$, forming a query $Q$ of length 5.
We compute the distance matrix $d_{i,j}$ between $Q$ (5 frames) and each buffer trajectory (20 frames).
Dynamic programming finds the minimal‑cost alignment for each buffer; assume costs $c_1=0.8$ and $c_2=1.2$.
We select the $K=2$ intervals with lowest cost (both trajectories) and expand each by one frame before and after, yielding intervals of length $5+2=7$.
Priorities $w_i$ for the selected chunks are increased by the overlap fraction (here $7/20$), while all other chunks decay by factor $\rho=0.9$.
Even with a tiny query window, the DTW alignment can locate a relevant sub‑segment in a much longer successful trajectory, and the priority update focuses future training on exactly those overlapping frames.
**Figure 2. Failure-guided experience retrieval fine-tuning.** Robots asynchronously collect autonomous experience with binary task-outcome labels. Observations are encoded using visual and proprioceptive features. Failure contexts retrieve related segments from successful trajectories, and these segments receive increased sampling priority during policy fine-tuning. Updated policies return to the fleet to generate further experience.
Fault-Tolerant Asynchronous Training
Describes the distributed system that keeps robots learning despite faults.
Learning from deployed experience requires a loop that links robot execution, experience collection, and policy improvement while tolerating individual robot interruptions. Two properties drive the system design: (1) interaction and experience have distinct timing constraints, and (2) robots suffer availability failures that differ from compute‑resource failures.
FATS decouples real‑time action inference from offline policy training, letting robots keep moving even when parts of the system fail.
How does FATS differ from a naïve asynchronous training loop that simply retries failed workers?
In a naïve loop the failed worker’s identity is lost, so stale replies can be mistakenly accepted and data‑plane artifacts may be replayed incorrectly. FATS attaches a persistent station identity to every artifact and checks it on every use, guaranteeing that only the current worker’s messages are processed.
t=0 s: R₁ and R₂ send observations; I₁ batches them and returns two action chunks at t=0.5 s.
t=1 s: R₃ sends observation O₃, but the network drops; SwarmLink records O₃ in R₃’s reply channel.
t=1.5 s: I₂ receives O₄ from R₄, forms a batch with the stale O₃ entry, but HotSwarm rejects the batch because O₃’s worker identity no longer matches the current station.
HotSwarm retires the failed environment worker for R₃, launches a new worker on the same station, and rebuilds its channels (epoch increments).
t=2 s: R₃’s new worker sends a fresh observation O₃′; I₁ batches O₃′ with a new observation from R₁ and returns actions, allowing R₃ to resume without delaying other robots.
By binding every artifact to a station identity and checking it on delivery, FATS prevents stale actions from contaminating training while keeping the rest of the fleet fully operational.
Retire the failed worker and mark its record as dead in the registry.
Launch a new worker on the same station (same node and robot).
Rebuild all connection groups the failed worker participated in, advancing their epoch numbers.
Replay any in‑flight calls that were interrupted, discarding results that carry the old identity.
Resume collection once the robot has been reset and the new worker is registered.
**Table 1.** Interaction and experience paths in FATS. Sizes describe model-input images and a complete episode record, respectively.
**Figure 3.** Fault-tolerant asynchronous training system (FATS). **a**, The SwarmLink data plane connects robot workers at the edge to shared cloud inference and assembles streamed trajectory fields for training. The HotSwarm control plane manages membership and worker identity, and recovers failed workers. **b**, Synchronous collection makes each policy update wait for every robot, which leaves faster robots idle. In asynchronous collection, robots continue independently. **c**, HotSwarm restarts the worker of a failed robot in place while the other robots continue, and the action computed for the failed worker (#1) is discarded. **d**, Inference data and execution records are joined in the cloud, and an episode is released to training only after its required fields and outcome arrive. **e**, Robot time available for collection after fault handling, measured for FATS in the 96-robot run and modeled for restarting the entire system after each fault event with restart times between 2 and 10 min (Appendix D.4). Durations in **b** and **c** are illustrative.
Experimental Evaluation
FERT lifts task success to 97.3% across five robot tasks, scaling with fleet size.
Collective Experience Learning lets a fleet of robots improve its policy by learning from its own autonomous successes and failures, without needing additional human corrections.
FERT raises the mean success rate across the five tasks to 97.3%, a 16.3‑point gain over multi‑robot HG‑DAgger and cuts the average failure rate by 86.0%.
Figure 5 and Table 3.
When the training fleet grows from 16 to 96 robots on Phone Packaging, FERT success climbs from 86.7% to 98.3%, an 11.6‑point improvement.
Figure 6 (Phone Packaging panel).
Disabling failure‑guided retrieval drops success to 64.2%, a 22.5‑point loss versus the full FERT configuration (86.7%).
Figure 7 (a) – “w/o FR”.
Using a sync interval of 100 optimizer steps reduces success by 11.7 pp compared with the optimal 500‑step interval.
Figure 7 (b) – 100‑step bar.
Extending the sync interval to 1,500 steps reduces success by 6.7 pp relative to the 500‑step optimum.
Figure 7 (b) – 1,500‑step bar.
Removing segment extension (`h_pre` = `h_post` = 0) cuts success to 57.5%, a 29.2‑point drop from the full 86.7% configuration.
Figure 7 (c) – “0 frames”.
Setting the priority‑retention factor to $\rho$ = 0.8 lowers success to 70.8%, a 15.9‑point loss versus the default 0.88 (86.7%).
Figure 7 (d) – $\rho$ = 0.8 bar.
After a median fault incident, the fleet maintains 99% of its episode throughput, never falling below 88%.
Section 4.5.1 fault‑containment analysis.
Independent collection yields 33% of the episodes that would be gathered under fully synchronous rounds.
Section 4.5.2 synchronous‑round replay.
A full system restart with a 10‑minute pause retains 74% of episodes compared with uninterrupted collection.
Section 4.5.2 restart‑duration replay.
A full system restart with a 5‑minute pause retains 86% of episodes.
**Figure 5.** Main results across five tasks. Each subfigure compares the success rates of human teleoperation, SFT, the selected multi-robot HG-DAgger (labelled (M)HG-DAgger), RLT, RECAP, and FERT on one task. For FERT, we report the checkpoint trained with the largest robot fleet. Error bars indicate 95% Wilson confidence intervals.
**Figure 6.** Scaling with fleet size across five tasks. Each panel shows success rate as a function of the number of training robots. The blue and pink curves represent HG-DAgger and FERT, respectively. HG-DAgger starts from SFT, whereas FERT starts from the selected (M)HG-DAgger checkpoint; horizontal dashed lines in different colours indicate the success rates of these initial policies. Error bars indicate 95% Wilson confidence intervals.
**Figure 7.** Ablation studies of FERT on Phone Packaging with 16 training robots. Pink bars indicate the full FERT configuration, while gray bars indicate the corresponding ablated variants. In (a), FR denotes failure-guided retrieval. In (b), the interval is measured in optimizer steps. In (c), each value specifies both $h_{pre}$ and $h_{post}$, with $L = 90$ frames held fixed. In (d), $\rho$ is the priority retention factor; larger values retain previous priorities for longer. Error bars indicate 95% Wilson confidence intervals.
**Figure c.** Replaying the measured traces under other designs
**Figure d.** Collection varies across robots
**Table 3.** Failure rate table. (M)HG-DAgger reports the performance of the checkpoints used to initialize FERT. FERT reports results obtained with the maximum number of training robots for each task.
**Table 1.** Comparison of success rates across different methods and robot counts.
Related Work
We situate our approach among prior robot learning, fleet-scale, retrieval, and distributed system works.
Related work spans four themes: (1) robot learning from real‑world interaction, (2) fleet‑scale policy sharing, (3) experience retrieval and prioritization, and (4) distributed infrastructures that enable large‑scale learning.
Human‑gated DAgger lets an operator intervene only when they deem it necessary, collecting corrective actions just for those moments.
Section 5.1 surveys methods that blend expert supervision with autonomous data. DAgger formalizes iterative dataset aggregation; HG‑DAgger refines it with human‑gated interventions. DART injects noise into demonstrations to expose the learner to off‑policy states, while ThriftyDAgger caps the intervention budget using novelty and risk estimates. HIL‑SERL, ConRFT, and RECAP combine demonstrations, interventions, and autonomous rollouts for manipulation or vision‑language tasks. Offline‑to‑online approaches such as AWAC use advantage‑weighted updates to merge prior data with online interaction.
Section 5.2 highlights fleet‑scale learning where many robots share experience. Ghasemipour et al. predict steps‑to‑go to construct rewards and success detectors, enabling largely autonomous practice. SOP and LWD connect fleet execution to online learning via asynchronous systems that gather execution traces and human signals, then broadcast updated policies back to the robots.
Section 5.3 reviews retrieval‑based and prioritized‑learning techniques. Behavior Retrieval, Graph Search and Retrieval, and STRAP query large offline datasets for relevant trajectories using task‑specific or graph‑based cues. Prioritized experience replay and self‑imitation learning allocate updates based on TD‑error or return superiority. AWR and AWAC weight replayed actions by estimated advantage, focusing learning on high‑value samples. FERT differs by using failure contexts as queries and boosting sampling of successful segments without a learned value estimator.
Section 5.4 surveys distributed systems that decouple data collection from policy optimization. Ape‑X, IMPALA, and SEED RL aggregate experience from many actors into centralized learners, using off‑policy correction or prioritized replay. RLlib offers composable abstractions for such pipelines. In robotics, Gu et al. and Yahya et al. demonstrated asynchronous off‑policy learning across multiple robots, QT‑Opt combined real‑robot data with distributed replay, and RLinf‑USER built on Ray for heterogeneous robots with crash‑resilient storage. The Fault‑tolerant Asynchronous Training System (FATS) builds on these ideas to keep CEL operational despite robot interruptions.
**Figure 9.** Measured communication-operation durations across observation, action and experience channels. Bars report the fraction and count of operations in each duration interval, and each panel shows the total number of events.
Theoretical Analysis of FERT
Theoretical guarantees show FERT improves over uniform replay via positive alignment terms.
This appendix formalizes how the Failure‑guided Experience Retrieval Fine‑tuning (FERT) update improves expected return compared with a uniform replay baseline.
Lemma 1 shows that the scalar $\mu = v^{\top}G_{\text{unif}}$ is strictly positive, guaranteeing that a uniform replay of successful experience provides a beneficial gradient direction.
Lemma 2 proves that when retrieval weights $\omega_{k}$ prioritize stages with higher local failure rates $\phi_{k}$, the covariance term $\kappa$ is also strictly positive, adding a further positive first‑order contribution.
Theorem 1 combines these results: for any step size $0<\eta$ satisfying the explicit bounds, the FERT update yields $J(\theta_{\text{FERT}}) > J(\theta_{\text{unif}}) > J(\theta_{0})$, i.e., a strictly higher expected return than uniform replay.
Policy Initialization Details
Details of expert‑supervised initialization before the CEL learning loop.
We first adapt the base policy $π_0$ by supervised fine‑tuning (SFT) on a fixed set of human demonstrations $D_{\text{demo}}$, then aggregate human‑gated interventions across the robot fleet using multi‑robot HG‑DAgger.
In SFT each training sample consists of an observation $o_t$, a task instruction $g$, and an action chunk $A_t$ of horizon $H$ and dimension $d$; the policy predicts a velocity field $v_θ$ conditioned on $o_t$ and $g$.
The conditional flow‑matching objective perturbs each action chunk with Gaussian noise $\epsilon_{\text{noise}}$ and a flow time $\alpha$, forming an interpolated chunk $A_{\alpha}^t = (1-\alpha)A_t + \alpha\,\epsilon_{\text{noise}}$; the loss is the mean‑squared error between the predicted velocity and the target $\epsilon_{\text{noise}} - A_t$.
After SFT we run multi‑robot HG‑DAgger, where each robot executes the current policy while a human operator can take over; every intervention is recorded with a binary indicator $m_t$.
Only contiguous chunks of length $H$ where every timestep is marked as an intervention (eligible chunk $\chi_t = \prod_{t'=t}^{\min(t+H,T)-1} m_{t'}$) are used; we discard the first $K_{\text{int}}$ frames of each intervention segment to avoid transient teleoperation artifacts.
Training on eligible chunks uses the same flow‑matching loss, but we apply a batch‑level loss filter: samples with loss above the $q$‑quantile or a fixed floor $\ell_{\min}$ are dropped, yielding a retained set $V_B$ that drives the gradient update.
During online rollout the HG‑DAgger workers run on the Fault‑tolerant Asynchronous Training System (FATS); rollout workers receive the latest policy parameters after every $K_{\text{sync}}$ optimizer steps, and environment workers suppress policy inference while the human is in control.
**Figure 10** Multi-robot HG-DAgger with human interventions. Human operators take over execution when correction is needed. Successful trajectories are stored with intervention indicators, and eligible intervention chunks are used for supervised policy updates.
Deployment Infrastructure
Details of robot fleet resources, network layout, worker placement, and recovery cost modeling.
The deployment hosts up to 100 robots, with the largest evaluated fleet comprising 96 units. Robot‑side computation resides at a single collection site, while policy inference and optimization run on a cloud cluster of six servers, each equipped with eight NVIDIA A800 GPUs (48 GPUs total). Shared storage is provided via a network‑attached NAS at the site and a GPFS system in the cloud.
The site‑to‑cloud link is a dedicated 30 Gbps connection. Inside the site, two tier‑1 switches interconnect three tier‑2 switches via dual 100 Gbps fiber links, forming two hierarchical layers. The cloud servers reside in the provider’s data center, and the link terminates at tier‑1 switch 2.
Each node advertises its hardware resources; accelerators are discovered through their runtimes. Robots register with explicit communication endpoints and camera IDs, optionally undergoing connectivity checks. Resources are grouped by name, and a node may belong to multiple groups, enabling flexible placement of environment, inference, and training workers.
The recovery model quantifies the fraction of robot time available for experience collection under two strategies. If the whole system restarts after each fault, the availability is $A_{\text{sys}} = \frac{1}{1 + N\,\lambda\,t_{\text{restart}}}$ (Eq. 50). If only the affected robot restarts, availability is $A_{\text{robot}} = 1 - \frac{\sum_i \Delta t_i}{N\,T_{\text{run}}}$ (Eq. 51).
From the 96‑robot run we estimate a per‑robot fault rate $\lambda = 0.0288$ events / robot‑hour, yielding $T_{\text{run}} = 6.88$ h after removing a one‑hour lunch break. The fleet lost 8.3 robot‑hours, giving $A_{\text{robot}} = 98.7\%$. Plugging $t_{\text{restart}} = 2, 5, 10$ min into Eq. (50) yields system‑wide availabilities of 91.6 %, 81.3 % and 68.5 % respectively for 96 robots, and 50.4 %, 28.9 % and 16.9 % for a hypothetical 1,024‑robot fleet.
The model assumes fault rates and downtime distributions are independent of fleet size and that faults on different robots are independent aside from the defined grouping. It ignores repair time of the failed robot and discards in‑progress episodes on other robots during a whole‑system restart, focusing solely on recorded robot‑local incidents.
**Figure 11.** Robot-site network and cloud resources for FATS. The site supports 100 robots, with 96 used in the largest training run. Six cloud servers provide 48 NVIDIA A800 GPUs for inference and policy optimization. The robot site and cloud communicate through a dedicated 30 Gbps link.
Task Setup and Evaluation
Defines observation preprocessing, action format, and task‑specific success criteria for evaluating CEL and baselines.
The policy receives an observation $o_t$ and a task instruction $g$. $o_t$ contains three RGB images—one head‑mounted ($400 \times 640$) and two wrist‑mounted ($1056 \times 1280$)—plus a 16‑dimensional proprioceptive vector (seven joint values per arm and two gripper measurements).