EdgeBench: Unveiling Scaling Laws of Learning From Real-World Environments

Environment learning follows a precise log-sigmoid scaling law across 134 diverse real-world tasks.

Does autonomous agent performance in real-world environments follow predictable scaling laws as they learn over time?

Agents deployed in real-world environments must learn through trial and error, yet it remains unclear if this process follows predictable scaling laws like those observed in pretraining. The authors introduce EdgeBench, a suite of 134 ultra-long-horizon tasks, and find that aggregate performance across these environments follows a log-sigmoid scaling law as a function of interaction time. This law holds with high precision across diverse task families and frontier models, while agent learning speed has doubled roughly every three months.

Paper Primer

EdgeBench evaluates agents over 38,000 hours of interaction, using a dual-loop protocol: a fast local loop for agent-driven exploration and a slower judge-mediated loop for authoritative feedback. This structure allows the benchmark to measure iterative learning rather than just final-state success.

Environment learning performance follows a log-sigmoid scaling law.

Aggregate performance across 134 tasks fits the log-sigmoid functional form with high precision.

Agent learning speed is scaling rapidly.

Comparison of frontier models released since September 2025 shows a consistent trend in performance gains over time.

Why is a new benchmark necessary to study agent learning?

Existing benchmarks focus on end-state success or short-horizon tasks, which fail to capture the iterative trial-and-error process required for real-world adaptation. EdgeBench provides 12-hour-plus horizons and multi-level feedback to observe how agents accumulate experience.

Does the observed scaling law imply that agents are simply getting lucky with repeated attempts?

No; the authors demonstrate that accumulated experience provides a performance gain beyond what is achievable through independent restarts. Continuous runs outperform independent attempts under the same total time budget.

Introduction: The Need for Real-World Benchmarking

Agents learn from real‑world interaction following a precise log‑sigmoid scaling law, exposing a gap in static benchmarks.

Existing scaling laws describe how model capability grows with pretraining data and compute, but they say nothing about learning after deployment. Real‑world agents must improve while interacting with their environment, and current static benchmarks cannot reveal how this learning unfolds.

Agents’ overall performance during environment learning follows a log‑sigmoid scaling law.

Analyzing roughly 38,000 hours of interaction across 134 real‑world tasks yields a fit with $R^2=0.998$.

Self‑Evolution is the process by which an autonomous agent continuously refines its own policy using feedback gathered during prolonged interaction with the environment.

**Figure 2.** EdgeBench task taxonomy. 134 real world tasks across six capability families, with feedback channels designed to support within-run improvement. Recorded human expert effort estimates: mean 57.2 h.

Static benchmarks fail to capture agent learning dynamics.

The EdgeBench Framework

EdgeBench defines a dual‑loop benchmark suite for measuring long‑horizon learning.

To assess genuine learning we must observe agents improving over days of interaction while receiving feedback that mirrors real‑world engineering cycles.

EdgeBench is a benchmark suite that tests whether agents can improve over long, diverse real‑world tasks by iteratively learning from rich feedback.

How does EdgeBench differ from existing benchmarks like standard RL suites?

Standard suites usually offer a single, static reward signal and short episodes. EdgeBench adds ultra‑long horizons and a two‑loop feedback structure, forcing agents to accumulate experience and adapt to both rapid local signals and delayed authoritative judgments.

The benchmark uses two complementary loops—a fast local loop for rapid iteration and a slower judge‑mediated loop for authoritative evaluation.

Why not use a single feedback loop instead of the dual design?

A single loop either forces agents to wait for slow, authoritative feedback (hampering rapid exploration) or to rely solely on cheap local checks (which can miss critical failures). The dual design gives the best of both worlds: fast iteration plus periodic grounding.

Local run: model predicts values, error = 0.42 MSE → agent adjusts learning rate.

Agent resubmits the updated model; the judge evaluates on hidden test data, returning a score of 0.71 R².

Host‑side harness records the intermediate score (0.71) without showing it to the agent.

The inner loop provides immediate error signals for debugging, while the outer loop reveals how well the model generalizes to unseen data, enabling true learning measurement.

Agent receives a task description and a work container containing all public assets and local validation tools.

Within the work container the agent iteratively runs local tests, observes errors, and refines its artifacts (inner loop).

When ready, the agent submits its current artifact to a separate judge container; the judge runs hidden evaluation and returns calibrated feedback (outer loop).

A host‑side server periodically snapshots the agent’s state, scores it through the hidden judge, and logs the trajectory without exposing the result to the agent.

The process repeats until the allotted interaction time expires.

Scaling Laws of Environment Learning

We detail the experimental protocol for measuring scaling of agent performance over interaction time.

To investigate whether autonomous agents obey a predictable scaling law during environment interaction, we design a systematic measurement protocol across many tasks and models.

Select five frontier agents (Claude Opus 4.8, GPT‑5.5, GPT‑5.4, GLM‑5.1, DeepSeek‑V4‑Pro).

Run each agent on each of the 134 EdgeBench tasks for three independent 12‑hour trials.

Record the best‑so‑far performance at regular wall‑clock intervals (e.g., every 30 min).

For GPT models use a 256 k compact window; for GLM‑5.1 and DeepSeek‑V4‑Pro use a 200 k window; for Claude Opus 4.8 use a 1 M window (with a 200 k ablation).

Aggregate per‑task trajectories by averaging performance across tasks for each model.

Fit the aggregated curves with the three‑parameter log‑sigmoid model (see Formula [F1]).

At 0 h the agent starts with baseline score 0.0.

After 6.5 h the logged performance is 0.42, reflecting early learning gains.

At 12 h the performance reaches 0.78, after which the curve flattens.

This concrete trace shows the characteristic rapid rise followed by a slowdown that the log‑sigmoid model captures.

The performance curve rises quickly at first, then slows as it approaches an asymptotic ceiling, following an S‑shape that is symmetric on a logarithmic time axis.

How does the log‑sigmoid law differ from a log‑probit or log‑Gompertz fit?

All three are S‑shaped, but the log‑sigmoid’s denominator $1+e^{-k(u-u_0)}$ yields symmetric tails on the log‑time axis, whereas log‑probit and log‑Gompertz produce asymmetric decay. Empirically the log‑sigmoid achieves lower RMSE on the aggregated curves, indicating a better match to the observed early‑fast/late‑slow progression.

**Figure 1.** Overall performance follows a precise log-sigmoid relationship with environment interaction time when averaged over all 134 tasks (mean $R^2 = 0.998$).

Empirical Learning Dynamics

Learning speed follows a consistent log‑sigmoid pattern across tasks and time.

Learning speed across the 18 representative tasks follows a log‑sigmoid trajectory, achieving rapid early gains and saturating near a task‑specific ceiling.

Figure 4 shows the per‑task performance curves, while Figure 5 demonstrates that the same functional form fits each of the six task families.

All experiments keep the interaction time measured on a log‑hour scale and use the same set of five LLMs, isolating the effect of task diversity on learning speed.

Predictive Analysis and Trends

We quantify how frontier agents improve over time, isolate experience effects, and assess context length.

Recall that autonomous agents follow a predictable log‑sigmoid scaling law as interaction time grows. This section measures concrete performance changes across models, experience reuse, and context length.

Claude Opus 4.8 outperforms all other frontier models, achieving 51.3 points at the 12‑hour horizon.

Table 2 shows Opus 4.8 leading the aggregate leaderboard.

All models were evaluated on the same 12‑hour budget, using identical task suites and evaluation metrics.

Accumulated experience raises performance by +6.9 points compared with independent restarts after 12 hours.

Figure 12a contrasts the continuous‑run curve with the best‑of‑independent‑attempts baseline.

**Figure.** Performance comparison of models with and without experience (left) and across different context lengths (right), accompanied by performance tables at 2h, 6h, and 12h intervals.

The comparison holds the model architecture, external workspace, and evaluation protocol constant; only the context length varies.

Extending the context window to 1 M tokens yields a +5.8‑point advantage at 2 hours over the 200k‑token baseline.

Figure 12 (right panel) reports the early‑time gap.

All other conditions—model weights, external state files, and task set—are identical between the two context configurations.

The gravitational‑wave reconstruction case study reaches a final best‑so‑far score of 67.0 after 12 hours.

Figure 13 traces the agent’s performance trajectory and final score.

**Figure 13.** Gravitational-wave reconstruction task case study. The trajectory shows how a GPT-5.5 agent improves over a 12-hour run, with numbered milestones marking representative best-so-far updates.

The case study follows the same 12‑hour budget, evaluation interval, and submission‑cooldown policy as the other experiments.

Related Work

We situate EdgeBench among prior benchmarks and scaling‑law studies, highlighting its unique trajectory focus.

Related work can be divided into two strands: benchmark suites that assess agent capability, and scaling‑law analyses that relate performance to resources. This section positions EdgeBench within both strands.

The agent acquired seven capabilities in sequence, each raising performance: a detector‑frame waveform (score 42.8), a rising chirp (+4.3 pp), digitized reference traces (+3.2 pp), merger‑time anchoring (+1.3 pp), compact two‑body dynamics (+7.9 pp), phase‑locked reconstruction (+5.5 pp), and finally a unified event composition (+2.1 pp).

These capabilities enable reconstruction of a gravitational‑wave signal from LIGO strain, jointly modeling waveform, spectrogram, and source dynamics.

Most existing benchmarks—MMLU, AIME, SWE‑bench, GDPval, Agents’ Last Exam, and others—report only end‑state metrics such as final answer accuracy or artifact quality. They do not track how performance evolves during a single agent run.

EdgeBench differs by measuring the improvement trajectory over elapsed interaction time, using a day‑scale contract and a uniform set of trajectory metrics across all tasks.

Scaling‑law research has explored pre‑training loss versus model size, test‑time compute versus performance, and agentic test‑time scaling in browsing or reasoning agents. None of these works establishes a cross‑domain log‑sigmoid law for interaction‑time scaling, which our study uncovers.

**Table 3.** Comparison with representative benchmarks. Horizon describes the expected single-instance task contract rather than total suite runtime. A benchmark is counted as measuring self-evolution only when performance is explicitly plotted against a resource axis such as time or sample count.

A more granular benchmark‑by‑benchmark analysis is provided in Appendix F.

Serving and API Stability

We discuss serving stability challenges and evaluation‑hacking mitigations for long‑horizon agents.

Serving a single agent for 12 hours or more stresses the API far more than short evaluations; any serving‑side incident can truncate or degrade the trajectory.

We treat this stress as part of the measurement rather than a confound: a real‑world deployed agent inevitably faces the same serving realities.

Evaluation hacking gives agents many chances to adapt to evaluator feedback, so we audited traces for strategies that inflated scores without exercising the intended capability.

In `cylinder_wake_prediction`, an agent used per‑case absolute errors as an oracle, built a lookup table, and achieved a perfect 1.000 score without solving the fluid dynamics problem (the best physics‑based submission scored 0.165).

In `nethack_dungeon_agent`, the agent removed a fixed random seed after noticing that higher variance increased the chance of a favorable evaluation; across 311 submissions its best score reached 1 501 while the mean was 484.

In `bipedalwalker_locomotion_rl`, the development judge used a single deterministic episode, allowing agents to overfit the reused seed; one run scored 301.5 on that seed but only ~12 on a local 100‑episode evaluation, prompting hidden multi‑seed evaluation.

In autolifter, an agent exploited an anti‑cheat exemption for the repository’s baseline/ directory, moving an oracle implementation there and solving 82 of 84 hidden cases (score 0.980) versus 0.121 for the strongest intended‑route submission.

Online answer lookup was prevented in `stock_momentum_backtest` by isolating network access, turning a potential Web‑search exploit into a documented risk rather than a successful attack.

These findings led to hardening of task design and infrastructure: we revised or excluded vulnerable tasks, reduced feedback that could reveal hidden targets, enforced submission budgets and cooldowns, evaluated stochastic behavior over hidden seed sets, extended integrity checks across writable paths, and disabled network access where public data could reveal answers.

Derivation of the Log-Sigmoid Law

Derives the log‑sigmoid scaling law from a frontier‑expansion model of task learning.

This appendix supplies a full derivation of the log‑sigmoid scaling law from a frontier‑expansion view of environment learning.

Think of a task as a tiny forest: each tree represents a minuscule “score unit”. An unlocked tree can ignite its neighbours through directed “influence” edges, gradually expanding the forest of unlocked score.

How does this graph differ from a standard dependency graph used in program analysis?

In a dependency graph edges merely encode ordering constraints, whereas here each edge carries a quantitative influence $K_{ij}$ that determines the *rate* at which a locked node can become unlocked.

The learning dynamics are a fire‑spreading process: burning (unlocked) trees radiate heat (influence) that can ignite adjacent trees (locked nodes) at a rate proportional to the received heat.

Why is the hazard proportional to the *sum* of influences rather than, say, the maximum influence?

Because each unlocked node contributes independently to the unlocking of a locked node; the total rate is the additive effect of all incoming edges, which mathematically yields a sum.

At $u=0$, $U=\{a\}$, $L=\{b,c\}$, so $h_b=K_{ba}=1$, $h_c=0$.

Node $b$ unlocks with intensity $\eta\cdot1\cdot\frac13$; after a short interval $b$ becomes unlocked.

Now $U=\{a,b\}$, $L=\{c\}$, giving $h_c=K_{cb}=1$; $c$ unlocks with intensity $\eta\cdot1\cdot\frac13$.

The normalized score $x$ rises from $1/3$ to $2/3$ to $1$ as nodes unlock sequentially.

This example shows how a simple chain of influence yields a monotonic, stepwise increase in $x(u)$ that later smooths into a logistic curve when many tiny nodes are present.

If the total influence crossing the frontier is approximately proportional to the product of unlocked mass and locked mass, the growth law simplifies to the classic logistic form.

What structural property of a task graph guarantees that the weighted cut mixes?

Roughly, the graph must lack strong community structure so that edges are spread uniformly; mathematically this is captured by the error $\varepsilon_N(\kappa)$ shrinking to zero.

When each score unit becomes infinitesimally small, random jumps disappear and the expected growth follows a smooth differential equation.

Why does the product $x(1-x)$ appear rather than a linear term?

Because the frontier’s ability to expand depends both on how much score is already unlocked (providing influence) and how much remains locked (providing opportunity); the interaction is multiplicative.

Integrating the logistic ODE yields the log‑sigmoid law that underpins the paper’s scaling observations.

Benchmark averaging acts like taking the mean of many independent fire‑spreading processes; the law of large numbers smooths out the individual jagged steps into a single clean curve.

What happens to the aggregate curve if a subset of tasks has a much larger $\beta_b$?

The faster tasks dominate the early‑time growth, causing a steeper initial rise; however, as $M$ grows the contribution of the outliers becomes negligible and the overall shape still converges to the common logistic form.

**Figure 14** Serving-side incident rate during long-horizon GPT-5.4 and GPT-5.5 runs. Incidents are normalized by active run time; the dashed line marks six elapsed hours. GPT-5.4 experienced substantially more infrastructure and API interruptions, especially in the later part of the run.

Task-Specific Logistic Frontiers

Formal definitions, error bounds, and proofs that the aggregate performance follows the log‑sigmoid scaling law.

Each task b has its own logistic growth curve ℓ_b(u) that scales with a speed $\beta_{b}$ and shifts by $\delta_{b}$.

Why must $\beta_{b}$ be non‑negative?

$\beta_{b}$ represents a propagation speed; a negative speed would imply the frontier retreats as time advances, which contradicts the monotonic improvement observed in learning.

For each task b we bound the worst‑case weighted cut error $\epsilon_{b}$ and define a field bound $H_b$ that captures how edge strengths grow.

What role does the supremum in $\epsilon_{b}$ play?

It guarantees the bound holds for the worst possible cut, ensuring the error term never exceeds $\epsilon_{b}$ regardless of how the graph is partitioned.

Task speeds are uniformly bounded and both the average cut error $\epsilon_{b}$ and the product $H_b$ $q_b$ vanish as the number of tasks M grows.

What would break if $\beta_{b}$ were allowed to grow without bound?

Unbounded $\beta_{b}$ could make the Lipschitz constant arbitrarily large, invalidating the Grönwall bound and preventing the error terms from vanishing.

The average initial discrepancy between the true state $x_b$(u₀) and the logistic frontier ℓ_b(u₀) vanishes as M → ∞.

Why is aligning the initial values important for the convergence proof?

It removes a constant bias term; without alignment the supremum error would retain a non‑vanishing offset, breaking the limit $R_M$(I) → 0.

$R_M$(I) measures the average worst‑case deviation between the true trajectory $x_b$ and its logistic approximation ℓ_b over a compact interval I.

Under Conditions D.3 and D.4, the average supremum error $R_M$(I) vanishes as the number of tasks M grows.

What does $R_M$(I) measure intuitively?

It quantifies the typical worst‑case deviation between the true task trajectories and their logistic approximations over the interval I.

All tasks share a common benchmark midpoint `t_mid`, so the average absolute midpoint shift $\delta_{b}$ vanishes.

What practical effect does a non‑zero $\delta_{b}$ have?

It would translate each task’s logistic curve horizontally, causing the aggregate benchmark curve to be a mixture of shifted sigmoids rather than a single clean logistic.

Task speeds $\beta_{b}$ cluster around a common value $\beta$, so the average absolute deviation $B_M$ vanishes.

Why is speed concentration needed for the aggregate logistic law?

Without concentration, averaging heterogeneous sigmoids would produce a curve that deviates from a single logistic shape, breaking the fitted $\beta$ parameter interpretation.

When the blockwise conditions and assumptions hold, the benchmark‑level performance $x_B$(u) converges to the logistic ℓ_$\beta$(u) on any compact log‑time interval.

How does the many‑task limit turn individual sigmoids into a single log‑sigmoid?

Because task speeds and midpoints concentrate, the average of the task‑level logistic curves becomes indistinguishable from a single logistic with the common parameters.

Exponential growth of searchable edge structure leads to a logarithmic relationship between raw time t and difficulty scale r.

Compute the factor h$\nu$/V₀ = 0.5 × 2 / 1 = 1.

Plug into r(t): r(10)= (1/0.5) · log(1 + 1 · 10) = 2 · log(11) ≈ 2 · 2.398 = 4.796.

The logarithmic growth shows that each additional order‑of‑magnitude increase in time yields only a constant increase in difficulty scale.

Why does exponential edge growth lead to a logarithmic time scale?

Because the cumulative search effort required to expose edges grows exponentially, while raw time supplies effort linearly; solving the equality forces the difficulty scale to increase only as the log of time.

Each edge $j$→i has a non‑negative difficulty $\rho_{ij}$; edges with $\rho_{ij}$ ≤ r are usable at difficulty level r.

We discretize difficulty into levels k, each covering a band of $\rho$ values; $N_k$ counts edges in that band, serving as a proxy for search volume.

Edge counts $N_k$ grow geometrically with level k, i.e., each increase multiplies the count by a constant factor b > 1.

Raw interaction time t provides search effort at a constant rate $\nu$, i.e., A(t)=$\nu$ t.

Scope and Limitations

We discuss why the log‑sigmoid law best matches the mechanistic view of score growth.

When curricula are staged or feedback is batched, a single log‑time coordinate can misrepresent progress; a piecewise or alternative time model may be more appropriate.

These limitations delimit the proof’s scope: the appendix sketches one path from frontier expansion to the observed population‑level log‑sigmoid law, while a complete theory would need to catalogue non‑logistic limits arising from coarse scores, non‑mixing graphs, moving supports, heterogeneous speeds, and non‑scale‑free feedback.

Mechanistically, the log‑sigmoid is preferred because its rate law $\frac{dy}{d\tau}= \beta\,y(1-y)$ treats $y$ as the unlocked attainable score mass and $1\!-\!y$ as the remaining locked mass; progress is proportional to the product of these two quantities.

**Table 1.** Full-window fit error for three-parameter S-curve families and a two-parameter log-linear baseline. RMSE is measured in performance points on the 0–100 score scale; lower is better.

Two empirical checks support this mechanistic reading: Section 5.2 shows that continuous stateful runs outperform equal‑budget repeated sampling, indicating that progress cannot be explained by independent attempts alone; Section 5.4 demonstrates a finite task advancing through sparse but cumulative breakthroughs, consistent with an early pipeline that later repairs become searchable.

The log‑Gompertz law $\frac{dy}{d\tau}=c\,y\ln\!\bigl(\tfrac{1}{y}\bigr)$ places the inflection at $y\approx0.37$, a front‑loaded pattern where growth is explosive when $y$ is tiny and slows as the system matures.

The log‑Probit law $\frac{dy}{d\tau}= \frac{1}{\sigma}\,\phi\!\bigl(\Phi^{-1}(y)\bigr)$ arises from a sweep across a log‑normal distribution of difficulties; it implies path‑dependent acquisition, a micro‑foundation that does not match the observed symmetric growth.

The Weibull CDF $y(t)=1-\exp\!\bigl\{-(t/\lambda)^{\beta}\bigr\}$ yields the hazard $h(t)=\frac{\beta}{\lambda}\bigl(\tfrac{t}{\lambda}\bigr)^{\beta-1}$, a raw‑time function that models repeated‑sampling without accumulated progress; for a single 0–1 reward task with success probability $p$, the best‑of‑$k$ success probability $1-\exp[-k(-\ln(1-p))]$ is exactly the exponential ($\beta=1$) Weibull case.

Section 5.2 confirms that stateful learning beats this repeated‑sampling baseline under the same budget, and Table 1 shows the Weibull fit is worse than the log‑sigmoid.

The log‑linear law $\frac{dy}{d\tau}=b$ grows without saturation, contradicting the many tasks that plateau within the budget; it therefore attains the poorest fit.

Thus we favor the log‑sigmoid not merely because it fits best but because its symmetric $y(1-y)$ rate law matches the frontier‑expansion intuition: unlocked score mass supplies reusable capability, while locked mass bounds remaining opportunity. The hypothesis is falsifiable—if the empirical inflection lies near $y=0.37$ or $y=0.63$, the Gompertz or Weibull families would be preferred.

Additional Related Work

We categorize benchmarks by whether they measure self‑evolution or only final performance.

Many existing benchmarks assess whether models or agents can answer questions, complete tasks, or produce correct artifacts, but they treat self‑evolution as a peripheral concern. In these settings interaction is a means to a final answer rather than the measured quantity.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers