From Human-Centric to Agentic Code Review: The Impact of Different Generations of Generative AI Technology on Review Quality

Suzhen Zhong, Shayan Noei, Bram Adams, Ying Zou

Agentic code review improves speed but risks quality, often narrowing reviewer perspectives through repeated AI-human pairings.

How does the shift from human-only to LLM-assisted and agentic code review impact review quality and efficiency?

Code review is a critical bottleneck in software development, yet the transition from human-centric to AI-supported review remains poorly understood in terms of actual quality and efficiency. The authors analyze 1.02 million pull requests across 207 projects, categorizing AI adoption into three distinct practices and modeling human-AI collaboration as sequences of reviewer interactions. While agentic reviews significantly accelerate decision-making, they do not consistently improve review quality and are frequently associated with "Review Buddies"—a pattern of repeated, narrow reviewer pairings.

Paper Primer

The paper investigates how the shift from human-only to LLM-assisted and agentic code review impacts software quality. It identifies three adoption trajectories: Gradual AI Adoption, Rapid LLM Adoption, and Rapid AI Agent Adoption.

The authors model review processes as Markov chains of reviewer interactions: they map the sequence of human, LLM, and agent participation to identify collaboration patterns. This approach treats the review process as a state-transition system: it captures who acts, in what order, and how those sequences correlate with review duration and anti-patterns.

Agentic collaboration patterns significantly improve review efficiency.

In the agent era, reviews initiated by AI agents or involving multiple agents are faster than human-only reviews under Gradual and Rapid AI Agent adoption practices. Review time drops by 2.5 to 4.5 days per thousand lines of code (KLOC) compared to pre-AI baselines.

Efficiency gains come at the cost of increased review-quality risks.

AI-involved collaboration patterns show higher prevalence of the "Review Buddies" smell, where repeated reliance on the same reviewers narrows perspectives. Review Buddies prevalence rises from 16% in human-only reviews to 53–60% in AI-involved patterns.

Why does the paper distinguish between "LLM-assisted" and "Agentic" review?

LLM reviewers primarily generate natural-language feedback, whereas AI agents autonomously retrieve project context, run development tools, and verify findings before providing feedback.

Does AI adoption make traditional review factors obsolete?

No. Traditional factors like author experience, commit counts, and discussion threads remain strong explanatory variables for review quality, suggesting that AI should augment rather than replace established review practices.

The Evolution of Code Review

We frame the empirical gap in understanding how AI‑driven code review reshapes efficiency and quality.

Code review has long been a human‑centric gatekeeper of software quality, yet the rise of generative AI is turning it into a hybrid human‑AI process.

We lack systematic evidence on whether this transition improves review efficiency or harms quality, and prior studies have only examined isolated aspects such as reviewer workload or “review smells”.

The software‑engineering workflow is shifting from a purely human review loop to an agentic one where AI reviewers actively participate, and we must quantify how that shift affects both speed and defect detection.

To address the gap we formulate three research questions (RQ1–RQ3), each probing a different facet of AI adoption and its consequences for review outcomes.

RQ1 asks which AI adoption practices emerge and how they relate to review quality; RQ2 examines how human‑AI collaboration patterns influence quality; RQ3 evaluates how these patterns interact with traditional factors such as pull‑request size and activity.

The shift from human‑centric to agentic review is a fundamental change in software engineering workflows.

Methodology and Data Collection

We detail the dataset, labeling, metrics, and factors used to evaluate agentic code review.

This section describes how we assembled the empirical case study that underpins all three research questions. We begin with a broad project filter, then collect review conversations, label reviewer types, define eras, and finally compute the Review Quality Metrics that feed the explanatory models.

**Fig. 1:** Overview of our approach.

Filter GitHub repositories created before May 2022, with ≥100 stars and ≥1 reviewed PR per month from May 2022–Feb 2026, yielding 2,490 candidates.

Collect all reviewed pull requests and review events via the GitHub REST API (timestamps, decisions, comments).

Label each reviewer account as human, rule‑based bot, traditional ML reviewer, LLM reviewer, or AI‑agent reviewer using API metadata and manual inspection of tool documentation.

Validate the AI‑agent labels on a random sample of 384 PRs (95 % confidence, 5 % margin); 360 show planning or context‑retrieval behavior.

Define per‑project eras: pre‑LLM (no generative AI), LLM (first LLM review to first AI‑agent review), and agent (from first AI‑agent review onward).

Retain only projects with >400 reviewed PRs in each era, resulting in 207 projects and 1.02 M PRs.

Classify each PR into one of 11 types (Chore, Fix, …) using GPT‑4.1‑mini on titles/descriptions; manual check on 384 PRs yields $\kappa$ = 0.91.

Compute Review Quality Metrics (efficiency, smells) for every PR; assemble the explanatory factor table for RQ3.

Two complementary lenses capture how well a code review works: speed (how fast a PR is decided) and health (whether the review exhibits known anti‑patterns).

Compute elapsed time: 3 days 6 hours = 3.25 days.

Normalize by KLOC: 3.25 days / 1 KLOC = 3.25 days/KLOC.

This concrete calculation shows how a seemingly long three‑day review becomes comparable to a five‑day review of a 2 KLOC change (5 / 2 = 2.5 days/KLOC), highlighting the importance of size‑normalization.

The explanatory factor set combines traditional review attributes (PR type, churn, reviewer count) with AI‑specific experience variables (LLMExp, AgentExp) to model how collaboration patterns drive efficiency and smells.

AI Adoption Practices

Key adoption patterns shift review quality and efficiency across eras.

Rapid LLM Adoption significantly raises review‑smell rates, while Gradual AI Adoption and Rapid AI‑Agent Adoption markedly improve review efficiency in the Agent era.

Table IV shows an 8.0 % smell increase for Rapid LLM Adoption (p < 0.05) and Wilcoxon tests reveal 2.5 days/KLOC and 4.5 days/KLOC efficiency gains for Gradual and Rapid AI‑Agent practices respectively.

All projects were clustered with soft‑DTW on resampled 10‑point adoption curves, and the same Wilcoxon signed‑rank test with Bonferroni correction was applied across practices.

Projects adopt AI reviewers in distinct temporal patterns—some switch quickly, others transition gradually—affecting how code is inspected.

In the Agent era, autonomous AI agents act as reviewers, making decisions without direct human prompting.

**Fig. 2:** Three identified AI reviewer adoption practices as code review shifts from human-centric to LLM-assisted and agentic review. Higher trend lines indicate greater AI reviewer participation.

AI adoption is not uniform; it follows distinct patterns (Gradual vs. Rapid) that define the review era.

Human-AI Collaboration Patterns

AI‑involved collaboration patterns raise Review Buddies prevalence and alter efficiency across adoption practices.

AI‑involved collaboration patterns exhibit a markedly higher prevalence of the Review Buddies smell.

Table V reports 60 % prevalence for LLM‑Assist and 53 % for agent‑involved patterns versus 16 % for human‑only reviews.

**Table III.** Independent explanatory variables used to model how reviewer collaboration patterns and alternative factors are jointly associated with review efficiency and review smells. The Abbrev. column shows the shortened metric names used in Table VI.

Factors Impacting Review Quality

Agentic collaboration patterns explain review quality better than traditional metrics.

Collaboration patterns show stronger associations with review quality than traditional project metrics.

Maximum impact score for a collaboration pattern reaches +214, whereas the strongest traditional‑factor impact is only +22.

**Table.** Comparison of software development metrics across different eras (PreLLM, LLM, Agent) and adoption patterns (Gradual AI, Rapid LLM, Rapid Agent).

**Table.** Collaboration patterns and their associated efficiency (Effi.), code smells (Smell), and pull request types (PR Type) across three adoption scenarios: Gradual AI Adoption, Rapid LLM Adoption, and Rapid Agent Adoption.

**Table.** Impact scores of various factors on review processes across different eras (Pre-LLM, LLM, Agent) and adoption speeds (Gradual AI, Rapid AI/LLM/Agent). The table categorizes factors into "Collaboration" and "Traditional" types, evaluating them against four outcomes: (1) Lack of Efficiency (Review Delay), (2) Review Buddies, (3) Sleeping Review, and (4) Large Changeset.

Collaboration patterns are often more predictive of review quality than traditional project metrics.

Implications for Engineering Practice

Implications, validity limits, related work, and concluding takeaways for AI‑augmented code review.

The paper shows that moving from human‑centric to agentic code review reshapes how software quality is maintained, and it evaluates whether AI‑driven collaboration improves review efficiency and reduces code smells.

Practitioners should adopt AI reviewers selectively, tailoring the configuration to each pull‑request’s context, because Gradual AI Adoption and Rapid AI Agent Adoption speed reviews while Rapid LLM Adoption harms review quality.

Tool builders can expose signals such as pull‑request type, changeset size, discussion activity, and author or reviewer history to help decide whether a request stays human‑led, receives lightweight LLM summarization, or warrants full AI agent inspection.

AI collaboration does not uniformly improve review quality; some agent‑era patterns reduce “Sleeping Reviews” but increase “Review Buddies” and larger changesets, so traditional factors like author experience remain essential.

Future research should probe why developers assign AI reviewers to particular pull requests, how they interpret AI feedback, and when they keep humans in the lead, using interviews or surveys to complement the observational findings.

Benchmarks that only measure generic issue‑finding miss important context‑dependent differences; incorporating signals like request type and changeset size will enable evaluation of AI reviewers across routine maintenance, bug fixing, refactoring, and large changes.

Construct‑validity threat: the “agent era” marks the first AI‑agent reviewer but does not exclude later LLM reviewers, so some “agent‑era” pull requests still contain LLM participation.

Internal‑validity threat: the logistic‑regression models used for RQ3 capture statistical associations, not causal effects, so changing a factor may not directly alter efficiency or smell outcomes.

External‑validity threat: AI reviewer capabilities evolve rapidly; patterns observed from May 2022 to February 2026 may shift as tools improve, limiting direct generalization to future systems.

Prior work has documented code‑review practices, their impact on efficiency, and taxonomies of review smells, establishing a baseline for human‑only processes.

Recent studies on LLM‑assisted review show mixed effects on time and comment quality, but this work moves beyond tool‑level evaluation to analyze project‑wide transitions to agentic review.

We identified three AI‑adoption practices: Gradual AI Adoption and Rapid AI Agent Adoption accelerate reviews without improving smell metrics, while Rapid LLM Adoption raises smell prevalence without speed gains.

Developers should therefore adopt AI reviewers selectively, considering adoption history, reviewer identity, and pull‑request characteristics, and future work should explore how agentic systems can boost efficiency while preserving established quality standards.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers