Peerless

Referee Report

Manuscript: AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices

Recommendation: Reject

Summary and Overall Assessment

This paper investigates whether AI-generated reviews outperform reviews written by highly qualified human experts. Using 124 manuscripts and 134 human reviews collected through the rapidPeer service, the authors compare human reviews with reviews generated by Claude Opus 4.8. The resulting reviews are evaluated primarily by another language model, GPT-5.5, with a smaller validation exercise involving three former journal editors. The authors report overwhelming advantages for AI-generated reviews across all seven dimensions of review quality and discuss the implications for replacing human peer review.

The question is important, and the authors have assembled an interesting dataset. The attempt to compare AI-generated reviews with particularly strong human reviews, rather than with arbitrarily selected reports, is potentially valuable. The paper also contains several thoughtful robustness exercises.

Nevertheless, I recommend rejection. My central concern is that the paper makes claims considerably stronger than its research design can support. The authors purport to establish the superiority of AI peer review, but their evidence primarily establishes that AI-generated reviews are preferred by AI evaluators, with limited corroboration from human evaluators. This is a fundamentally different proposition.

The distinction matters because the purpose of scholarly peer review is not to produce reports that appear comprehensive, persuasive, or well written. It is to identify substantive problems, exercise informed scientific judgment, and ultimately improve research quality. The paper provides little direct evidence that AI is better at these tasks. Several additional problems concerning the evaluation procedure, selection of human reviewers, temporal comparability, and interpretation of the results further undermine the conclusions.

I elaborate below.

Major Comments

1. The paper does not measure the outcome that matters

The most fundamental weakness is a mismatch between the paper’s central claim and the outcomes actually measured.

The authors claim that AI-generated reviews dominate human reviews in providing constructive advice to improve scholarly work. Yet the primary outcome is whether an evaluator prefers one review to another. The study does not establish whether the criticisms raised are scientifically important, whether they correctly identify consequential errors, or whether implementing the suggested changes improves the manuscript.

A review can be articulate, comprehensive, and apparently sophisticated while offering recommendations that are unnecessary, misleading, or even detrimental to the research. Conversely, a relatively short human review that identifies one fundamental problem in identification, research design, or theoretical reasoning may be substantially more valuable than a lengthy AI-generated report covering numerous relatively minor issues.

The evaluation framework does not adequately distinguish between these possibilities.

The authors acknowledge that they do not measure whether subsequent revisions improved the papers. This acknowledgment is important, but it undermines rather than merely qualifies their central conclusion. Without evidence concerning the actual consequences of the reviews, the study cannot establish the substantive superiority of AI reviewing.

A convincing test would require independent subject-matter experts to evaluate the scientific validity, originality, and importance of the critiques, or an experimental design examining how reviews affect subsequent manuscript quality. Neither is provided.

2. The overwhelming reliance on AI judges creates a fundamental construct-validity problem

The headline result—that AI reviews win essentially every comparison—is based principally on evaluations conducted by another large language model.

The authors recognize the possibility that AI judges favor AI-generated text and attempt to address it through stylistic rewriting and cross-model evaluation. These are useful exercises, but they do not resolve the central issue.

The problem is not simply that models might recognize and prefer AI writing styles. It is that models may share implicit preferences concerning what constitutes a good review: comprehensiveness, systematic organization, explicit identification of weaknesses, extensive recommendations, and particular forms of argumentation.

These properties may improve the appearance of a review without necessarily improving its scientific value. Rewriting human reviews in an AI-like style does not eliminate differences in the types of criticisms produced or the implicit standards by which language models evaluate them.

Likewise, obtaining similar results using models from different developers does not establish independence from these shared evaluative preferences.

The finding that an AI evaluator strongly prefers AI-generated reviews is interesting in its own right. However, it should not be treated as compelling evidence that those reviews are objectively superior.

3. The human validation exercise is too small and limited to support the headline claims

The authors attempt to validate the AI judgments using three former journal editors who assess ten review pairs.

Although the paper reports 26 favorable judgments out of 30, these are not 30 independent evaluations of different manuscripts. They concern only ten manuscripts evaluated by three individuals. The effective number of distinct manuscript comparisons is therefore very small.

The paper provides insufficient information about variation across manuscripts and individual evaluators. It is unclear whether the four judgments favoring human reviews were concentrated on particular manuscripts, whether there were systematic differences across editors, or whether evaluations varied by research methodology.

Moreover, the description suggests that the editors performed the same overall preference task as the AI judge, rather than independently assessing the scientific merits of each critique against the original manuscript. If so, the human exercise largely reproduces the central construct-validity problem rather than resolving it.

The fact that the ten manuscripts resemble the full sample according to the AI evaluator’s own ratings does not establish that they constitute a representative sample for independent human assessment.

The human results may indicate that some experienced editors prefer the AI-generated reports. They do not establish that AI reviews are generally superior to expert human reviews, much less that they dominate human peer review at its best.

4. The claim to compare AI with human peer review “at its best” is not established

The authors emphasize that rapidPeer reviews are unusually high quality and argue that this makes the comparison especially demanding for AI.

However, the evidence supporting this characterization is limited. The principal quantitative evidence comes from satisfaction ratings supplied by authors receiving the reviews, while the evidence from former journal editors is described as informal.

Author satisfaction is not a reliable substitute for review quality. Authors may particularly appreciate constructive, detailed, and supportive reports, while finding scientifically justified but highly critical reviews less satisfactory. A review that identifies a fatal problem may be exceptionally valuable scientifically while being poorly received by its author.

Furthermore, rapidPeer reviews are produced under a specific set of instructions emphasizing comprehensive, constructive feedback before journal submission. The authors explicitly distinguish this task from assessing whether a manuscript meets a journal’s publication standards.

This is a consequential difference. The study evaluates a particular form of developmental manuscript feedback, not peer review in its full institutional and scientific sense.

The assertion that the human reviews represent peer review “at its best” therefore lacks a convincing empirical foundation. It would be more accurate to describe the sample as a collection of paid, positively evaluated pre-submission reviews drawn from a specific institutional service.

5. The comparison is temporally asymmetric

The human reviews were written between 2019 and 2026, whereas the AI reviews were generated using models available in 2026.

This introduces an important asymmetry that the robustness analysis does not adequately resolve.

The AI models may have been trained on scientific developments that occurred after the original human reviews were written. More importantly, they may have encountered subsequently published versions of the manuscripts, related working papers, or scholarly discussions directly relevant to the questions under review.

The authors attempt to address this issue by comparing submissions from different periods and examining whether the AI cites work published after the original review. Neither exercise rules out the underlying concern.

A model can benefit from later information without explicitly citing it. Likewise, the absence of a detectable difference between submission cohorts does not establish that the information environments were comparable.

The study therefore cannot cleanly distinguish superior reviewing ability from advantages arising from access to later scientific knowledge.

This limitation is particularly important given the exceptionally strong claims about AI’s superiority.

6. The analysis of factual accuracy is insufficiently convincing

One of the paper’s most striking findings is that 5.5% of AI-generated claims about manuscripts were classified as incorrect, compared with 15.9% of human-generated claims.

This result requires considerably more scrutiny than it receives in the main text.

First, the validity of the accuracy measure depends on how individual claims are identified, classified, and adjudicated. The main text does not establish that these assessments were independently validated by qualified subject-matter experts.

Second, the proportion of incorrect claims is not necessarily the appropriate measure of scientific reviewing quality. Errors differ enormously in their importance. A minor misdescription of a table is not equivalent to a fundamentally mistaken assertion about a causal identification strategy, mathematical proof, or substantive interpretation.

Third, the authors' validation exercise—in which the AI judge correctly identifies deliberately introduced contradictions of explicit statements in the manuscript—is not a convincing test of its ability to evaluate complex scientific criticism.

Recognizing an explicit factual contradiction is substantially easier than determining whether a subtle methodological objection is valid or whether a theoretical criticism is justified.

Finally, comparing percentages of incorrect claims without adequately considering the number and scientific importance of those claims may be misleading.

The conclusion that AI reviews make approximately one-third as many mistakes as human reviews is therefore insufficiently supported as a general statement about scientifically consequential errors.

7. The robustness checks do not address the central identification problem

The paper presents ten categories of robustness checks, but most establish the stability of results under variations of essentially the same evaluation framework.

For example, demonstrating that AI reviews remain preferred after controlling for length, modifying writing style, changing prompts, or switching AI judges is useful. It does not establish that the preferences correspond to greater scientific value.

There are also important differences between the matched and tuned review conditions. In particular, the authors report that the AI’s win rate for the sources dimension falls from 100% to 62% under the matched condition without the additional retrieval capabilities.

This is a substantial change and illustrates that performance depends meaningfully on the resources and configuration provided to the AI.

The main analysis would benefit from a more transparent separation of matched and tuned results, rather than emphasizing the most favorable configuration.

There is also a reporting inconsistency that needs correction: Section 3 reports 132 wins among 132 decided overall comparisons, while Figure 1 reports 134 out of 134. In addition, the 134 comparisons arise from 124 distinct manuscripts, which should be reflected in uncertainty calculations.

More generally, the paper offers striking percentages but limited presentation of statistical uncertainty, especially for the much smaller human evaluation exercise.

These concerns do not necessarily negate the observed preference advantage. They do, however, reinforce the need to distinguish that advantage from the much broader claims made in the paper.

8. The policy conclusions go substantially beyond the evidence

Section 4 considers sweeping changes to academic publishing, including eliminating routine human peer review.

Yet the study does not directly investigate editorial decisions, the quality of published research, the detection of consequential scientific errors, incentives for strategic behavior, or the long-run effects of AI reviewing on scientific diversity and originality.

The authors acknowledge several of these limitations and discuss potential harms. Nevertheless, their discussion repeatedly treats the demonstrated superiority of AI reviews as an established premise for considering institutional changes.

That premise has not been demonstrated.

Even if AI-generated reports receive higher evaluations in a particular experimental setting, it does not follow that replacing human reviewers would improve scientific quality or the functioning of journals.

Indeed, the authors' own concern that AI may encourage intellectual homogeneity points to a potentially important trade-off that the study does not evaluate.

The policy proposals would be more appropriate as hypotheses motivating future research, rather than implications supported by the present findings.

Overall Recommendation

The paper addresses a timely and potentially consequential question. Its dataset and experimental comparisons may be useful for understanding how contemporary language models generate and evaluate manuscript feedback.

However, the study does not establish its main substantive claim. It demonstrates a strong preference for AI-generated reviews under a largely AI-based evaluation system, accompanied by a small human preference exercise. It does not establish that AI-generated reviews identify more consequential scientific problems, provide more valid expert judgments, or lead to greater improvements in research quality.

These are not peripheral limitations that can be resolved through additional robustness checks or modest revisions to the exposition. They concern the central outcome, the validity of the evaluation framework, and the interpretation of the empirical evidence. Addressing them convincingly would require a substantially different study, with independent domain-expert assessments and preferably direct measures of improvements to manuscripts.

I therefore do not believe the manuscript supports its central conclusions or its proposed implications for the future of peer review.

My recommendation is rejection.