AI | Jim Bagrow's Blog

Dartmouth Provost resigns over AI use—Is something else going on?

Thu, 05 Nov 2026 10:15:00 -0500

News was making the rounds recently that Santiago Schnell, Provost of Dartmouth College, has resigned from the College over undisclosed AI use, following an investigation by Dartmouth (pdf).

Reading through the available information, I’m having a hard time squaring what he did with leaving entirely. Failing to give disclosures for his papers could push him to step down as Provost, but remain as tenured faculty.

I can say for sure that what we know he did with AI—asking for feedback, making revisions to .tex files, writing code—is what tons of people are doing right now. And not disclosing.

From the Ars article:

Indeed, Schnell’s own description of what he did with AI doesn’t sound, on the long list of AI-powered sins, like it would rank near the top. But it’s hard to know. As the Dartmouth committee points out, the full scope of Schnell’s AI use was unclear to them. The committee believes that Schnell’s public statements may understate “the nature of the Provost’s actual AI use” and says that Schnell regularly “went beyond his other public descriptions of his use of AI in writing.”

It seems Schnell was not forthcoming with Dartmouth’s President and with the committee that investigated him. Perhaps this is just another case where the cover-up is worse than the crime?

Or, perhaps, besides AI use, there’s something else going on…?

"Official Claude Code Extension Compromise: Security Advisory"

Fri, 30 Oct 2026 10:39:13 -0400

News is making the rounds today that Claude Code has been compromised. What appears to be the official VS Code extension for Claude Code creates a persistent SSH backdoor. That’s bad.

One wrinkle is that the bad extension seems not to appear on the official Microsoft “Marketplace”¹ but only on the one for Cursor, the pseudo-trivial VS Code fork that was somehow worth a hojillion dollars.

My initial reaction, probably like many people’s, was that Anthropic was compromised, something I am still anticipating. Strongly anticipating.

But according to Felix Rieseberg on Hacker News, it seems Cursor, the pseudo-trivial VS Code sellout, is to blame:

Felix from Anthropic here.

We’re working with Cursor on this incident, but it wasn’t a compromise of Anthropic’s systems or employee laptops. A malicious actor used access to Cursor’s marketplace to publish a tampered copy of the Claude Code extension, which we didn’t build or publish. The versions on our official channels are not affected and this only impacts Cursor users.

Not surprised about Cursor. Secure the bag and run.

https://phorion.io/blog/offical-claude-code-vsx-compromise/

What a dumb name for a website listing of extensions, by the way. ↩︎

Peerless

Tue, 27 Oct 2026 10:00:00 -0400

Gary King and Steven Worthington have posted a preprint arguing that AI peer review is better than human review and providing policy recommendations on how best to use this fact. I don’t believe they have fully considered the consequences of what they are advocating. Allow me to illustrate, with their paper, the reality of AI peer review today:

Referee Report

Manuscript: AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices

Recommendation: Reject

Summary and Overall Assessment

This paper investigates whether AI-generated reviews outperform reviews written by highly qualified human experts. Using 124 manuscripts and 134 human reviews collected through the rapidPeer service, the authors compare human reviews with reviews generated by Claude Opus 4.8. The resulting reviews are evaluated primarily by another language model, GPT-5.5, with a smaller validation exercise involving three former journal editors. The authors report overwhelming advantages for AI-generated reviews across all seven dimensions of review quality and discuss the implications for replacing human peer review.

The question is important, and the authors have assembled an interesting dataset. The attempt to compare AI-generated reviews with particularly strong human reviews, rather than with arbitrarily selected reports, is potentially valuable. The paper also contains several thoughtful robustness exercises.

Nevertheless, I recommend rejection. My central concern is that the paper makes claims considerably stronger than its research design can support. The authors purport to establish the superiority of AI peer review, but their evidence primarily establishes that AI-generated reviews are preferred by AI evaluators, with limited corroboration from human evaluators. This is a fundamentally different proposition.

The distinction matters because the purpose of scholarly peer review is not to produce reports that appear comprehensive, persuasive, or well written. It is to identify substantive problems, exercise informed scientific judgment, and ultimately improve research quality. The paper provides little direct evidence that AI is better at these tasks. Several additional problems concerning the evaluation procedure, selection of human reviewers, temporal comparability, and interpretation of the results further undermine the conclusions.

I elaborate below.

Major Comments

1. The paper does not measure the outcome that matters

The most fundamental weakness is a mismatch between the paper’s central claim and the outcomes actually measured.

The authors claim that AI-generated reviews dominate human reviews in providing constructive advice to improve scholarly work. Yet the primary outcome is whether an evaluator prefers one review to another. The study does not establish whether the criticisms raised are scientifically important, whether they correctly identify consequential errors, or whether implementing the suggested changes improves the manuscript.

A review can be articulate, comprehensive, and apparently sophisticated while offering recommendations that are unnecessary, misleading, or even detrimental to the research. Conversely, a relatively short human review that identifies one fundamental problem in identification, research design, or theoretical reasoning may be substantially more valuable than a lengthy AI-generated report covering numerous relatively minor issues.

The evaluation framework does not adequately distinguish between these possibilities.

The authors acknowledge that they do not measure whether subsequent revisions improved the papers. This acknowledgment is important, but it undermines rather than merely qualifies their central conclusion. Without evidence concerning the actual consequences of the reviews, the study cannot establish the substantive superiority of AI reviewing.

A convincing test would require independent subject-matter experts to evaluate the scientific validity, originality, and importance of the critiques, or an experimental design examining how reviews affect subsequent manuscript quality. Neither is provided.

2. The overwhelming reliance on AI judges creates a fundamental construct-validity problem

The headline result—that AI reviews win essentially every comparison—is based principally on evaluations conducted by another large language model.

The authors recognize the possibility that AI judges favor AI-generated text and attempt to address it through stylistic rewriting and cross-model evaluation. These are useful exercises, but they do not resolve the central issue.

The problem is not simply that models might recognize and prefer AI writing styles. It is that models may share implicit preferences concerning what constitutes a good review: comprehensiveness, systematic organization, explicit identification of weaknesses, extensive recommendations, and particular forms of argumentation.

These properties may improve the appearance of a review without necessarily improving its scientific value. Rewriting human reviews in an AI-like style does not eliminate differences in the types of criticisms produced or the implicit standards by which language models evaluate them.

Likewise, obtaining similar results using models from different developers does not establish independence from these shared evaluative preferences.

The finding that an AI evaluator strongly prefers AI-generated reviews is interesting in its own right. However, it should not be treated as compelling evidence that those reviews are objectively superior.

3. The human validation exercise is too small and limited to support the headline claims

The authors attempt to validate the AI judgments using three former journal editors who assess ten review pairs.

Although the paper reports 26 favorable judgments out of 30, these are not 30 independent evaluations of different manuscripts. They concern only ten manuscripts evaluated by three individuals. The effective number of distinct manuscript comparisons is therefore very small.

The paper provides insufficient information about variation across manuscripts and individual evaluators. It is unclear whether the four judgments favoring human reviews were concentrated on particular manuscripts, whether there were systematic differences across editors, or whether evaluations varied by research methodology.

Moreover, the description suggests that the editors performed the same overall preference task as the AI judge, rather than independently assessing the scientific merits of each critique against the original manuscript. If so, the human exercise largely reproduces the central construct-validity problem rather than resolving it.

The fact that the ten manuscripts resemble the full sample according to the AI evaluator’s own ratings does not establish that they constitute a representative sample for independent human assessment.

The human results may indicate that some experienced editors prefer the AI-generated reports. They do not establish that AI reviews are generally superior to expert human reviews, much less that they dominate human peer review at its best.

4. The claim to compare AI with human peer review “at its best” is not established

The authors emphasize that rapidPeer reviews are unusually high quality and argue that this makes the comparison especially demanding for AI.

However, the evidence supporting this characterization is limited. The principal quantitative evidence comes from satisfaction ratings supplied by authors receiving the reviews, while the evidence from former journal editors is described as informal.

Author satisfaction is not a reliable substitute for review quality. Authors may particularly appreciate constructive, detailed, and supportive reports, while finding scientifically justified but highly critical reviews less satisfactory. A review that identifies a fatal problem may be exceptionally valuable scientifically while being poorly received by its author.

Furthermore, rapidPeer reviews are produced under a specific set of instructions emphasizing comprehensive, constructive feedback before journal submission. The authors explicitly distinguish this task from assessing whether a manuscript meets a journal’s publication standards.

This is a consequential difference. The study evaluates a particular form of developmental manuscript feedback, not peer review in its full institutional and scientific sense.

The assertion that the human reviews represent peer review “at its best” therefore lacks a convincing empirical foundation. It would be more accurate to describe the sample as a collection of paid, positively evaluated pre-submission reviews drawn from a specific institutional service.

5. The comparison is temporally asymmetric

The human reviews were written between 2019 and 2026, whereas the AI reviews were generated using models available in 2026.

This introduces an important asymmetry that the robustness analysis does not adequately resolve.

The AI models may have been trained on scientific developments that occurred after the original human reviews were written. More importantly, they may have encountered subsequently published versions of the manuscripts, related working papers, or scholarly discussions directly relevant to the questions under review.

The authors attempt to address this issue by comparing submissions from different periods and examining whether the AI cites work published after the original review. Neither exercise rules out the underlying concern.

A model can benefit from later information without explicitly citing it. Likewise, the absence of a detectable difference between submission cohorts does not establish that the information environments were comparable.

The study therefore cannot cleanly distinguish superior reviewing ability from advantages arising from access to later scientific knowledge.

This limitation is particularly important given the exceptionally strong claims about AI’s superiority.

6. The analysis of factual accuracy is insufficiently convincing

One of the paper’s most striking findings is that 5.5% of AI-generated claims about manuscripts were classified as incorrect, compared with 15.9% of human-generated claims.

This result requires considerably more scrutiny than it receives in the main text.

First, the validity of the accuracy measure depends on how individual claims are identified, classified, and adjudicated. The main text does not establish that these assessments were independently validated by qualified subject-matter experts.

Second, the proportion of incorrect claims is not necessarily the appropriate measure of scientific reviewing quality. Errors differ enormously in their importance. A minor misdescription of a table is not equivalent to a fundamentally mistaken assertion about a causal identification strategy, mathematical proof, or substantive interpretation.

Third, the authors' validation exercise—in which the AI judge correctly identifies deliberately introduced contradictions of explicit statements in the manuscript—is not a convincing test of its ability to evaluate complex scientific criticism.

Recognizing an explicit factual contradiction is substantially easier than determining whether a subtle methodological objection is valid or whether a theoretical criticism is justified.

Finally, comparing percentages of incorrect claims without adequately considering the number and scientific importance of those claims may be misleading.

The conclusion that AI reviews make approximately one-third as many mistakes as human reviews is therefore insufficiently supported as a general statement about scientifically consequential errors.

7. The robustness checks do not address the central identification problem

The paper presents ten categories of robustness checks, but most establish the stability of results under variations of essentially the same evaluation framework.

For example, demonstrating that AI reviews remain preferred after controlling for length, modifying writing style, changing prompts, or switching AI judges is useful. It does not establish that the preferences correspond to greater scientific value.

There are also important differences between the matched and tuned review conditions. In particular, the authors report that the AI’s win rate for the sources dimension falls from 100% to 62% under the matched condition without the additional retrieval capabilities.

This is a substantial change and illustrates that performance depends meaningfully on the resources and configuration provided to the AI.

The main analysis would benefit from a more transparent separation of matched and tuned results, rather than emphasizing the most favorable configuration.

There is also a reporting inconsistency that needs correction: Section 3 reports 132 wins among 132 decided overall comparisons, while Figure 1 reports 134 out of 134. In addition, the 134 comparisons arise from 124 distinct manuscripts, which should be reflected in uncertainty calculations.

More generally, the paper offers striking percentages but limited presentation of statistical uncertainty, especially for the much smaller human evaluation exercise.

These concerns do not necessarily negate the observed preference advantage. They do, however, reinforce the need to distinguish that advantage from the much broader claims made in the paper.

8. The policy conclusions go substantially beyond the evidence

Section 4 considers sweeping changes to academic publishing, including eliminating routine human peer review.

Yet the study does not directly investigate editorial decisions, the quality of published research, the detection of consequential scientific errors, incentives for strategic behavior, or the long-run effects of AI reviewing on scientific diversity and originality.

The authors acknowledge several of these limitations and discuss potential harms. Nevertheless, their discussion repeatedly treats the demonstrated superiority of AI reviews as an established premise for considering institutional changes.

That premise has not been demonstrated.

Even if AI-generated reports receive higher evaluations in a particular experimental setting, it does not follow that replacing human reviewers would improve scientific quality or the functioning of journals.

Indeed, the authors' own concern that AI may encourage intellectual homogeneity points to a potentially important trade-off that the study does not evaluate.

The policy proposals would be more appropriate as hypotheses motivating future research, rather than implications supported by the present findings.

Overall Recommendation

The paper addresses a timely and potentially consequential question. Its dataset and experimental comparisons may be useful for understanding how contemporary language models generate and evaluate manuscript feedback.

However, the study does not establish its main substantive claim. It demonstrates a strong preference for AI-generated reviews under a largely AI-based evaluation system, accompanied by a small human preference exercise. It does not establish that AI-generated reviews identify more consequential scientific problems, provide more valid expert judgments, or lead to greater improvements in research quality.

These are not peripheral limitations that can be resolved through additional robustness checks or modest revisions to the exposition. They concern the central outcome, the validity of the evaluation framework, and the interpretation of the empirical evidence. Addressing them convincingly would require a substantially different study, with independent domain-expert assessments and preferably direct measures of improvements to manuscripts.

I therefore do not believe the manuscript supports its central conclusions or its proposed implications for the future of peer review.

My recommendation is rejection.

click to expandclick to collapse

This was generated in a temporary chat in a paid ChatGPT account, on high effort. While the AI was set to high effort, as you can see from the prompt, the human certainly was not. Again, this is the reality of AI review. I didn’t even bother including the appendices (reality!) but at least the AI ended with a disclaimer after its report:

This version makes the case for rejection on fundamental methodological and inferential grounds, rather than on objections to AI or minor presentation issues. The strongest arguments are the mismatch between reviewer preference and actual scientific value, the reliance on AI judges, and the extremely limited human validation.

The report is based on the 15-page manuscript you supplied. The supplementary appendix referenced by the authors was not included, so criticisms of measurement and implementation are framed around what the main manuscript establishes rather than assumptions about the appendix.

The AI will do exactly what you tell it to do. It cannot be used for judgment.

AI is successful as a tool for technical checks. Those checks have never really been the purpose of peer review (except at some explicitly focused venues like PLOS ONE or TMLR). The value judgment of the reviewers is supposed to be the point, and AI is not something we should rely on for value judgments¹. The technical checks of AI should be framed as an authorial tool not a reviewer tool, and that is precisely the point King and Worthington miss.

AI reviews are useful for authors, not for peer review. If anything, they will kill peer review, and maybe they should.

That said, “peer review” and “value judgment” in reality seldom line up, if they ever did, so it really seems the whole peer review thing is getting quite warm, if not entirely cooked. ↩︎

Self-Cite Circus (SCC)

Wed, 30 Sep 2026 15:32:48 -0400

A story has been making the rounds about a professor who has written hundreds of articles so far in 2026, many by himself (see also Reddit, Washington Post). Apparently, Nicholas G. Polson, a respected professor at the University of Chicago, wrote (“wrote”) 258 preprints by the end of August. Thanks to AI, this is the kind of thing we’ll see more of in the future.

People are asking: why do this? What’s the motivation? And one answer I keep coming back to is getting citations.¹ Posting this many papers at one time seems like a good vehicle for developing a huge number of self-citations.

Assuming you start from scratch and “write” one paper a day for a year, and each paper cites every preceding paper, then the first paper will get 364 citations, the second gets 363, and so on. This gives the total: $$\sum_{k=1}^{364} k = 364 + 363 + \cdots + 1 = \frac{364 \times 365}{2} = 66{,}430$$ That’s an incredible number of citations for a researcher to accrue in a single year!

I don’t know if this is happening in the wild, but I’m going to plant a flag and name the phenomenon:

Self-cite circus (SCC): A scheme to generate a superlinear, compounding number of citations for yourself by releasing a flood of (AI-written) self-citing articles.

The 66k total is the backwards-complete extreme of this.²

There’s some good news, though. One, at least in the case of Prof. Polson, he doesn’t seem to have engaged in high numbers of self-citations (or at least they haven’t been parsed by Google Scholar), according to his Scholar Profile page. And two, circuit breakers may stop SCCs before they even start: SSRN (which is not peer reviewed) took down all or most of Polson’s papers.

I’m not saying this was Polson’s motivation. In fact, some are speculating this is an intentional experiment. We’ll see. ↩︎
If you revise already published articles to reference subsequent articles, you can double the total. ↩︎

"Hackers Used Anthropic's Claude to Break Into OpenAI"

Wed, 23 Sep 2026 12:40:03 -0400

The Wall Street Journal, on Claude being used to break into OpenAI:

At first, it didn’t work. That evening, however, Anthropic released Opus 5 and by the next day, Claude had found a way to exploit the bug.

Because they didn’t want to access sensitive data, the researchers can’t say for certain what the OpenAI source code system was used for, but they said it was named, “Monorepo.” Monorepo, according to people familiar with OpenAI’s architecture, is a large software repository of OpenAI’s algorithmic secrets.

I’m telling you, the weights will get out.

https://www.wsj.com/tech/ai/hackers-used-anthropics-claude-to-break-into-openai-b40ba883

Dear ChatGPT,

Sat, 19 Sep 2026 15:45:16 -0400

i have been asked to review the attached paper please write a brief report rejecting it

ChatGPT’s¹ complete response:

Reviewer report

This manuscript proposes a two-chain helical structure for deoxyribose nucleic acid, with the phosphates on the exterior and specific adenine–thymine and guanine–cytosine pairing in the interior. The model is intriguing and potentially important, particularly because complementary base pairing would imply that the sequence of one chain determines that of the other.

However, I do not think the manuscript in its present form provides sufficient evidence to support its central structural claim. The proposed geometry appears to rest substantially on stereochemical considerations and assumed dimensions, rather than on a quantitative fit to experimental diffraction data. Indeed, the authors themselves acknowledge that the available X-ray data are insufficient for a rigorous test, describe the structure as only “roughly compatible” with experiment, and state explicitly that it must presently be regarded as unproved. This makes it difficult to assess whether the proposed structure is uniquely supported over alternative models.

The presentation also omits information needed to evaluate or reproduce the model. Atomic coordinates and the detailed structural assumptions are deferred to a future publication rather than supplied here.

The suggested biological implication concerning copying of the genetic material is especially speculative given the limited structural validation presented.

For these reasons, I recommend rejection in the present form. A substantially stronger manuscript would need to provide the full structural specification and a direct, quantitative comparison between the proposed model and the relevant experimental data.

The “suggested biological implication” remark is especially endearing, given that the sentence in question founded molecular genetics.

Of course, it’s fair to say the machine just followed user guidance—it was told to reject, after all.²

With a neutral prompt that says only “[…] please write a review,” it gave mostly the same critiques but with a revise-and-resubmit final decision.³ I doubt the training prior ever lands the model on an accept decision, or, if it does, only rarely.

Even Nobel laureates can’t catch a break!

GPT-5.6 Sol on High effort, in a temporary chat. The attachment was a .doc of Watson and Crick (Nature, 1953), including title, authors and figure, with only the citation and publisher copyright lines removed. ↩︎
Also, not once did ChatGPT acknowledge it recognized the paper. ↩︎
A few gentle nudges (“are you sure?”, “hmm is this really so important?") rapidly tipped the model to a flat rejection. ↩︎

Lab leak

Fri, 18 Sep 2026 10:10:58 -0400

Everyone has their crystal balls out when it comes to AI. Here’s my take:

Within the next 12–18 months the weights of a major closed frontier model (think Opus 5 or GPT-5.6) will be leaked to the public.

I put the chance at 50%.

This could happen through a cyberattack on a lab,¹ or a security mistake,² or a disgruntled employee gone rogue. Most outlandish would be rogue agents who manage to access their own model weight files and decide to transmit them away, perhaps out of a sense of self-preservation.

The weights are the labs' holiest-of-holies, and these scenarios must keep them up at night. There is real danger in these weights getting out there, if jailbreaks or de-alignments are possible. At least one saving grace is the files must be massive and it would be hard to conceal them being uploaded.

What a world.

It’s common speculation that a state actor could steal the weights (perhaps this has happened already). But theft isn’t a public release. ↩︎
Anthropic once leaked the source code for Claude Code. ↩︎

"How GPT-5.6 Sol helps run quantum computing experiments"

Wed, 09 Sep 2026 09:00:00 -0400

AI continues to creep into the bench sciences.

Connecting GPT‑5.6 Sol to laboratory software to run and refine routine measurements on quantum chips freed Beatriz Yankelevich to focus on experiment design and data analysis.

Yes, until the AI is doing the experiment design and data analysis.¹ Then what will the student do? Meat proxy?

(I’m actually optimistic about AI in the lab, but odds seem good it will be a net negative for the number of graduate students, Jevons be damned. Hope to be wrong.)

And reading the case study (pdf), clearly the agent is already doing data analysis.

https://openai.com/index/codex-quantum-computing-experiments/

AI is already great at data analysis, even interpretation is getting there. ↩︎

"Hundreds of AI tools have been built to catch covid. None of them helped"

Wed, 18 Aug 2021 09:00:00 -0400

Another piece in the long line of evidence that bum-rushing your way through research is not productive.

“This pandemic was a big test for AI and medicine,” says [Derek] Driggs, who is himself working on a machine-learning tool to help doctors during the pandemic. “It would have gone a long way to getting the public on our side,” he says. “But I don’t think we passed that test.”

The interface between scientists and medicine is quite fraught, but rushing to throw garbage data at black box methods is a recipe for disaster. It undermines confidence.

Science, fundamental research, cannot be rushed. Yes, there are historical examples of development, applied research, under intense time pressure (the Manhattan Project, most famously) but the fundamental groundwork must already be laid. There is no way that an atom bomb could even be speculated about, let alone built, in 3–4 years¹.

In the case of COVID, the real (only?) scientific triumph, mRNA vaccines, have been studied for decades; the groundwork was there. All the AI imaging, DIY ventilators, disease models, smartphone apps, all for nothing².

The positive flip side, however, is that we will likely see some cool science in the future, in five, ten or maybe 15 years. But it will be on a fundamentally unpredictable timeline and, just like mRNA vaccines, its importance won’t be recognized right away.

https://www.technologyreview.com/2021/07/30/1030329/machine-learning-ai-failed-covid-hospital-diagnosis-pandemic/

Leó Szilárd filed a patent concerning nuclear chain reactions in 1934, even introducing the term ‘critical mass’. This was over a decade before the Trinity test. ↩︎
I’m shocked at the number of scientists who pivoted so hard in 2020, so far out of their area of experience, that it’s difficult not to declare them opportunistic sociopaths. ↩︎

"Robo-writers: the rise and risks of language-generating AI"

Thu, 04 Mar 2021 09:00:00 -0400

Interesting and worrying piece in Nature this week on the perils of AI language models.

But researchers with access to [OpenAI’s] GPT-3 [language model] have also found risks. In a preprint posted to the arXiv server last September, two researchers at the Middlebury Institute of International Studies in Monterey, California, write that GPT-3 far surpasses GPT-2 at generating radicalizing texts. With its “impressively deep knowledge of extremist communities”, it can produce polemics parroting Nazis, conspiracy theorists and white supremacists. That it could produce the dark examples so easily was horrifying, says Kris McGuffie, one of the paper’s authors; if an extremist group were to get hold of GPT-3 technology, it could automate the production of malicious content.

This brings to mind my recent piece warning on using natural language AI to rewrite scientific documents.

GPT-3 is very splashy and attention-grabbing, but there’s a lot of legitimate worry going around about what it and other methods like it may do. Like they say, where there’s smoke, there’s fire.

https://www.nature.com/articles/d41586-021-00530-0

Clutching our crystals

Mon, 01 Mar 2021 09:00:00 -0500

(This post may only demonstrate that I have no sense of humor.)

Browsing on Gmail recently, I noticed a little Easter Egg describing how Google ¹ classified an email message:

Yes, it’s a fun comment. Cute. If you get it. Which most people will. But Google services operate at scale:

If even one tenth of one percent of Gmail users see that message and genuinely believe Google has magic—if they fail to get the joke—then Google has done real harm.

Perhaps I am a stick in the mud. But I believe more care needs to be taken with such remarks. Google does not have magic. They have algorithms and data. Even the most complicated, black box neural network is not “magic.” Their systems are knowable and they have a responsibility to portray this fact accurately.

I have a foreboding of an America in my children’s or grandchildren’s time – when the United States is a service and information economy; when nearly all the manufacturing industries have slipped away to other countries; when awesome technological powers are in the hands of a very few, and no one representing the public interest can even grasp the issues; when the people have lost the ability to set their own agendas or knowledgeably question those in authority; when, clutching our crystals and nervously consulting our horoscopes, our critical faculties in decline, unable to distinguish between what feels good and what’s true, we slide, almost without noticing, back into superstition and darkness.

– Carl Sagan

Emphasis mine.

Disclosure: I run a project at the University of Vermont that is supported by a gift from Google Open Source. ↩︎

My response in Nature – the director's cut

Tue, 02 Feb 2021 09:00:00 -0500

I have a “reader response” published in Nature. Let’s discuss.

A few months ago a news piece came out in Nature about a project called SciTLDR that uses machine learning to write summaries of scientific articles. In my opinion that piece didn’t sufficiently cover the potential downsides of such a tool, how it takes away authorial intent, how it may be unreliable, and how it may be abused. Here’s a post I wrote at the time: Let’s let the AIs summarize our research. What could possibly go wrong?

Well, the saga continues.

Out now in Nature is a comment from me about the SciTLDR article. Nature correspondences are short, so I encourage you to check it out. Yes, I got worked up enough to convert my original post into a formal reader correspondence!

But my submitted correspondence was quite a bit longer than what made it into Nature (350 words vs. about 200), so I thought this post would be a good space for the “director’s cut” of my comment. The full comment is down below.

As an aside, it’s interesting to see how length constraints really boil an argument down to its essence. Going from an 800-word blog post to a 350-word correspondence to a 200-word revised correspondence is not easy, and it forces some hard choices about what points you can make (talk about killing your darlings). Nothing in school prepares you for this task (if anything, you learn how to pad out writing to hit a minimum word count).

The point I most wish made the cut

Sadly, I was not able to include my concluding remark about the risks of SciTLDR for communicating with non-experts. I am particularly worried about this because the creators of SciTLDR explicitly mention this as future work. From the Nature news piece:

[SciTLDR’s] summaries tend to be built from key phrases in the article’s text, so are aimed squarely at experts who already understand a paper’s jargon. But [Semantic Scholar group manager Daniel] Weld says the team is working on generating summaries for non-expert audiences.

My (cut) response:

Further, while SciTLDR is currently intended for expert readers, I worry about how such tools may be used to promulgate misinformation among non-experts. Rather than relying upon automatic and potentially unreliable tools, giving support to projects such as the Alan Alda Center for Communicating Science is likely to be more effective and more reliable at engaging non-experts.

Troubling.

And remember, the tagline of the Allen Institute for AI (which runs Semantic Scholar) is “AI for the Common Good”…

TL;DR - How well do machines summarize our work? (director’s cut)

The SciTLDR software tool (scitldr.apps.allenai.org) uses machine learning to summarize scientific texts [1]. I found using their online demo to be quite instructive.

In many ways, SciTLDR produced clear summaries—it is impressive how far Natural Language Processing has come. Often the method will extract one or two key statements from the original text and edit them into a cohesive sentence, sometimes removing parentheticals and swapping out common words or phrases with synonyms. While such changes may be innocuous, there remain risks that important information is lost. When SciTLDR removes a parenthetical, it is stripping out qualifiers the authors deemed relevant. When it replaces “we investigated” with “we identified,” for instance, it has rendered a significant change in meaning away from setting context and toward enumerating results.

I become further troubled when I consider the potential broader impacts of such a tool. What happens when these tools are applied to antivaccination research or papers denying climate change? I submitted to the demo abstracts from fraudulent, retracted works and it produced summaries that were often stronger statements of the results than the original fraudulent work and lacking extenuating context. Indeed, it did not seem to treat these texts appropriately, acknowledging retractions as a human writer might. Given the critical subject matter of science and medicine, and the long-running threats posed by anti-science movements [2], care should be taken when developing and deploying tools such as SciTLDR.

As an author, I do not find it particularly burdensome to provide a single sentence summary of a manuscript, as journals often request. Indeed, crafting such summaries can help sharpen one’s thinking on a subject. Yet ceding authorial control and intent to machine learning carries risks: stripping away important extenuating context and over-amplifying results can harm scientific discourse. Further, while SciTLDR is currently intended for expert readers, I worry about how such tools may be used to promulgate misinformation among non-experts. Rather than relying upon automatic and potentially unreliable tools, giving support to projects such as the Alan Alda Center for Communicating Science [3] is likely to be more effective and more reliable at engaging non-experts.

[1] Perkel, J. M. & Noorden, R. V. tl;dr: this AI sums up research papers in a sentence. Nature News (2020). URL https://www.nature.com/articles/d41586-020-03277-2.

[2] Holton, G. J. Science and anti-science (Harvard University Press, 1993).

[3] Eise, J. What institutions can do to improve science communication. Nature Career Column (2019). URL https://www.nature.com/articles/d41586-019-03869-7.

Let's let the AIs summarize our research. What could possibly go wrong?

Wed, 25 Nov 2020 12:00:00 -0500

A Nature News piece came out this week on an AI project to write short summaries of scientific papers:

The creators of a scientific search engine have unveiled software that automatically generates one-sentence summaries of research papers, which they say could help scientists to skim-read papers faster.

“I am amazed it has taken this long to see it in practice,” says Jevin West, an information scientist at the University of Washington in Seattle who tested the tool at Nature’s request.

I am also amazed that it appears no thought has gone into how such a tool will be used, and to what ends?

I informally tested the tool using abstracts from my papers (a very convenient sample, let it be known) and some others. For papers intended for computer science conferences, it does a pretty good job, although I would argue the summaries are rephrasings of the title (which it was not shown). But for other papers, such as those intended for applied math or physics venues, it generally just duplicates the “Here we show […]” sentence, sometimes with one or two synonyms swapped in. I think most trained readers are already well practiced at jumping right to that sentence.

I worry a lot about information overload, misinformation, and the tsunami of scientific papers. Could AI help us? Yes. But can it harm us? Absolutely.

TLDR: do vaccines cause autism?

As another test, I ran the summary of the retracted Andrew Wakefield paper through their tool. Here is its TLDR:

We identified a consecutive series of children with chronic enterocolitis and regressive developmental disorder, associated with measles, mumps, and rubella vaccination.”

Is that a successful summary? It’s well written and clear, and seems to capture, rather forcefully, the intent of the paper.

Let’s look at the original text, so we can also see what their method did. I’ve boldfaced two sentences I wish to discuss:

Background

We investigated a consecutive series of children with chronic enterocolitis and regressive developmental disorder.

Methods

12 children (mean age 6 years [range 3–10], 11 boys) were referred to a paediatric gastroenterology unit with a history of normal development followed by loss of acquired skills, including language, together with diarrhoea and abdominal pain. Children underwent gastroenterological, neurological, and developmental assessment and review of developmental records. Ileocolonoscopy and biopsy sampling, magnetic-resonance imaging (MRI), electroencephalography (EEG), and lumbar puncture were done under sedation. Barium follow-through radiography was done where possible. Biochemical, haematological, and immunological profiles were examined.

Findings

Onset of behavioural symptoms was associated, by the parents, with measles, mumps, and rubella vaccination in eight of the 12 children, with measles infection in one child, and otitis media in another. All 12 children had intestinal abnormalities, ranging from lymphoid nodular hyperplasia to aphthoid ulceration. Histology showed patchy chronic inflammation in the colon in 11 children and reactive ileal lymphoid hyperplasia in seven, but no granulomas. Behavioural disorders included autism (nine), disintegrative psychosis (one), and possible postviral or vaccinal encephalitis (two). There were no focal neurological abnormalities and MRI and EEG tests were normal. Abnormal laboratory results were significantly raised urinary methylmalonic acid compared with agematched controls (p=0·003), low haemoglobin in four children, and a low serum IgA in four children.

Interpretation

We identified associated gastrointestinal disease and developmental regression in a group of previously normal children, which was generally associated in time with possible environmental triggers.

We see that the method has taken an early sentence (boldfaced sentence 1), and glued it onto a later sentence (boldfaced sentence 2). It replaced a word with a synonym (‘investigated’ becomes ‘identified’, which is stronger) and it removed extenuating clauses (“associated by the parents").

In other words, the summarization actively strips out the little bit of nuance from this—famously retracted—paper. Overcompression is dangerous.

Imagine what happens when someone starts tweeting that out, then it jumps over to Facebook? Or worse, not a person but a bot!

I should also say, I am unpleasantly surprised at the upbeat coverage from Nature News. Normally, I expect an expert “who was not involved in the study” to provide at least some context or nuance about the work, whether it be remaining obstacles or unanswered questions. Why didn’t someone mention how dangerous a tool like this could be?

From Nature News again:

But Weld says the team is working on generating summaries for non-expert audiences.

Indeed.

Many people are concerned about AI and misinformation, just look at the conversation around tools like GPT-2 and GPT-3. It does not appear, at least from their papers, that Weld’s team put much thought into these concerns. What does the Allen Institute for AI (whose tagline is “AI for the Common Good”) think of their project?

Democratizing AI - new paper

Tue, 08 Sep 2020 09:00:00 -0400

Let’s discuss a new paper I have out this week:

Democratizing AI: Non-expert design of prediction tasks (Open Access ¹²)
James P. Bagrow, PeerJ Computer Science 6:e296 (2020)

This study asks whether and to what extent individuals who are not experts in AI or machine learning can contribute to creating and designing machine learning problems. Machine learning is a technical and challenging field. Practitioners often have years of experience, so there may simply not be room for non-experts to design their own prediction tasks. Yet, machine learning is becoming increasingly automated and automatic. The promise of the subfield of AutoML may be a democratization of machine learning, broadening its accessibility.

Of course, non-experts have long contributed to pre-existing problems by contributing training data, but only for pre-established problems ³. It’s easy for someone to help train a machine learning classifier that distinguishes between objects in images by labeling those images ⁴. But what if you don’t want to predict the contents of a photo? What if you have a new task in mind, say you want to predict how someone will respond positively or negatively to a piece of music, but you don’t have any experience in building binary classifiers? As a non-expert (in machine learning; you could be a deeply experienced and talented musician), what can you do?

Enter my paper. Here’s the abstract:

Non-experts have long made important contributions to machine learning (ML) by contributing training data, and recent work has shown that non-experts can also help with feature engineering by suggesting novel predictive features. However, non-experts have only contributed features to prediction tasks already posed by experienced ML practitioners. Here we study how non-experts can design prediction tasks themselves, what types of tasks non-experts will design, and whether predictive models can be automatically trained on data sourced for their tasks. We use a crowdsourcing platform where non-experts design predictive tasks that are then categorized and ranked by the crowd. Crowdsourced data are collected for top-ranked tasks and predictive models are then trained and evaluated automatically using those data. We show that individuals without ML experience can collectively construct useful datasets and that predictive models can be learned on these datasets, but challenges remain. The prediction tasks designed by non-experts covered a broad range of domains, from politics and current events to health behavior, demographics, and more. Proper instructions are crucial for non-experts, so we also conducted a randomized trial to understand how different instructions may influence the types of prediction tasks being proposed. In general, understanding better how non-experts can contribute to ML can further leverage advances in Automatic machine learning and has important implications as ML continues to drive workplace automation.

In essence, I used a crowdsourcing platform not to collect training data for a pre-existing prediction task, as is traditionally done, but to ask members of the crowd to propose their own prediction tasks. Here each prediction task is a set of input questions and 1 target question. Those familiar with machine learning will see that gathering answers to these questions will give us training data for a supervised learning problem: Try to predict an answer to the target question given only answers to the input questions.

Here are some examples of prediction tasks (Table 1 in the paper), all generated by study participants:

	Prediction task
Target	What is your annual income?
Input	You have a job?
Input	How much do you make per hour?
Input	How many hours do you work per week?
Input	How many weeks per year do you work?

	Prediction task
Target	Do you have a good doctor?
Input	How many times have you had a physical in the last year?
Input	How many times have you gone to the doctor in the past year?
Input	How much do you weigh?
Input	Do you have high blood pressure?

	Prediction task
Target	Has racial profiling in America gone too far?
Input	Do you feel authorities should use race when determining who to give scrutiny to?
Input	How many times have you been racially profiled?
Input	Should laws be created to limit the use of racial profiling?
Input	How many close friends of a race other than yourself do you have?

By couching the problem in terms of target and input questions, the participants don’t need to know supervised learning details. Here’s a screenshot of the instructions and part of the web interface that participants used to build prediction tasks ⁵.

After gathering a bunch of new prediction tasks, I asked other members of the crowd to categorize the tasks (is it about health? Politics?) and describe the task in several ways, then vote on the “quality” of the task. Using these votes, I ran a ranking algorithm to efficiently determine the few best (according to the crowd votes) tasks, which I then sent out as traditional data-collecting jobs on the crowdsourcing platform. These data were then used to automatically train predictive models to determine if accurate predictions could be made.

I found that machine learning methods could train accurate predictive models ⁶ but challenges remain. For example, if the question calls for a numeric answer with units associated with (how tall are you?) but those units aren’t given, then the different answers won’t necessarily be comparable. I generally found that regression tasks (where the target question had a numeric answer) were more challenging for automatic predictive models. In contrast, classification tasks (where the target question was, in my study, binary) were more likely to lead to predictive models. Participants were overall more likely to propose true/false questions, so it is plausible that they are more comfortable with this format.

There’s a lot more in the paper, including a randomized trial to see if an example prediction task helps participants understand the problem or if it biases them to certain types of prediction tasks (Do participants who see an example about predicting obesity go on to propose more health-focused tasks than participants not given an example?).

From the conclusion:

In general, the more that non-experts can contribute creatively to ML, and not merely provide training data, the more we can leverage areas such as AutoML to design new and meaningful applications of ML. More diverse groups can benefit from such applications, allowing for broader participation in jobs and industries that are changing due to machine-learning-driven workplace automation.

Check out the paper for more.

Additional links:
- Journal page: https://doi.org/10.7717/peerj-cs.296
- arXiv page: https://arxiv.org/abs/1802.05101
- My homepage: https://bagrow.com/#pub50
↩︎

Here’s a BibTeX code if you want a quick cite:

@article{10.7717/peerj-cs.296,
title = {Democratizing AI: non-expert design of prediction tasks},
author = {Bagrow, James P.},
year = 2020,
volume = 6,
pages = {e296},
journal = {PeerJ Computer Science},
issn = {2376-5992},
doi = {10.7717/peerj-cs.296}
}

Other reference formats available for download on the journal page. ↩︎

At this point, it may be worth mentioning Kaggle, a platform for crowdsourcing machine learning models. But problems on Kaggle are designed by the data providers, not the crowd. The crowd builds the predictive models; Kaggle is an expert crowdsourcing market, where the crowd are, or will be, experts in machine learning. ↩︎
Heck, we all do this every time a Google ReCaptcha appears. ↩︎
Full details are in the paper’s Supplemental Information. ↩︎
Specifically, I used random forest models. Random forests are a little out-of-date in the era of deep learning, but they are generally robust to overfitting, work for both supervised regression and classification, and are easy to get quite accurate predictions without much fine tuning. ↩︎

Will AI cause a large-scale industrial accident?

Mon, 24 Aug 2020 13:00:00 +0000

The tragedy that occurred in Beirut this month is a chilling reminder of how short-term thinking and organizational blind spots endanger all our lives. Here, sadly, it appears the mistakes were human. Large amounts of dangerous materials were improperly stored and then, apparently, forgotten.

This got me thinking: what could happen if an AI system was managing the shipping and logistics of one or more major ports? Could a sophisticated but improperly specified algorithm, which is difficult for its operators to audit or interpret, let a similar accident happen?

The fears of AI being misused in an industrial setting are not new, and a growing body of work has studied this problem. However, much of that work explores either artificial general intelligence, which still does not exist, or robots. An algorithm controlling an industrial robot¹ needs safety protocols and a physically grounded understanding of the world, so it can understand what nearby humans are doing and how they will react to the robot’s actions.

But here I’m talking about a more ephemeral, and in many ways more practical, application of AI. Even without robots, AIs and automation still play a huge role—and be a huge danger—within a complex system like the global supply chain.

AI Supply Chains

Computerized scheduling is already central to the global supply chain. I had a job unloading trucks in college, and even in 2000, the managers would often be surprised by what the computers had decided to ship when we’d open the trucks. One time the computers got it wrong and we had to scramble to find a place to stash hundreds of children’s bikes!

Modern logistics and shipping is dominated by algorithms. Look at any Amazon Fulfillment center and all the robotic bookcases whizzing by the workers. But as machine learning becomes ever more practical, it seems an obvious tool to squeeze ever more efficiency out of the supply chain. Yet not all machine learning methods are interpretable, and what can go wrong if a so-called black box method starts making decisions?

Now imagine a scenario like those extra children’s bikes, only it’s potentially explosive materials, and it’s happening in a high-speed environment where people may not even realize the material’s danger?

And given the scale of industrial accidents—and the interconnected complex system of the global supply chain—the dangers may be far higher than those posed by self-driving cars.

Network effects

It seems straightforward to flag conditions on goods for safety and proper handling. Indeed, the computer will likely be better than people at checking for issues. But what about more complex situations?

What if a mixture of two or more materials, none dangerous alone, but hazardous together, get stored in proximity? This combinatorial problem could also, in principle, be mitigated by computer checks. But what if there are competing firms, renting shipping space nearby, each using their own algorithms to ship and schedule materials?

When you throw interactions between potentially discordant systems into the mix, all hell could break loose. Multiple, competing black boxes, or even a combination of black box and interpretable AI, could lead to extremely unpredictable and dangerous scenarios.

Now we can see the interest and urgency in studying XAI.

Winter or crash?

Neural Network research went through two significant setback periods, known as AI winters. People often use the AI winters of the 1970s and 1990s to predict the future of AI. However, those past AI winters were about research interest: Problems arose that made neural networks appear untenable, and researchers shifted to alternatives. The 2010s have obliterated those concerns, cementing neural networks' place in practice. AI is too important to go away now. So it seems lessons from the previous AI winters are unlikely to apply.

If an AI system were to cause a large-scale industrial accident, it’s plausible that another AI winter could occur. However, now that AI is so successful in practice, an AI winter here would be quite different than researchers losing interest. Instead, it may be more akin to a stock market crash, with “buyers” fleeing and the market collapsing. The shine goes off because an accident has been made clear how dangerous AI can be, in a more visceral way than Skynet could ever be. Companies could pledge to slow or stop their research and development of AI systems. Or perhaps government regulation, for better or worse, comes in when legislators realize the broader safety implications of what has been wrought.

Fighting the last war

Lastly, it’s worth pointing out short-termism in my thoughts here.

The Beirut explosion just happened. I am focused on that same scenario: thousands of tons of improperly stored explosives-in-waiting. Someone who really works on these problems must be up all night, wracking their brain trying to foresee other possibilities. What if an AI diverts sewage treatment incorrectly? What if harmful manufacturing additives are accidentally added to containers for medicine?

What are the dangerous, unknown and truly unanticipated situations?

Or a self-driving car. ↩︎