Checking the model's work: nextest's 25-hour review, outside physicists, JetBrains' samples
Three posts published on 14 and 15 November describe work done with AI models: an install script for a Rust test runner, a proof in particle physics and a survey of the models' own opinions. The tasks have nothing in common. What they share is that each author describes a step, outside the model, that decided whether the result could be trusted.
The nextest maintainer used Claude Code and Codex to build a test harness of dozens of cases across shells, with fault injection and a TLS test server, work the maintainer says would not otherwise have been possible in reasonable time. Two models then reviewed the code automatically. The defects came out in about 25 hours of manual review over four weekends: no test for the default install, case-insensitive hash comparisons, no Ctrl-C tests on Windows, and scripts that allowed redirects from HTTPS to HTTP with tests asserting that they worked. The maintainer's explanation is that once one model had written the wrong behaviour into the tests, later reviewers read the passing tests as proof of intent.
Nathaniel Craig's account on OpenAI's science blog divides the work differently. He brought the strategy, GPT-6 Astra proposed how to carry the force fields onto finer grids, showed that the comparisons converge and led the detailed proof. Confidence came from three lattice theorists who read the proof and asked questions. Even after that, Craig calls the result a candidate and lists properties, such as reflection positivity at finite lattice spacing, that are still unchecked.
JetBrains' experiment shows why a single output is weak evidence. Asked 50 times to pick a language at random, models returned the same few languages and mostly attacked them; one model defended and attacked Go's error handling in almost equal numbers across fresh conversations; some repeated a sentence word for word. One answer would have looked like an opinion. Fifty showed a habit.

What to watch
None of the three treats the model's output as finished. One relied on a human reading every line, one on domain experts outside the project, and one on repetition large enough to show a pattern. Projects adopting coding agents for infrastructure will have to decide which of these checks they can afford, because the nextest post shows that automated review by another model can confirm a mistake instead of catching it.