Testing how well Lemma catches errors in scientific papers
We are sharing the first results from our tests of paper validation in Lemma, our AI co-scientist. We tested it on papers whose errors are already known, and on about 28,000 references from recent papers.
- lemma
- paper validation
- benchmark
A paper can take months or years to write. By the final draft, the authors have read it dozens of times, and familiarity makes small errors easy to miss: a dropped factor, a weak step in a proof, a reference that does not match the published record. Some of these errors pass peer review and only come to light when the authors publish a correction or withdraw the paper.
Lemma gives a paper one more careful read before it goes out. It checks the math, the proofs, the claims and the references. It does not only report that something looks wrong. For each finding, Lemma shows the validation steps it took and the reasoning behind its conclusion, so the author can inspect the check and judge it for themselves. If Lemma cannot confirm something, it says so. What to change stays with the author.
To measure this, we started where the answers are known. SPOT is a public benchmark of papers that their authors later corrected or withdrew because of an error, with each error confirmed against the authors’ own account. We took the 26 mathematics and computer science papers in it whose errors are in equations or proofs. They contain 29 known errors. We then asked one question of every system we tested: how many of those known errors does it find? Lemma found 26 of the 29, or 89.7%. Google reports the same figure for its Paper Assistant Tool on the same papers. The general-purpose AI models we ran found between 48% and 79%.
We repeated the test on a set that has never been published: 50 recent physics papers, which the authors later corrected or withdrew because of an error, with each error confirmed against the authors’ own account. Lemma found 84% of those errors. Gemini 3.1 Pro found 52%.
A tool could score well on tests like these by flagging everything. So we also ran Lemma on 20 papers from leading mathematics and computer science journals that have no known errors, and on later versions of papers where the authors had already fixed the error. It raised less than a third as many issues per page on the journal papers, and it did not report any of the fixed errors.
References go wrong in their own way: a wrong year, a missing author, or a citation to a paper that does not exist. We built a test of 104 references with known answers, and Lemma gave the right verdict on 103 of them. We then ran it on the reference lists of 459 recent preprints, most of them in physics, about 28,000 references in all. This set has no answer key, so the numbers show what Lemma found, not what it may have missed. 98.7% of the references matched the published record. More than one paper in three had at least one reference with a single small error. About one paper in twenty had a reference with a wrong title, several wrong details, or no matching work at all.
These tests have limits. They cover mathematics, computer science and physics, and other fields may differ. The papers and their corrections are public, so an AI model may have seen them in training. We ran two checks for this and saw no sign that Lemma was recalling the errors from memory, though checks of this kind cannot rule it out. Google’s Paper Assistant Tool is offered to authors at some conferences but is not open for general use, and its prompts and grader are not published, so we could not run it ourselves. We compare against the figure in Google’s paper. When we scored the baseline model from that paper with our own grader, we got the same 55.2% that Google reports.
Lemma does not replace peer review, and it does not certify that a paper is correct. It is one more careful reader before you submit. We are continuing to test it on more fields and more kinds of error, and we will share those results as we have them.
You can run these checks on your own paper today. Try Lemma free.