SciLitBench's strongest models recovered 30% of results and 25% of limitations
A language model can find relevant papers reliably and still produce an incomplete evidence table. SciLitBench measured both jobs on the same corpus. The strongest supported result for recovering reported evaluation evidence was 30%. For paper limitations, it was 25%.
That is a different failure from hallucination. The extracted rows can be accurate, tidy, and useful while leaving most of the annotated evidence behind. A researcher who checks a few plausible entries may never notice what is missing.
The August 29 preprint SciLitBench follows one systematic review through title and abstract screening, full-text screening, and structured extraction. LLMs reduced screening work when the decision protocol was explicit. Evidence-complete extraction still needed direct verification.
One review, three different jobs
The benchmark comes from a real systematic review of computational methods for literature-review automation. A librarian-informed search across PubMed, Semantic Scholar, and Scopus produced 42,981 candidate records. The researchers retrieved 1,012 full texts, confirmed 888 included papers, and annotated 16,777 data elements across six fields.
Those fields range from publication year to domain, review stage, computational approach, evaluation results, and limitations. The first is a canonical value. The last two are open-ended evidence questions where correct answers can differ in wording and granularity.
SciLitBench evaluated 22 open-weight models from the Llama, Gemma, Qwen, Mistral, Phi, and gpt-oss families under frozen prompts. The models ran locally through Ollama. The released repository includes labels, prompts, schemas, predictions, scoring code, audit materials, calibration artifacts, and raw and harmonized annotations.
"Literature review" is doing too much work as a single label. Screening asks whether a paper should continue through the pipeline. Extraction asks which facts and arguments inside that paper answer a research question. A single model score hides that difference.
Screening improved when the protocol got specific
For title and abstract screening, a prompt that explicitly asked whether the record met the inclusion criteria improved average F2 by 17.0% relative to a generic classification prompt. Requiring exclusion reasoning raised the total relative gain to 28.8%. Few-shot examples did not improve the aggregate zero-shot result consistently.
The metric was F2 because recall mattered more than precision. A false positive costs reviewer time. A false negative can remove a relevant study from every later analysis. This matches the Cochrane Handbook, which says searches should aim for high sensitivity even when precision is relatively low.
At the selected voting threshold, the benchmark reached perfect recall with precision of 0.275 and cut the manual screening workload by about 90%. Relaxing recall to at least 0.98 raised precision to 0.411 and workload reduction to 93.7%. These are operating points, not universal settings. A medical safety review and a low-stakes product survey should not share a threshold merely because both use abstracts.
Full-text screening also benefited from human reasoning. Researcher-authored rationales, made from quoted evidence plus a short justification, raised average F2 from 0.551 to 0.633, a 15% relative gain. The best single configuration, Llama 3.3-70B, reached F2 0.981 at perfect recall and 0.910 precision on this corpus.
Model-generated structured reasoning improved average F2 by 3.8%. Researcher-authored rationales helped more because the examples recorded why a person made the inclusion decision.
Extraction failed by omission
Extraction performance fell as the requested field became more interpretive. The strongest open-weight models reached 0.97 accuracy for publication year. Jaccard overlap then fell to 0.70 for domain, 0.61 for review stage, and 0.37 for computational approach.
For open-ended fields, the researchers used a calibrated model judge because exact string matching would punish valid paraphrases. The highest supported calibrated recall was 0.30 for evaluation results and 0.25 for limitations. Some configurations reached precision as high as 0.92 and 0.82 on those fields, but they did so while recovering little of the available evidence.
A sparse extraction can be correct row by row and misleading as a whole. It can omit failed experiments, boundary conditions, or caveats that change how a result should be interpreted without inventing anything.
The cascade experiment shows how this compounds. Title and abstract screening recovered 97.6% of relevant records. After later screening, 33 papers reached extraction. Only four were fully correct across the evaluated fields, or 12.1% conditional complete-extraction accuracy.
We saw a related problem in SocSci-Repro-Bench: an agent could reproduce an analysis yet agree too readily with the paper's interpretation. Evidence work fails when a plausible final answer replaces a traceable path through the source.
A safer division of labor
A review team should measure screening recall and extraction completeness separately. The same model may be dependable for one and weak for the other.
- Write inclusion and exclusion rules before screening, then save a short researcher rationale with representative decisions.
- Optimize screening for recall. Report precision and the chosen threshold, and audit a random sample of model-excluded records.
- Evaluate extraction separately. Finding the right paper does not prove that a model can recover every relevant result inside it.
- Attach a page, section, table, or quoted evidence span to every extracted claim. Empty fields deserve as much review as populated ones.
- Run a second extraction pass or use human adjudication for results and limitations. Compare the evidence sets rather than only their summaries.
- Preserve the model identifier, prompt, schema, parser, raw output, judge configuration, and correction log.
The PRISMA 2020 statement provides checklists and flow diagrams for transparent review reporting. An AI-assisted workflow should add its model decisions and verification record to that trail rather than compressing them into "AI was used for screening."
ish.chat can help a team refine criteria and rationales, while api.ish.chat can run repeatable screening and extraction matrices. Keep the final evidence ledger outside the conversation: every included record, exclusion reason, extracted claim, source locator, and human correction should be exportable and reviewable.
What the benchmark does not prove
SciLitBench is a preprint built from one review protocol about literature-review automation. Its label audits measured repeat agreement by the same annotator and sampled contamination among silver exclusions, not independent validation of every field. The open-ended scores depend on a calibrated model judge, and the main text reports supported estimates for 78% of model-by-field-by-metric combinations.
The repository reproduces reported metrics from released predictions and annotations. Regenerating every model output requires separately obtained PDFs, model-serving infrastructure, and additional configuration. Frontier models were compared on four closed extraction fields, not the open-ended result and limitation fields that produced the headline gap.
The findings belong to this corpus and protocol. Here, screening saved labor because the team measured omissions. Extraction still looked polished while dropping most of the evidence the review set out to collect. Any review that automates extraction has to test what is missing, not only whether the visible rows are correct.
