[Technical Explainer] Literature Screening Automation: Lineage of Methods and Evaluation Metrics

1. Task Definition

Screening in an SR is binary classification in which positives (included studies) make up only a few percent (for the SR workflow, see [Practice Explainer] Systematic Review). Most automation research targets title/abstract screening, the stage with the most reading.

There are two types of automation. Classification outputs “keep / exclude” for each record, while ranking orders records by how likely they are to be included. In classification, a record excluded by mistake is simply missed; in ranking, where you stop reading determines both what is missed and how much work is done.

A single missed study can change an SR’s conclusion, while an extra record read only costs time. This asymmetry has shaped how evaluation metrics are designed (Section 3).

2. Lineage

Twenty years of work can be read as successive answers to one question: “How far can we get without gold-standard labels for this particular review?”

2-1. Supervised Classification

The starting point is Cohen et al. (2006). They trained a classifier (voting perceptron) on screening decisions from 15 SRs of drug classes; human reading was reduced in 11 of the 15 reviews, and by 50% or more in 3. The WSS metric was also proposed in this paper (Section 3).

The weakness is that each review needs its own labeled training data. When you start a new SR, those labels do not exist yet.

2-2. Active Learning

Active learning was the answer to this. Each time a human screens a few records, the model is retrained and the records to read next are moved to the top. Wallace et al. (2010), abstrackr (2012), and ASReview (2021) belong to this line.

If the ranking works, most included studies cluster early. The remaining problem is when to stop. In real use you cannot see the curve, so you cannot say for certain that “no included studies remain.”

A practical procedure is SAFE (Boetje & van de Schoot 2024). It has four phases: read records at random to create training labels (S), screen with active learning (A), re-rank with a different model to look for missed studies (F), and have another person check the excluded records (E). Counts such as “no included study in the last 50 records” for each phase are examples; the paper itself states they are not values that apply to every review.

Instead of heuristics, there are also statistical stopping criteria (Callaghan & Müller-Hansen 2020). You draw a random sample from the unscreened records, read it, test the hypothesis that “recall is below the target,” and stop once you can reject it. The advantage is that you can report the basis for stopping as a probability.

2-3. Specialized Classifiers and Pretrained Models

Classifiers that target a narrow type of decision are already used in practice. Cochrane’s RCT Classifier missed 224 (0.5%) of the 44,007 RCTs included in Cochrane Reviews (Thomas 2021).

Around the same time, the way documents are represented also changed.

Early on, abstracts were represented by word occurrences. Next came embeddings from pretrained models such as PubMedBERT (pretrained on PubMed abstracts) and MedCPT (trained on 255 million pairs from PubMed search logs). Because they measure semantic similarity between the question and the abstract, they can rank records even with few labels. TrialMind uses MedCPT as one of its baselines (2-5).

2-4. LLM Zero-Shot Screening

LLMs can read eligibility criteria as written text and make decisions without training data. Because the output can include a written rationale, not just a label, it became easier for humans to check the decisions. Results, however, vary widely across studies.

The main sources of variation are the prompt (whether the criteria are passed together or judged one at a time, and which way to lean when uncertain), class imbalance in the data, and how the gold standard was created. Unless these are aligned, numbers from different papers cannot be compared side by side.

2-5. Agentic Systems

The latest direction splits screening into multiple stages and has LLMs handle them.

TrialMind has the LLM write out eligibility criteria from the PICO, then judge each record one criterion at a time as −1 (ineligible) / 0 (unclear) / 1 (eligible), and ranks records by the sum. GPT-4 and Claude 3 Sonnet were used in the evaluation, and the Recall@k of its ranking was 1.5 to 2.6 times that of ranking with MPNet or MedCPT. In a human-AI collaboration trial, screening time was 44.2% shorter.

otto-SR runs everything from screening to data extraction in a single workflow and reports, in its abstract, a sensitivity of 96.7% and specificity of 97.9% (humans: 81.7% and 98.1%). The human values are from the full-text stage; stage-specific values are reported separately in the main text. However, it has not been peer reviewed, and the code is not public.

The difference between the two lies less in performance numbers than in whether they can be verified. When choosing a paper as the basis for an implementation, I first check three things: peer review, availability of code and data, and conflicts of interest. Authors of both papers have ties to companies, so I judge conflicts of interest by reading the disclosures.

3. Evaluation Metrics

Metric	What it measures
recall (sensitivity)	Proportion of records that should be included that were kept. Top priority
WSS@95%	Proportion of records humans did not need to read while keeping recall at 95% (Cohen 2006)
Recall@k	Proportion of included studies that appear in the top k records (TrialMind mainly uses k = 20 and 50)
kappa	Agreement between two raters (or a human and a model), corrected for chance agreement (Cohen 1960)

I do not use accuracy. If 2% of records are included, answering “exclude” for every record already gives 0.98.

The maximum possible WSS depends on the proportion of included records (about 0.45 for a 50/50 dataset, about 0.90 with 5% included; Kusa 2023). Do not compare WSS values across different datasets.

4. Benchmark Datasets and How to Read the Numbers

Public datasets for comparing methods have also become available. The CLEF TAR tasks (2017 to 2019) are close to real practice in design, because they had participants compete not only on ranking but also on where to stop.

When reading the numbers, I check the following three points.

• The gold standard is also human judgment: Even dual human screening misses studies. When a model picks up an included study that is not in the gold standard, it is counted as a false positive
• Stage: Is the gold standard from the abstract stage, or from after reading the full text? otto-SR also reports separate values for the abstract and full-text stages
• LLM memorization: Published SRs and their included studies may be in an LLM’s training data. For evaluation, it is safer to include SRs published after the model’s training cutoff

5. Position in Practice

The main tools include EPPI-Reviewer, Covidence, DistillerSR, Rayyan, Laser AI, and Elicit. Most offer prioritization and classifiers, and are adding LLM-based screening and extraction. Vendor claims such as “X% reduction in work time” have not been peer reviewed.

In 2025, four organizations (Cochrane, Campbell, JBI, and CEE) issued a joint position statement requiring that final responsibility rests with the authors and that any use of AI that affects judgments be fully reported. Tool developers follow the RAISE guidance.

6. Summary

• Screening is imbalanced binary classification. The cost of a missed study is overwhelmingly larger
• The lineage runs supervised classification → active learning → specialized classifiers and pretrained models → LLM zero-shot screening → agentic systems. Each is an answer to “how far can we get without gold-standard labels”
• In ranking, the stopping point is key. Use a procedure like SAFE or a statistical stopping criterion
• Recall comes first. Measure work saved with WSS@95% and ranking with Recall@k. Do not use accuracy
• When reading the numbers, check how the gold standard was made, the stage, and LLM memorization
• When choosing a paper, look first at peer review, code availability, and conflicts of interest

7. Next Steps

In the main series, I rebuild the TrialMind methods introduced in 2-5 (search, screening, and data extraction) with Claude Code, and compare the results with the original paper using the metrics in Section 3.