[Claude Code] Systematic Review × AI Subagents Part 0: The Original Paper and This Trial's Design

1. Introduction

A systematic review (SR) is the work of collecting literature exhaustively by a predefined procedure, selecting it against criteria, and synthesizing the results. Of all the steps, title/abstract screening, which sorts hundreds to thousands of candidates by title and abstract, takes the most human effort. I summarized the procedure and quality standards in the explainer page “[Practice Explainer] Systematic Review”, and the lineage of automation techniques in “[Technical Explainer] Literature Screening Automation”.

In the second project of this series, I take up TrialMind, a paper that carried out SR work with LLMs, and rebuild its methods with Claude Code. I measure the results with the same yardstick as the original paper to see how far they match and where they differ.

The first project was about “having an agent build an app.” The second project is about “handing a work procedure to an agent and measuring how well it does with the same metrics as the original paper.” In Part 0, I lay out what the original paper contains, and this trial’s question, conditions, and flow.

Of the SR steps, this trial covers the following three.

Step	Input	Output	This trial
Search	PICO	Search query and candidate papers (hundreds of records)	Yes (Part 1)
Title/abstract screening	Titles and abstracts of all candidates	Decision on “should this be read” and a ranking	Yes (Part 2)
Full-text screening	Full text of the remaining papers	Final decision on inclusion	No (not in the original paper either)
Data extraction	Full text (including tables) of only the included papers	A table of values such as number of patients, age, target antigen, and dose	Yes (Part 3)
Synthesis	Extracted values	Meta-analysis, forest plot	No (the original paper has it; it amounts to a clinical conclusion, so I do not cover it)

Title/abstract screening decides “which papers to read,” and its metrics are Recall@20 and @50. Data extraction is “copying values from the papers chosen for reading,” and its metric is Accuracy. The answer papers, the data used, and the metrics are all different. In other words, this trial rebuilds the three tasks of the original paper (search, screening, extraction), stopping just before synthesis.

The code is public on GitHub (MIT). You can view the results on the sample pages in three tabs: overview, paper list, and extraction (a public version with abstracts and quotes hidden).

2. The Original Paper: TrialMind

2-1. Overview of the Paper

This paper was published in npj Digital Medicine in 2025 by a group including the US NIH’s National Library of Medicine (NLM) (Wang et al., 2025;8:509). It had LLMs handle three SR tasks, search, screening, and data extraction, and evaluated each on public benchmarks.

Task	Method in the original paper	Metric	Value in the original paper
Search	Build a Boolean query from the PICO, add or remove terms with reference to abstracts from a preliminary search, and search PubMed	Recall (how many included studies were retrieved)	0.711 to 0.834 (human-built search queries: 0.138 to 0.232)
Screening	Write out eligibility criteria from the PICO, judge each candidate one criterion at a time as −1 (ineligible) / 0 (unclear) / 1 (eligible), and rank by the sum	Recall@20 and Recall@50 (what fraction of included studies appear in the top 20 and 50 records)	1.5 to 2.6 times embedding-based ranking
Data extraction	Extract study characteristics (design, population, intervention, etc.) and results from the full text	Accuracy (scored by humans against the tables in the original reviews)	0.72 to 0.83 for study characteristics

The values are ranges across four topics (immunotherapy, radiation/chemotherapy, hormone therapy, and hyperthermia). The models used in the evaluation were GPT-4 and Claude 3 Sonnet.

The evaluation dataset, TrialReviewBench, consists of 100 SRs on cancer treatment and their 2,220 included studies. The gold standard is “the PubMed IDs (PMIDs) of the studies the original SR included.”

2-2. Limitations of the Original Paper

The original paper itself states the following limitations.

• Search was PubMed only. Full texts were limited to those available in PubMed Central (PMC)
• Topics were limited to cancer treatment; whether the results apply to prevention or diagnostic studies is unknown
• In the screening evaluation, ranking was done within 2,000 candidates per SR that contained all included studies

The last point matters when reading the numbers. A real search can miss included studies, but the original paper separated screening from search and measured ranking ability with all included studies present among the candidates.

2-3. Why TrialMind Rather Than otto-SR

A closely related study is otto-SR (Cao et al., medRxiv 2025), which runs everything from screening to data extraction in a single workflow. Its reported sensitivity and specificity are high, but it has not been peer reviewed, and the code is not public.

TrialMind has been peer reviewed, and its code (MIT) and evaluation data (Apache-2.0) are public. Because I can measure the same metrics on the same data, I can put my rebuilt results next to the original paper’s. As the basis for a re-implementation, I prioritized being verifiable over how high the numbers are. Authors of both papers have ties to companies, so conflicts of interest were not a reason for the choice. A detailed comparison is in 2-5 of the technical explainer page.

3. This Trial’s Question and Positioning

3-1. Question

If I rebuild the original paper’s methods with Claude Code and measure them with the same metrics, do they reach the same level as the original paper?

I follow the original paper’s three tasks (search, screening, and data extraction). I do not cover synthesis of results (meta-analysis), because it amounts to a clinical conclusion.

3-2. Positioning

This workflow is teaching material for learning SR methods, and a tool for checking a paper’s methods yourself. I do not use it as a substitute for title/abstract screening in real work. A real SR also searches Embase, Cochrane CENTRAL, clinical trial registries, and conference abstracts, but this trial, like the original paper, searches only PubMed.

For the same reason, this series avoids the following phrasings.

• “Replication” or “successfully reproduced”: the data retrieval date, the way candidates were collected, and the model differ from the original paper
• “Accuracy was proven”: it is just one run on 3 reviews

4. Differences from the Original Paper

Rebuilding inevitably changes conditions from the original paper. I list them up front.

Item	Original paper	This trial
Target	100 SRs on cancer treatment	3 SRs on CAR-T for hematologic cancers, from the immunotherapy set in TrialReviewBench
Screening candidates	2,000 records containing all included studies	Search hits plus included studies the search failed to retrieve (485 to 698 records; 1,866 records across the 3 reviews)
Model	GPT-4, Claude 3 Sonnet	Claude Sonnet (Claude Code subagents)
How decisions are made	−1 / 0 / 1 per criterion, ranked by the sum	The same. In addition, a verbatim quote from the abstract is required for each decision and checked by a hook
Human involvement	Designed so humans can edit the criteria	In the flow compared against the original paper, the criteria are used as drafted by the LLM, and humans do not take part in decisions
Full text	Those available in PMC	The same (only studies whose full text can be retrieved from PMC)
Scoring of extraction	Manual checking by 3 annotators	Exact matches are marked correct by rule; everything else is scored by a human one by one

The difference in the number of candidates directly affects how to read Recall@k. The fewer the candidates, the more easily included studies land in the top 20 and 50. Following the original paper, this trial also adds included studies the search failed to retrieve to the candidates, but the difference in numbers remains. Comparisons with the original paper are made with this difference noted (Part 2).

5. Target Reviews

PMID	Topic	Included studies (gold standard)
33746596	Efficacy and safety of CAR-T therapy for relapsed/refractory multiple myeloma	9 studies
31190844	Survival and efficacy of autologous CD19 CAR-T therapy for hematologic cancers	7 studies
37168849	Outcomes of CAR-T therapy for relapsed/refractory acute myeloid leukemia	11 studies

The 3 reviews have closely related topics and few included studies, so I can check the contents one by one. The upper limit of the search period was set to each original review’s publication date. This keeps papers the original review could not have seen out of the comparison against the gold standard.

6. This Trial’s Flow

6-1. Overall Flow

The animation at the top runs through this flow once. Stage by stage, steps ① to ⑥ and the record-keeping in the table are as follows.

Stage	Handled by	What it does
① Eligibility criteria	skill /pico-to-criteria	Drafts inclusion criteria (I) and exclusion criteria (E) from the PICO
② Search query	subagent query-builder	Tests partial queries with Anthropic’s official PubMed connector (MCP) and builds one search query
③ Candidate retrieval	script fetch_pubmed.py	Retrieves bibliographic data and abstracts for all hits of the finalized search query from E-utilities in bulk
④ Screening	subagent screener + hook	In batches of 20, writes 1 / 0 / −1 and a verbatim quote for each criterion. A hook checks whether the quote actually exists in the abstract and sends it back if not
⑤ Ranking and evaluation	script, skill /eval	Sums the decisions to rank records, and computes search Recall and Recall@20 and @50
⑥ Data extraction	subagent (extractor)	Extracts study characteristics from PMC full texts and computes Accuracy (Part 3)
Record-keeping	skill /prisma-record + hook	Records the counts at each stage in PRISMA 2020 form, and a hook checks that the counts add up

6-2. The Subagents

Three subagents handle the work. The main session (the main Claude Code conversation) advances through the stages, runs scripts, splits candidates into batches, and launches subagents. The main session itself does not make screening decisions or extractions.

subagent	Task	Input	Available tools	Launch unit
query-builder	Builds a search query from the PICO and revises it by checking counts for each partial query	PICO, eligibility criteria, upper limit of the search period	Only the two PubMed connector tools (search, bibliographic retrieval)	Once per review
screener (definition file screener-a.md)	For each candidate, writes 1 / 0 / −1 per criterion and a verbatim quote from the abstract	One batch file (criteria and 20 candidates)	Only Read and Write	Every 20 records; 95 batches in this trial
extractor	Extracts study characteristics item by item from the full text (built in Part 3)	One full text and the extraction items	Only Read and Write	Once per study

6-3. Why Split into Subagents

With hundreds of candidates, screening does not fit in a single conversation. Each decision uses about 1.5k tokens, so even a single review overflows the context.

Also, SR decisions should be independent for each record. If decisions are made one after another in the same conversation, decisions on studies seen earlier, and a threshold that shifted along the way, carry over into later decisions. Each subagent launch starts with a fresh context, which cuts off this carry-over.

For that reason, I narrowed what each subagent receives.

• No gold standard: no subagent can read the original reviews’ included studies
• No documents meant for the main session: screener and extractor do not read the main session’s CLAUDE.md (omitClaudeMd: true)
• Restricted tools: query-builder can search PubMed but cannot read files. screener can read and write files but cannot search
• Rules passed as a skill: the screening rules screening-rules are loaded when screener is launched (skills preload)

6-4. What Humans Decided

What I handed to the agents was the work at each stage. What humans decided was the experimental conditions and where to stop.

What humans decided	Reason
The 3 target reviews and the upper limit of the search period	To align the ground for comparison against the gold standard
“Unclear (0)” leans toward inclusion	In an SR, a missed study cannot be recovered
In the flow compared against the original paper, humans revise neither criteria nor decisions	The original paper also evaluated with the criteria as written by the LLM
After seeing the answers, do not change the search query, criteria, or rules	Revising to fit the answers makes the evaluation meaningless
Only a human launches the full evaluation run	A human takes responsibility for what was measured and when

The last two are never left to the agent’s judgment. Only humans know the answers, and once you start revising the method based on those answers, you break the very thing you are trying to verify.

7. Series Structure

The series follows the three tasks of the original paper. Each part opens with an “in the original paper / in this trial” contrast and closes by showing how that stage was built (subagents, skills, hooks, MCP).

Part	Title	Contrast with the original paper	How it is built
0	The original paper and this trial’s design (this article)	Contents of the original paper, the question, differences in conditions	Overall flow, what humans decided
1	Search query construction and subagent roles	How search queries are built, search Recall, limits of comprehensiveness	Division of labor between the PubMed connector (MCP) and the bulk retrieval script
2	Screening	Per-criterion decisions, Recall@20 and @50	subagent design, the hook that checks verbatim quotes
3	Data extraction	Accuracy for study characteristics, studies without available full text	extractor, handling full texts
4	Discussion and reproduction	What can and cannot be said, use as teaching material	/eval, steps to reproduce with the repository

8. Summary

In the second project, I rebuild TrialMind, which used LLMs for SR search, screening, and data extraction, with Claude Code subagents. The aim is to measure with the same metrics as the original paper and see whether the results reach the same level. I do not use it as a substitute for real work; I treat it as a trial for learning and verifying the methods.

I listed the differences in conditions (how candidates were collected, the number of reviews, the model) up front. Fixing the conditions before comparing, and deciding not to change them after seeing the answers, is what makes the results of work handed to an agent measurable. What you can hand over is determined by how verifiable you made it in advance.

Next Steps

Part 1 covers search query construction and subagent roles. I put the procedure for building a search query from the PICO next to the original paper’s, and show the proportion of included studies the search retrieved (Recall). I also describe how I split the stage that tests queries with the PubMed connector from the stage that retrieves records in bulk with the finalized query, and the limits of searching only PubMed (limits of the search query and limits of the database).