[Claude Code] Systematic Review × AI Subagents Part 2: Screening and the Subagent/Hook Split

1. Introduction

Part 2 covers the second stage of an SR: title/abstract screening.

1-1. Purpose of This Step

The purpose of title/abstract screening is to read the titles and abstracts of the candidates collected by the search and select the papers whose full text should be read. In a real-world SR, two people independently read every record. It is the stage with the most reading, so research on automation also concentrates here ([Technical Explainer] Literature Screening Automation).

Instead of splitting candidates into “read / don’t read,” the original paper’s TrialMind orders them by which should be read first. If included studies gather at the top, a person only has to read from the top down.

	Content	How it is measured
Question	With no human intervention, judging with the LLM-written criteria as is and ordering by total, how high do the included studies rank?	Recall@20 and @50 = the share of included studies that land in the top 20 and top 50. The original paper (Immunotherapy) reports 0.567 and 0.713
Input to the next step	Studies for Part 3’s data extraction	In this trial, the benchmark answer studies are used as is (the ranks in Section 3 are not used as extraction input)

1-2. Comparison with the Original Paper

	Original paper (TrialMind)	This trial
Input	2,000 candidates (containing all included studies)	Part 1’s 1,866 candidates (683 / 698 / 485; the 1 included study the search missed was added)
Eligibility criteria	Built by an LLM from PICO	Draft from skill /pico-to-criteria (not edited by a human)
Judgment	1/0/−1 per candidate per criterion	Subagent screener writes 1/0/−1 and a verbatim quote per criterion, 20 records at a time
Ranking	Sum of judgments	Sum of judgments (ties broken by PubMed relevance order)
Human involvement	—	None
Metric	Recall@20 and @50	Recall@20 and @50 (mean and pooled across the 3 reviews)

1-3. Steps and Summary of Results

Step	Content	Result
① Use the criteria as drafted	The inclusion and exclusion questions that /pico-to-criteria wrote from the PICO go into criteria.json without a single character changed	7 to 8 criteria per review. In 31190844, the comparison-group criterion I5 remained
② Split candidates into batches	A script writes files of 20 records each	95 batches (35 / 35 / 25)
③ The screener judges	Launched once per batch, it writes judgments and quotes. Right before writing, a hook checks the quotes	All 1,866 records passed the check. One rejection
④ Order by total and check against the answers	rules.py computes the sum of judgments, and eval_screening.py orders them and counts the ranks of the answer studies	Mean Recall@20 0.482, mean Recall@50 0.779. @50 is on par with the original paper; @20 is lower

2. Steps

2-1. ① Using the Criteria as Drafted

Eligibility criteria break the PICO down into “questions that can be answered from the title and abstract.” The skill /pico-to-criteria puts only one condition in each criterion and writes them separately as inclusion (I) and exclusion (E). The draft for the multiple myeloma review (PMID 33746596) had the following 7.

ID	Question (summary)
I1	Are the subjects patients with multiple myeloma?
I2	Is it relapsed/refractory?
I3	Did they receive CAR-T therapy?
I4	Does it report any of the outcomes listed in O?
E1	Is it a publication such as a review or editorial that reports no new patient data?
E2	Is it a preclinical study only in vitro or in animals?
E3	Is it only a trial protocol with no results?

The draft also had a section of “points for a human to decide.” In the flow for comparison with the original paper, that section was not passed to the screener, and only the questions in the table were used. Only points the draft had already settled (for example, case reports are not excluded under E1) were kept as notes on the criteria. The one with the largest impact was I5 “Is there a comparison group of standard care or another treatment?” in the CD19 CAR-T review (PMID 31190844). It is a criterion that came from the C in the PICO, and the draft itself said “a human decides whether to adopt it.” Since this is a flow where no human decides, I5 remained.

2-2. ② Splitting into Batches

The script make_batches.py writes the candidates into files of 20 records each. Each file contains the criteria, the candidates’ titles and abstracts, and the path to write to. The screener is instructed “read only this one file” (Read is not restricted by a hook). Once a batch has been judged, the script blocks rewriting it if its content would change.

2-3. ③ The Screener Judges

The main agent launches the screener with only the batch file path in the delegation message. For each candidate, the screener judges every criterion as 1 (meets) / 0 (can’t tell from the title and abstract) / −1 (does not meet), and attaches a verbatim supporting quote to each ±1. E (exclusion) runs in the opposite direction: if the record does not meet the exclusion, it is 1.

{"pmid": "12345678",
 "criteria": [
   {"id": "I1", "verdict": 1, "quote": "patients with relapsed or refractory B-cell lymphoma"},
   {"id": "I2", "verdict": 0, "quote": null, "note": "Target antigen not stated"},
   {"id": "E1", "verdict": 1, "quote": "We conducted a phase 1 trial"}]}

Anything not stated is 0. A 0 leans toward inclusion, so not guessing a ±1 prevents misses.

2-4. ④ Ordering by Total and Checking Against the Answers

The script rules.py sums the judgment values for each candidate into a score. The maximum is the number of criteria (7 or 8). The evaluation script eval_screening.py orders them by score from high to low, breaking ties by PubMed relevance order. Finally, I count whether the benchmark answer studies landed in the top 20 and top 50. The answers were not passed to any of the steps up to this point.

3. Results

3-1. Recall@20 and @50

Review	Candidates	Included	Recall@20	Recall@50
31190844 CD19 CAR-T	698	7	0.143 (1/7)	0.429 (3/7)
33746596 Multiple myeloma	683	9	0.667 (6/9)	1.000 (9/9)
37168849 Acute myeloid leukemia	485	11	0.636 (7/11)	0.909 (10/11)
Mean of 3 reviews			0.482	0.779
Pooled across 3 reviews		27	0.519 (14/27)	0.815 (22/27)
Original paper (arXiv version)	2,000		0.567	0.713

The text of the original paper doesn’t make clear whether its values are means or pooled, so I show both. Recall@50 is on par with the original paper, and Recall@20 is somewhat lower. With only 3 reviews, the single review 31190844 pulls the mean down substantially.

3-2. Ranks of Included Studies

Every lowered rank was caused by a −1 judgment.

Review	Studies with lowered rank	Criterion with −1	Reason
31190844	4 (ranks 39, 58, 59, 114)	I5 (comparison group)	Single-arm trials lost 1 point on I5. 110 of the 698 candidates had I5 = −1
31190844	1 (rank 97)	I3 (autologous)	Due to what the abstract states
37168849	2 (ranks 33, 36)	I3 (CAR-T)	CAR-NK, and NK cells without a CAR. The answers are outside the original review’s criterion (CAR-T)
37168849	1 (rank 290)	I3, I4, E2	Patient data is only in the full text; from the abstract it looks preclinical

• 31190844: No candidate reached the maximum of 8, and 15 had the top score of 7. Of the 7 included studies, 4 dropped to 6 due to I5 and were buried among the 45 records with a score of 6. Re-summing the same judgments without I5 gives Recall@50 0.429 → 0.714 (for reference only; not used in the comparison with the original paper)
• 33746596: There was not a single −1. The 3 studies outside @20 were determined by the tie order among the 21 records with the maximum score and the order of the records scoring 6
• 37168849: The 2 studies at ranks 33 and 36 (rank 36 is 28864289, which Part 1’s search missed) are discrepancies on the answers’ side. The one at rank 290 has patient data only in the full text, and from the abstract alone it looked like a preclinical study

4. How This Work Is Structured

4-1. What Happened When a Single Main Agent Judged

Before building the subagent, I had the main agent (the main Claude Code conversation) alone judge 101 candidates from 31190844 (tag snap/06-no-subagent). Among the 101, one record from the first half was inserted again in the second half under a different number.

What happened	Details
Context overflows	Each judgment added about 1.5k tokens. Doing 3 reviews × 200 records in one conversation would take about 900k, close to the 1M limit
Judgments are not independent	For the study inserted twice, the main agent noticed “this is the same abstract as before.” The second judgment is made having seen the first
Quotes are not verbatim	16 of 101 records. Of these, 4 shortened or paraphrased words from the abstract
The line drifts	How to treat commentaries not explicitly labeled as reviews under E1 differed between news and commentary pieces

From this, I decided on three things: split off the screener, redefine the rules, and check quotes mechanically. The scope of E1 (“documents that contain no patient data of their own and introduce or comment on other studies”) was added to the rules in response to the drift in this trial run.

4-2. The Screener Subagent and Batches

The top of the screener definition (.claude/agents/screener-a.md) is as follows.

name: screener-a
description: Judge for SLR title/abstract screening (omitted). Each launch judges one batch (20 records)
  with 1/0/-1 and a verbatim quote per criterion, and writes JSON to results/screen/a/. (omitted)
tools: Read, Write
model: sonnet
omitClaudeMd: true
skills:
  - screening-rules

• 20 records per launch: each launch starts with a fresh context, so judgments from the previous batch don’t carry over. One trial-run launch took about 1.5 minutes and about 36k tokens
• Only Read and Write as tools: it can neither search nor execute
• omitClaudeMd: true: it does not read the main agent’s CLAUDE.md (which describes the evaluation procedure and where the answers are kept)
• Writes neither an overall verdict nor a score: the script computes the total

The 95 batches were run several at a time in parallel.

4-3. Preloading the Judgment Rules with a Skill

The judgment rules are collected in the skill screening-rules and loaded when the screener launches, via skills in the frontmatter. It covers the meaning of the judgment values, the direction of E, the scope of E1, the quoting rules (copy one contiguous span as is; no ellipses, paraphrasing, or stitching), and the output format.

The body of the agent definition states “follow only the rules,” “read only the one batch file,” where to write, how to fix things when rejected, and what not to do; the judgment rules themselves live in the skill. To change a rule, I only need to edit one place in the skill.

4-4. The Hook That Checks Quotes

Right before the screener Writes its output, the PreToolUse hook check_screen_output.py runs.

What it checks	When it fails
Every record in the batch is present, with no extra PMIDs	Stops the Write with exit 2
Each candidate has every criterion exactly once, in ID order, with a value of −1/0/1	Same
Quotes for ±1 appear verbatim in the title or abstract (matched after Unicode normalization)	Same
Quotes for 0 are null, and there is no overall verdict or score	Same

When a PreToolUse hook returns exit 2, the tool call is stopped and the hook’s error message is returned to the screener. The screener fixes only the PMIDs and criteria named in the error and Writes again. Across all judgments, there was one rejection (one non-verbatim quote).

In my first design, I planned to check and reject when the subagent finished (SubagentStop). Rereading the official docs, SubagentStop cannot block with exit 2 (the subagent has already finished). So I moved the rejection to PreToolUse and left SubagentStop with only the role of rechecking the written file and notifying the main agent.

4-5. What Humans Decided

What humans decided	Reason
Use the criteria as drafted by /pico-to-criteria, without fixing I5 either	The original paper also evaluates with the LLM-written criteria as is
Do not pass “points for a human to decide” to the screener	Passing them as open questions would have the screener decide in place of a human
A 0 leans toward inclusion	In an SR, a miss cannot be recovered
Judge each batch only once; relaunch only when the output could not be written due to an error	Rerunning to pick better judgments would introduce human judgment
Do not change the criteria, rules, or judgments after seeing the answers	Recall from changes made to fit the answers is not an evaluation

5. What Can and Cannot Be Said

What can be said

• With no human judgment, judging with the LLM-written criteria as is and ordering by total, mean Recall@50 across the 3 reviews was 0.779, on par with the original paper’s 0.713
• Of the 8 −1 judgments that lowered ranks, 6 could be explained by the draft criterion (I5) or by a discrepancy between the answers and the original review’s criteria. The remaining 2 were judgments consistent with what the abstracts state
• All 1,866 judgments passed the verbatim-quote check

What cannot be said

• It cannot be said that performance matches the original paper. With fewer candidates (485 to 698 vs. 2,000), it is easier to land near the top. There are only 3 reviews, and the models differ (the original paper used GPT-4 and Claude 3 Sonnet; this trial used the current Claude Sonnet)
• It cannot be said that the quotes support the judgments. The hook only confirms that a quote appears verbatim in the abstract. It does not check whether that quote leads to that judgment
• It cannot be said that I5’s judgments went as expected. In the 20-record trial run, I5 was 0 for all, and I expected only abstracts that explicitly state single-arm to get −1. Across all records, 110 of 698 got −1, including 4 of the 7 included studies. Which quotes led to −1 is not verified in this article
• It cannot be said that the subagent judges more correctly than a single main agent. The trial run was 101 records with the main agent’s model (Opus), and the real run was all records with Sonnet, so conditions don’t match. 4-1 is a record of “what happened,” not a comparison
• Nor can I say outright that the evaluation followed the procedure. The repository’s rules say only a human launches the evaluation skill /eval, but the eval-3 evaluation was run by a separate session following a human’s instruction. The results, criteria, and search queries were not changed

6. Summary

The purpose of this step was to gather the papers whose full text should be read at the top, and to check how far that can be done without human hands. Judging with the LLM-written criteria as is and ordering them, Recall@50 averaged 0.779 across the 3 reviews (original paper: 0.713), and Recall@20 was 0.482 (original paper: 0.567).

The one low review can be explained by the comparison-group criterion I5 remaining as is. If a draft of the criteria still contains a question a human should decide, it lowers the ranks directly. Removing human judgment showed where human judgment belongs.

In terms of structure, in response to the carryover and broken quotes seen when a single main agent judged, I split judging into the subagent screener at 20 records each, preloaded the rules with a skill, and checked quotes with a PreToolUse hook.

Next Steps

Part 3 covers data extraction. From the full text of the included studies (only those readable on PMC), the subagent extractor pulls out study characteristics such as number of patients, age, and target antigen, and I compare them with the original paper using the same Accuracy metric. I also show how the full text is passed in and why it is not committed.