[Claude Code] Systematic Review × AI Subagents Part 1: Search Query Construction and Subagent Roles

1. Introduction

Part 1 covers the first stage of an SR: search.

1-1. Purpose of This Step

The purpose of the search in an SR is to get every study that should be included into the candidate set without missing any. A paper the search fails to pick up cannot be recovered later in screening. So the search prioritizes few misses (Recall) even if the number of records grows. Still, the candidates are kept to an amount that the next step, screening, can read one by one (a few hundred to around a thousand records).

In this trial, I check the following two things in this step and produce the input for the next step.

	Content	How it is measured
Question	With no human intervention, how many of the studies included by the original review can a PubMed search query built by an agent from PICO alone pick up?	Search Recall = the share of the benchmark answers that appear in all search hits. The original paper reports 0.711 to 0.834
Input to the next step	Fix the candidates for Part 2’s screening in a reproducible form	Number of candidates, and search.json storing the PMID order

1-2. Comparison with the Original Paper

	Original paper (TrialMind)	This trial
Input	PICO	PICO (as given in TrialReviewBench)
How the search query is built	Builds a Boolean query from PICO, then adds or removes terms based on abstracts from a preliminary search	Subagent query-builder tests sub-queries through the PubMed connector (MCP) and revises them based on counts and query expansion
Database	PubMed	PubMed
Human involvement	—	None (the draft is used as is; run only once per review)
Metric	Search Recall	Search Recall (counted over all hits)

1-3. Steps and Summary of Results

Step	Content	Result
① Split PICO into two blocks	Use only P (population) and I (intervention); do not narrow by C or O	All 3 take the form P AND I (31190844 puts CD19 in a separate block)
② Test and revise the query	query-builder sends sub-queries through the PubMed connector and revises them based on counts and expansion	Finalized in 3 to 5 attempts per review
③ Fetch all hits and fix them	fetch_pubmed.py fetches citations and abstracts for all hits and saves the PMID order	683 / 698 / 484 records (1,865 in total)
④ Check against the answers	Count whether the 27 benchmark answers are among all hits	Search Recall 0.963 (26/27). Higher than the original paper’s 0.711 to 0.834, but the conditions differ, so the values are not comparable (Section 5)

2. Steps

2-1. ① From PICO to Two Blocks

PICO is a template that writes a question in four parts: population (P), intervention (I), comparison (C), and outcome (O). How to decide the PICO is covered in section 4.1 of the explainer page “[Practice Explainer] Systematic Review”, and how to build a search query (MeSH and free text, blocks and AND/OR) is in 4.2. The input for this trial is the PICO of the 3 reviews in TrialReviewBench, as is.

Review	P (population)	I (intervention)	C (comparison)	O (outcome)
33746596	Patients with relapsed/refractory multiple myeloma (RRMM)	CAR-T therapy	N/A	Response rate, complete response rate, MRD negativity rate, relapse rate, median progression-free survival, grade 3–4 CRS and neurotoxicity
31190844	Patients with diagnosed hematological malignancies	Autologous CD19 chimeric antigen receptor-T cell therapy	Standard care or alternative treatments	Survival outcomes and efficacy of the therapy
37168849	Patients with relapsed/refractory acute myeloid leukemia (RR-AML)	Chimeric antigen receptor T cell (CAR-T) therapy	N/A	Complete response rate, overall response rate, incidence of CRS, ICANS, and GVHD

For the two reviews with long O, I summarized the benchmark’s English text (the original text is in bench/reviews.jsonl).

query-builder splits the PICO into concept blocks. As a rule, only P (population) and I (intervention) become blocks. Narrowing by O (outcome) or study design drops papers whose abstracts don’t mention them, which lowers Recall. Within a block, MeSH terms and free-text terms ([tiab]) are joined with OR, and the blocks are joined with AND.

The final draft for the multiple myeloma review (PMID 33746596) was the following.

("Multiple Myeloma"[MeSH Terms] OR "multiple myeloma"[tiab] OR "myeloma"[tiab] OR RRMM[tiab])
AND
("Receptors, Chimeric Antigen"[MeSH Terms] OR "Immunotherapy, Adoptive"[MeSH Terms]
 OR "chimeric antigen receptor"[tiab] OR "CAR-T"[tiab] OR "CAR T"[tiab] OR "CAR-T cell"[tiab]
 OR CART[tiab] OR "CAR T-cell"[tiab] OR "CAR-modified"[tiab] OR "BCMA"[tiab]
 OR "idecabtagene"[tiab] OR "ciltacabtagene"[tiab])

The P in the PICO is “relapsed/refractory multiple myeloma”, but relapsed/refractory is not made into a block. Whether a study is relapsed/refractory is judged by the screening criteria.

2-2. ② Revising Based on Counts and Expansion

query-builder tests each sub-query through the connector, and revises it by looking at the returned count (total_count) and the query as PubMed actually interpreted it (query_translation). The review that needed the most revisions was the one on CD19-targeted CAR-T (PMID 31190844).

The first query joined CD19 with the CAR terms using OR, so it picked up papers that merely mentioned CD19, giving 4,483 records. Moving CD19 into a separate block joined with AND brought it down to 605, and after adding terms it was finalized at 698.

The final drafts for the 3 reviews are as follows.

Review	Blocks	Operators	Attempts	All hits
33746596 Multiple myeloma	P AND I	15	3	683 records
31190844 CD19 CAR-T	CAR AND CD19 AND hematological cancer	14	4	698 records
37168849 Acute myeloid leukemia	P AND I	12	5	484 records

The full queries and attempt logs are in the repository at reviews/<PMID>/eval-3/query.md. They are copies of query-builder’s replies without a single character changed.

2-3. ③ Fetching All Hits and Fixing Them

The finalized query is run via E-utilities by the script fetch_pubmed.py, which fetches the citation, abstract, and publication date of every hit. The PMID order of all hits, the query, and the fetch timestamp are saved in search.json and committed (the reason is in 4-3).

2-4. ④ Checking Against the Answers

Finally, I count whether the benchmark answers (PMIDs of the studies included by the original review) are among all hits. The answers were not passed to any of the steps up to this point.

3. Results

3-1. Search Recall

Search Recall is the share of the studies included by the original review (the benchmark answers) that appear in all search hits.

Review	Included	All hits	Search Recall	Missed studies
33746596	9	683	1.000 (9/9)	—
31190844	7	698	1.000 (7/7)	—
37168849	11	484	0.909 (10/11)	28864289
Total	27	1,865	0.963 (26/27)

The missed study, 28864289, is a phase 1 trial of activated NK cells derived from NK-92, and it has no CAR. It does not match the block of CAR terms, so it is the kind of miss that no revision of the search query would pick up. The original review’s own criterion is CAR-T, so I consider the benchmark answer itself to be outside the criteria (it comes up again in Part 2).

3-2. PRISMA Counts

	33746596	31190844	37168849
Identified by search (PubMed)	683	698	484
Duplicates	0	0	0
Missed answers added	0	0	1
To screening	683	698	485

The original paper ranks within “candidates that contain all included studies” during screening. To match that, for 37168849 only, I appended the one missed record to the end of the candidates. The screener was not told about the addition. The candidates for the 3 reviews total 1,866 records, and this is the input for Part 2.

4. How This Work Is Structured

4-1. Separating the Testing Step from the Fetching Step

I used two tools for the search: Anthropic’s official PubMed connector (MCP), and a script, fetch_pubmed.py, that calls E-utilities directly.

The connector is suited to trial and error. Counts and expansions come back right away, so the agent can proceed while revising the query. On the other hand, it is limited to 20 operators per query and 200 returned records, so it is not suited to fetching citations and abstracts for hundreds of records in bulk. So I split the work: the connector up to finalizing the query, and the script for fetching all hits with the finalized query.

Step	Owner	What it does	What it leaves
Test the query	Subagent query-builder	Tests and revises sub-queries	Queries tried, counts, expansions (query.md)
Fetch all hits	fetch_pubmed.py	esearch for PMIDs of all hits, efetch for citations and abstracts	Query, date range, fetch timestamp, PMID order (search.json)

Quirks of the connector I found while testing went into the agent definition. For example, passing only the upper date bound date_to does not narrow the results; you have to pass it together with date_from.

4-2. Restricting query-builder’s Tools

The top of the query-builder definition (.claude/agents/query-builder.md) is as follows.

name: query-builder
description: Builds a PubMed Boolean query from the review's PICO and the approved eligibility criteria,
  and tests and revises it sub-query by sub-query with the official PubMed connector. (omitted) Does not run the main search (fetch_pubmed.py)
tools: mcp__plugin_pubmed_PubMed__search_articles, mcp__plugin_pubmed_PubMed__get_article_metadata
model: sonnet

Of the connector’s 7 tools, it gets only 2: search and citation retrieval.

• No Read: it cannot read the file with the correct answers (benchmark answers). The main agent passes the PICO and the upper date bound in the delegation message
• Related-article and full-text tools removed: using find_related_articles and get_full_text_article would leak the answers through the original review’s references
• No Write: it returns the attempt log as its reply, and the main agent writes it to query.md

Each of the connector’s 7 tools is added by name, one by one, to permissions.allow in settings.json. With a wildcard, any tools later added to the plugin would be allowed automatically.

4-3. Fixing What Was Fetched

PubMed’s relevance order (Best Match) shifts slightly on every call. Refetching changes the order, and the later ranking evaluation becomes irreproducible. So fetch_pubmed.py writes the PMID order of all hits, the query, the query as PubMed interpreted it, and the fetch timestamp to search.json, and I committed it. Abstracts are copyrighted, so they are not committed. When you run the repository locally, only the abstracts are refetched from the fixed PMIDs (--from-search).

The script reads the API key from the repository’s .env, and Claude itself cannot open .env (permissions.deny). All requests are POST so the key never appears in a URL or an error message.

4-4. Recording the Counts

The counts at each step are tallied by the skill /prisma-record, which calls the script prisma_record.py, confirms that the sums add up, and then writes them to prisma.json. Claude never copies numbers by hand. If they are edited by hand, the PostToolUse hook check_prisma.py checks the same sums (identified − duplicates = screened + not screened, and so on) and notifies Claude if they don’t match.

4-5. What Humans Decided

What humans decided	Reason
Set the upper bound of the search period to the original review’s publication date	Keep papers the original review could not have seen out of the comparison with the answers
Run query-builder only once per review and use the final draft as is	Regenerating to pick a better draft would introduce human judgment
The main agent does not add conditions to the delegation message	Added conditions become indistinguishable from human instructions
Do not revise the search query after seeing the answers	Recall from a query revised to fit the answers is not an evaluation

The third came from something that actually happened. At the earlier stage where a human approved the search query, the delegation message contained the condition “P uses only multiple myeloma terms.” It was not a human instruction; the main agent had added it. In the flow for comparison with the original paper (eval-3), the delegation message is fixed to the same form for all 3 reviews, and only the <...> placeholders are substituted.

5. What Can and Cannot Be Said

What can be said

• With no human judgment, the agent’s queries picked up 26 of the 27 benchmark answers from PubMed
• The one miss was not due to missing terms in the query; it was because a study without a CAR was among the answers

What cannot be said

• It cannot be said that the search is better than the original paper’s. The original paper does not state how far it narrowed the candidates. This trial counted over all hits (hundreds of records). And there are only 3 reviews
• It cannot be said that the search was exhaustive. There are two limits on coverage. (a) The “search query limit,” papers in PubMed that the query does not hit, shows up in search Recall. (b) The “database limit,” papers not in PubMed, cannot be measured because the benchmark answers are a list of PMIDs. The 3 original reviews included 50 studies in total in their text, and 23 of them are not in the answers (in 33746596, 14 have no PMID, most of them conference abstracts)
• It cannot be said that it was built with the same procedure as the original paper. The agent definition says “pick up terms from the top abstracts of a preliminary search,” but in all 3 reviews query-builder did not read abstracts and built the queries from the PICO and common synonyms alone. It wrote this itself under “Unverified / points of concern”

The last point is also key when delegating to an agent. Writing a procedure into the definition does not guarantee it will be followed. Because I kept the attempt logs without changing a single character, I could find out afterward that it deviated from the procedure.

6. Summary

The purpose of this step was to get the studies that should be included into the candidates without missing any, and to check how far an agent alone can do that. In a flow with no human judgment, search Recall was 0.963 (26/27) across the 3 reviews, with 1,866 candidates.

In terms of structure, I split the search into a testing step (subagent query-builder + PubMed connector) and a fetching step (fetch_pubmed.py). The agent got only 2 tools, keeping it isolated from the answers; it recorded only the counts and expansions returned by the connector; and the fetched PMID order was fixed.

However, this is coverage within PubMed. This trial, which searches only PubMed, is not a substitute for the search in a real-world SR.

Next Steps

Part 2 covers screening. The 1,866 records collected in Part 1 are judged 1 / 0 / −1 per criterion, using the LLM-written eligibility criteria as is, and ranked by the total. I show what happened when a single main agent did the judging (context, quotes, and drift in where the line is drawn), the subagent screener split off in response, the hook that checks verbatim quotes, and the comparison of Recall@20 and @50 with the original paper.