[Claude Code] Systematic Review × AI Subagents Part 3: Data Extraction and Full-Text Handoff

1. Introduction

Part 3 covers the third stage of an SR: data extraction.

1-1. Purpose of This Step

The purpose of data extraction is to read the full text of the included studies and copy values such as number of patients, age, and target antigen into a table with fixed items. This corresponds to Table 1 (the table of study characteristics) in an SR paper. Where title/abstract screening decided “which papers to read,” this step “copies values from the papers we decided to read,” so both the papers read and the metric are different ([Practice Explainer] Systematic Review).

	Content	How it is measured
Question	Passing only the column names of the answer table as items and writing out values from the PMC full text with no human intervention, how well do they match the answers?	Accuracy = correct items ÷ all items. The original paper (Immunotherapy) reports 0.78. By type: study design 0.95, patient characteristics 0.74
Input to the next step	Normally synthesis such as meta-analysis	In this trial, no synthesis is done (it amounts to a clinical conclusion). Part 4 summarizes the comparison with the original paper

The items in this trial are the columns of Table 1 in the original reviews, mostly patient characteristics with some study design. The original paper’s 0.78 includes outcome items, so the closest comparison is the patient characteristics value of 0.74 (design columns are mixed in, so it is not strictly the same yardstick).

1-2. Comparison with the Original Paper

	Original paper (TrialMind)	This trial
Target	Included studies	Of the included studies in the 2 reviews with answers (33746596 multiple myeloma, 37168849 acute myeloid leukemia), the 7 pairs whose body text was available on PMC
Input	Full-text PDF or XML	Text built from the PMC XML (body, all tables, figure captions; no supplementary materials or author affiliations)
Items	Column names of the answer table as is	Same (no descriptions added)
Extraction	A value and source per item	Subagent extractor, once per pair, writes a value and verbatim quotes per item
Scoring	3 people checked against the original tables	Only exact matches are marked correct by rule; the rest are scored by a human (1 person) on screen
Denominator	1,334 (design 696, patient characteristics 353, outcomes 285)	102 (design and patient characteristics)

1-3. Steps and Summary of Results

Step	Content	Result
① Turn the full text into text	fetch_pmc.py fetches the PMC XML, and fulltext_to_text.py turns the body and tables into text	Of the 20 answer pairs, body text was available for 7 (6 papers). Of 10 tables, only 4 were inside <body>
② Build jobs	make_extraction_jobs.py puts the full-text path and item names for each pair into one file	7 pairs, 102 items. The answer values are not passed
③ The extractor writes	Launched once per pair, it writes values and verbatim quotes. Right before writing, a hook checks the quotes	All 7 pairs were written. One rejection; 17 items “not stated”
④ Score	Exact matches are correct by rule; a human scores the remaining 96 items	Accuracy 0.735 (75/102, 95% CI 0.642–0.811). On par with the original paper’s patient characteristics value of 0.74

2. Steps

2-1. ① Turning the Full Text into Text

The limitations section of the original paper says it is “limited to publicly available materials in PubMed Central.” This trial follows that and takes full text only from PMC. The script fetch_pmc.py follows the PMC record from the PMID, fetches the XML, and records whether it has a body (<body>).

What I found when converting to text was where the tables live. In the PMC XML, many tables were placed outside <body> (in <floats-group> and elsewhere). Of the 10 tables in the 6 papers, only 4 were inside <body>. Passing only the body drops the patient characteristics tables entirely.

fulltext_to_text.py includes the title, abstract, and each section of the body, plus all tables (wherever they are; one row per line, cells tab-separated) and figure labels and captions. References and supplementary materials are not included. On the finer points, 50×10<sup>6</sup> is written as 50×10^6 (otherwise the dose reads as “106”), and long paragraphs are split at sentence boundaries into lines of 1,000 characters or fewer (5 paragraphs exceeded 2,000 characters). This is because when the Read tool truncates a long line, the rest never reaches the extractor.

2-2. ② Building Jobs

The script make_extraction_jobs.py writes a job file for each pair (review × study). It contains the path to the full text, the list of item names, and where to write. The item names are the column names of the answer table as is, for example strings like these.

• Median age (range), Lines of Prior treatment (33746596, 14 items)
• Prior HCT, Manufacturing time in days, Costimulatory Domain (37168849, 15 items)

The original paper also writes “Each table’s column names served as input field descriptions” and adds no descriptions to the column names. The script reads only the PMID column from the answer file, never the values. The same paper, 30396908, is in both reviews, so it becomes 2 jobs with different items.

2-3. ③ The Extractor Writes

The main agent launches the extractor with only these 2 lines (the same form for all 7 pairs).

Please process one extraction job.

job: results/extraction/jobs/<review>/<pmid>.json

The extractor reads the job and the full text and writes, for each item, a value (value) and verbatim quotes from the full text (quotes). If nothing is found, it sets the value to “not stated” and leaves the quotes empty.

{"review_pmid": "<review>", "pmid": "<pmid>",
 "items": [
   {"name": "<item name as is>", "value": "<value>", "quotes": ["<string from the full text>"]},
   {"name": "<item name as is>", "value": "not stated", "quotes": []}]}

After a trial run on 1 pair (16.7 seconds, 21k tokens, 0 rejections), I launched the remaining 6 pairs in parallel. Each launch took 16.9 to 30.3 seconds, about 177k tokens for the 7 pairs.

2-4. ④ Scoring

Items that matched exactly after normalizing only leading/trailing whitespace and case (6 items) were marked correct by rule. The remaining 96 items, including “not stated,” were scored one by one by a human on the eval-3 report screen. The criteria were set before scoring and shown at the top of the screen.

1. Correct if the meaning is the same (notation, how units are written, and word order don’t matter)
2. Correct if it is merely more detailed than the answer. Incorrect if part of the answer is missing
3. If the answer is a category (such as region) and the extraction is the underlying value, correct if the category is uniquely determined from it
4. “Not stated” is correct if the answer also indicates nothing is reported; incorrect if the answer has a value

The screen showed each quote together with the 160 characters before and after it in the full text. This corresponds to the original paper’s “outputs are linked to their sources so that humans can verify them.”

3. Results

3-1. Accuracy

Scope	Correct / Items	Accuracy	95% CI
Overall	75 / 102	0.735	0.642–0.811
33746596 Multiple myeloma	33 / 42	0.786	0.641–0.883
37168849 Acute myeloid leukemia	42 / 60	0.700	0.575–0.801
Reference: excluding 37168849 × 33495835	66 / 87	0.759	0.659–0.836
Original paper Immunotherapy (overall)		0.78	0.75–0.81
Original paper Immunotherapy (patient characteristics)		0.74	0.67–0.80

The 95% CI is the Wilson interval (the original paper doesn’t say how its intervals were computed). Per pair, values ranged from 0.600 to 0.867. The interval overlaps with the original paper’s, but the denominator is small at 102, so it is fairly wide.

The results can be viewed in the “Extraction” tab of the sample screen (a public version with quotes and surrounding text hidden).

3-2. Items It Got Wrong

Of the 27 incorrect items, 11 were “not stated” and 16 were mismatched values. There were 17 “not stated” items in total, but for 6 of them the answer was also “not reported” or similar, so they were marked correct. The reasons for the 11 incorrect ones are as follows.

Reason	Count	Example
Requires calculation	3	The table has only per-patient values, or only the number of women, so the median or the number of men requires calculation (the rules forbid calculation)
The answer’s “No” is not written in the text	3	Prior HCT, Post CAR-T Allogenic-HCT
Apparently in the supplementary materials	2	Manufacturing days, whether a transplant was done (not verified)
Author affiliations are not included in the text	2	Country/region (Country, Location)
Only inside a figure image	1	Costimulatory domain shown in a schematic of the CAR structure

The original paper also did not use supplementary materials or figure images as input, and its limitations section also says “values in the appendix could not be extracted.” Author affiliations are included in a PDF, so that gap comes from how I built the input.

3-3. What Could Not Be Extracted

To begin with, of the 20 answer pairs, body text was available for only 7.

	33746596	37168849	Total
Answer pairs	9	11	20
Body text available (extracted)	3	4	7
On PMC but no body text	5	4	9
Not on PMC	1	3	4

The 13 pairs without body text are not in the denominator. The original paper also takes PDFs as input, so these 13 pairs would be covered there. Accuracy 0.735 is “a value limited to studies whose body text can be read on PMC.”

3-4. Suspected Flaws in the Answers

37168849 × 33495835 is an in vitro study, and no patients received CAR-T. The extractor reported this too and marked the patient characteristics items “not stated.” Yet the answers contain patient age and sex, and the Costimulatory Domain field contains a drug name. I scored without fixing the answers, and listed 0.759, excluding this pair (15 items, 9 correct), for reference.

4. How This Work Is Structured

4-1. The Extractor Subagent

The extractor definition (.claude/agents/extractor.md) is written the same way as Part 2’s screener.

name: extractor
description: Extractor of study characteristics for the SLR. Each launch handles one study (one job), (omitted)
tools: Read, Write
model: sonnet
omitClaudeMd: true
skills:
  - extraction-rules

• One pair per launch: full texts are long, so each study is read in a fresh context
• Only Read and Write as tools, and it does not read the main agent’s CLAUDE.md
• What not to do is written in the body: guessing, calculating, or converting; rewording or reordering item names; clinical interpretation

4-2. The Rules Skill and Not Passing the Answers

The rules are collected in the skill extraction-rules and preloaded. It covers how items are handled, how values are written (no calculation or conversion), the quoting rules (strings that appear in the full text as is; for tables, the cell strings), the conditions for “not stated,” and the output format. The output examples use only placeholders like <value>; no answer values or examples of how answers are written appear in the skill, the agent definition, or the delegation message.

The skill has only user-invocable: false. disable-model-invocation: true, which would also stop the model from invoking it, also stops preloading into a subagent according to the official docs. Since the skill contains no answers, nothing leaks even if the model invokes it.

4-3. The 2 Hooks

hook	When	What it does
check_extract_output.py	Right before Write (PreToolUse)	Checks that the write destination is the job’s output, that review_pmid and pmid match the path, that the items match the job exactly and in the same order, that no value is empty, that “not stated” has empty quotes and anything else has at least one, and that each quote appears verbatim in the full text. If not, it rejects with exit 2 (SubagentStop only inspects)
limit_reads.py	Right before Read (PreToolUse)	Binds the first job an extractor reads to its agent_id, and blocks reading other jobs and any file other than that job’s full text

limit_reads.py needs the binding mechanism because the hook’s input doesn’t say “which job this extractor was given.” The agent type (agent_type) and ID are available, so I decided to fix it on the first read.

There was one rejection in total. It was for 33495835, where values had been written for age, number of men, and number of evaluable patients without quotes. The extractor changed 2 items to “not stated,” added a quote to 1 item, and passed on the second attempt. Values without quotes were stopped here.

4-4. Not Committing the Full Text

The repository contains only the scripts that fetch the full text and turn it into text; the XML, the text, and the extraction outputs are written to results/ (under .gitignore). The README also says “the answer CSVs, abstracts, and full texts are not in the repository (fetch them with the scripts).” The reason is the same as for not committing abstracts: the full text of a paper is a copyrighted work of its authors or publisher. Even if it can be read on PMC, whether it may be redistributed depends on each paper’s license. Readers can fetch it again with the same procedure by running fetch_pmc.py and fulltext_to_text.py.

4-5. What Humans Decided

What humans decided	Reason
Full text from PMC only. Pairs without body text are not extracted	Matches the conditions in the original paper’s limitations
Include tables outside <body> too	The original paper feeds in the full-text PDF or XML as is
Keep items as the answer column names, with no descriptions added	Same as the original paper
No human judgment in the extraction flow (the human only checks the trial run and scores)	Accuracy of values a human corrected would not evaluate extraction
Set the scoring criteria before scoring and don’t change them midway	To avoid moving the criteria while looking at the answers
Don’t fix suspected flaws in the answers; list the value excluding them for reference	Fixing the answers changes the yardstick being compared against
Extraction is the last place features are added	The lack of author affiliations was also recorded rather than fixed

5. What Can and Cannot Be Said

What can be said

• For the 7 pairs whose body text can be read on PMC, the Accuracy of study characteristics written out with no human intervention was 0.735 (95% CI 0.642–0.811), on par with the original paper’s patient characteristics value of 0.74
• Every value among the 102 items, except “not stated,” has a verbatim quote from the full text (it passed the hook’s check)
• For all 11 incorrect “not stated” items, a reason can be given (some inferred): calculation, supplementary materials, affiliations, figure images, and so on

What cannot be said

• It cannot be said that performance matches the original paper. The denominator is small at 102, there are 2 reviews, and 1 scorer. The original paper’s 0.78 includes outcome items, which this trial does not have. The models also differ (the original paper used GPT-4 and Claude 3 Sonnet; this trial used the current Claude Sonnet)
• It cannot be said that this is usable for SR extraction. Of the 20 answer pairs, 13 had no available body text and were not even covered
• It cannot be said that a value is correct because it has a quote. The hook only confirms that a quote appears verbatim in the full text. The 16 mismatched values had quotes and still did not match the answers
• Nor can it be said that the extractor’s reports can be trusted. In its final report, the extractor for 33495835 wrote “the following 7 items” were “not stated” but listed only 6, and the output also had 6 (the output was correct). Also, for the same paper 30396908, the 33746596 job marked Country “not stated,” while the 37168849 job answered Location with the institution name. Just a different item name changed how it searched
• Nor can I say outright that it was built according to the procedure. An agent definition added mid-session was not loaded, and the trial run stopped with “Agent type ‘extractor’ not found.” Running general-purpose in its place would not apply omitClaudeMd or the skill preload, so I did not substitute it; a human restarted Claude Code

6. Summary

The purpose of this step was to copy study characteristics from the full text of the included studies into a table, and to check how far that can be done without human hands. For the 7 pairs and 102 items whose body text can be read on PMC, Accuracy was 0.735 (original paper’s patient characteristics: 0.74).

Of the 11 errors from “not stated,” 5 were things not in the input (supplementary materials, author affiliations, figure images), 3 were calculations forbidden by the rules, and 3 were a “No” not written in the text. What goes into the input directly sets the ceiling on what values can be extracted. And for 13 of the 20 answer pairs, PMC alone could not reach the body text.

In terms of structure, I turned the full text into text while collecting tables from outside <body> too, launched the subagent extractor per study, preloaded the rules with a skill, checked quotes with a PreToolUse hook, and used another hook to restrict the readable files to each job.

Next Steps

Part 4 covers discussion and reproduction. I put search, screening, and extraction side by side with the original paper and summarize what can and cannot be said. I also explain why only a human launches the evaluation skill /eval, the setup for running on a subscription without an API key, and the steps to clone the repository and reproduce the same flow.