Claude Code × Medical Application

[Claude Code] Treatment Pathways on One Page, Part 3: Verifying the Results — Fictitious Patient Data with the Answers Set First

日本語 / English

1. Introduction — a chart appearing is not the same as the counts being right

A Sankey diagram looks plausible as soon as you load some data. That is exactly why a chart appearing on the screen and its patient counts being correct are two different things.

  • If the same treatment is written in two ways, its bar splits in two, and both look thin
  • If patient IDs are duplicated, the same person is counted twice
  • If the second-line column is read one column off, the transitions are entirely different

None of these can be spotted just by looking at the chart. A tool meant to show how treatment actually happens is pointless if it neglects to check its own numbers.

In this post we show how we checked the numbers of the Treatment Line Sankey Diagram at four levels.

  1. The whole: compare patient counts with fictitious patient data whose answers were set first
  2. Breakage: watch how loading and the data quality checks behave on deliberately broken input
  3. Name variants: check the merge proposals from both rules and AI against the answers
  4. Size: measure the speed on a 100,000-row file
  • 145 / 145itemsitems where the answer file and the tool’s totals matched
  • 10 / 10runsruns in which the AI’s proposals (column mapping and name variants) matched the answers
  • 21warningswarnings raised by the data quality checks on the deliberately broken input
  • about 0.9sloading and checking a 100,000-row file

2. The whole — fictitious patient data with the answers set first

2-1. Why generate fictitious data yourself

Real prescription data cannot be put into a public tool. Even with publicly available data, the answer to “how many patients received what, and where did they move from and to” has to be counted by someone after the fact. That leaves room to look at the result and decide it is “roughly right”.

So we turned this around: set the answers first, then generate the data.

2-2. Fictitious rheumatoid arthritis patient data

The fictitious data built into the tool covers 3,000 rheumatoid arthritis patients.

Item Content
Patients and columns 3,000 patients, 34 columns, up to Line 5
Column layout The layout of an analysis listing (group names tx_groupname_1 to 5, start and end dates, individual regimen names, sort-order codes, facility, sex, age, lab values and so on)
Treatment groups MTX monotherapy, csDMARD combination, TNF inhibitor + MTX, non-TNF biologics, JAK inhibitors, other (drugs are given only as generic names and drug classes)
Planted name variants tnfi+mtx at facility B (upper/lower case), JAKi at facility C (full-width characters)
How it is generated A program with a fixed random seed. It produces the same data every time
The first 9 patients of the fictitious data demo_data.csv. After columns such as patientid, site, sex and age come regimen_name, tx_group, tx_groupname, tx_start and tx_end for Lines 1 and 2. The planted name variants tnfi+mtx and JAKi can be seen in the tx_groupname columns
Contents of the fictitious data built into the tool (some columns only)

One patient’s record looks like this, for example.

Timeline of one fictitious patient’s prescriptions. Four lines: csDMARD combination → JAK inhibitor → TNF inhibitor + MTX → non-TNF biologic
The four lines of one fictitious patient (RA000037) (fictitious data)

2-3. Writing out the answers at the same time

The program that generates the data writes out an answer file (JSON) at the same moment.

  • The number of patients for each treatment in each line
  • How many patients moved from a given treatment in one line to each treatment in the next line
  • The pairs of planted name variants, and the count for each

We load the fictitious data into the tool, merge the name variants, and then compare the resulting totals with these answers item by item.

2-4. Results

What was compared Number of items Matched
Patient counts by line 32 All
Patient counts for transitions to the next line 111 All
Counts of name variants (tnfi+mtx 348, JAKi 131) 2 All
Total 145 145
The answers and the tool’s totals side by side. All 145 items, patient counts by line and patient counts for transitions, match
Comparing the answers with the tool's totals (fictitious data)

This comparison is written as a test and runs automatically every time the tool is changed. Whenever we rebuilt the screens or revised the rules for the checks, we could confirm each time that the totals had not changed.

3. Breakage — deliberately broken input

3-1. Deciding how to break it

Real files have untidy parts. So, separately from the fictitious data, we prepared a small piece of deliberately broken input (160 rows, 16 columns). It is broken in the following eight ways.

How it is broken Examples
Shape of column names Inconsistent case and separators (line 1, Line_2, LINE-3)
How missing values and values are written Blanks, NA, null and so on mixed together. Some values have leading and trailing spaces (" Beta ")
Quotation marks Delimiters and line breaks inside quotation marks
Date formats 2020-01-05, 2020/1/5, 20200105 and others mixed together
IDs Exponential notation (1.05E+17) and duplicated IDs
Skipped lines First-line therapy is blank but second-line therapy has a value
Blank rows in the middle Empty rows in the middle of the data
Line endings Windows line endings (CRLF)
The first 8 rows of the deliberately broken input tricky.csv. Column names are inconsistent in shape, and the data mixes duplicated IDs, NA and NULL, impossible dates such as 2021-02-30, not a date, leading and trailing spaces, and alpha alongside the full-width ALPHA
Contents of the deliberately broken input (first 8 rows)

3-2. Results

  • Column mapping: even with the column names out of shape, the columns were matched by name
  • Data quality checks: 21 warnings were raised
Data quality checks on the deliberately broken input. The list includes duplicated IDs, IDs in exponential notation, leading and trailing spaces, values differing only in case, dates out of order, gaps between lines and blank dates
Data quality checks on the deliberately broken input
Warning Count Details
Duplicated patient IDs 1 160 rows with an ID, but only 149 distinct IDs
IDs in exponential notation 1 There are IDs such as 1.05E+17
Leading and trailing spaces 3 " Beta " appears in the Line 1 to 3 columns (on screen the value is wrapped in quotation marks to show the spaces)
Values differing only in case 3 Alpha and alpha in the Line 1 to 3 columns
Dates out of order 2 38 rows where the second-line start date is earlier than the first-line one, and 40 rows where the third-line start date is earlier than the second-line one
Gaps between lines 2 29 rows where first-line therapy is blank but second-line therapy has a value, and 14 rows where second-line is blank but third-line has a value
Dates without a line 3 Rows where the treatment name is blank but a start date is filled in: 25, 17 and 18 rows in Lines 1 to 3
Values that cannot be read as dates 3 Values that are filled in but cannot be read as dates: 12, 22 and 22 in the start dates of Lines 1 to 3 (blanks not included)
Blank dates 3 Rows where a line has a treatment but the start date is blank: 16, 13 and 10 rows in Lines 1 to 3

We confirmed that even with broken input, the tool does not quietly turn it into a chart. It still draws the chart, but the warnings are listed on the “Data quality” tab, so you can see what is going on.

3-3. Checking the checks themselves

This table first had 15 warnings. A review before publication found three discrepancies in the rules for the checks.

  • Leading and trailing spaces: the check looked at values after the spaces had been removed, so this warning could never appear. We changed it to check the values as loaded
  • Values that cannot be read as dates: blanks were also being counted as “cannot be read”. We changed it to count only values that are filled in but cannot be read, and to report blanks separately as “blank dates”
  • Duplicated patient IDs: blank IDs were included in the count. We changed it to look for duplicates only among IDs with a value, and to report blanks separately

The answers for the checks were produced by recounting from the input file with a separate program, without using the tool’s counting code. If the tool’s code were used to produce the answers, the same mistake would become part of the answers.

4. Name variants — checking rules and AI against the answers

4-1. Name variants change patient counts

The fictitious data has two pairs of facility-specific spelling differences planted in it. In the chart before merging, the same treatment is split into separate bars.

Sankey diagram before merging the name variants. TNFi+MTX and tnfi+mtx, and JAKi and JAKi, appear as separate bars
Before merging. The same treatment is split into separate bars (fictitious data)

4-2. Rule-based proposals

The tool tries to normalise names with the steps below, and proposes merging names that end up in the same form.

  1. Unify full-width and half-width characters
  2. Convert to lower case
  3. Split into individual drugs at separator symbols (+ + / / ・)
  4. Remove spaces from each drug
  5. Sort the drugs and join them again (so that “MTX+TNFi” and “TNFi+MTX” become the same)

The name to keep is the spelling with the highest count.

Rule-based merge proposals. TNFi+MTX 864 with tnfi+mtx 348, and JAKi 419 with JAKi 131
Rule-based proposals. Only the two planted pairs appear (fictitious data)

On the fictitious data, the proposals matched the two planted pairs exactly. No extra pairs appeared.

However, these rules only catch differences in the “shape” of the spelling. They do not catch the presence or absence of a hyphen, an abbreviation versus the full name (“MTX” and “methotrexate”), Japanese versus English, or typing mistakes.

4-3. AI proposals

For differences the rules cannot catch, you can also ask an AI for merge proposals. What the AI receives is only the regimen names and their counts (in the public edition, only for the fictitious data; see Part 4 (coming soon) for details).

AI merge proposals, each listed with a one-sentence reason
AI proposals. Listed with reasons; nothing changes until you confirm (fictitious data)
Before and after selecting and merging the AI's proposals (fictitious data)

The AI’s answers are not always the same. So for column mapping and for name variants, we called the AI five times each on the fictitious data and checked the results against the answers (no answers were reused; the AI was called every time).

Request Runs Matched the answers Time per run Cost per run
Column mapping (lines, start dates, patient ID) 5 5 3.4–4.2 s about 0.1 yen
Merging name variants 5 5 1.1–1.4 s about 0.03 yen

All 10 runs were correct. For column mapping, the AI got all of the lines, start dates and patient ID right; for name variants, it returned only the two planted pairs and no extra ones.

4-4. After merging

When you select the proposals and merge them, each treatment becomes a single bar and the patient counts match the answers (the comparison in section 2 is done in this state). Merging affects only the chart and the numbers; it does not change the values in the loaded file. “Undo the merges” puts things back.

Sankey diagram after merging the name variants. Each treatment is now a single bar
After merging (fictitious data)

5. Size — speed on 100,000 rows

Real data can be much larger than 3,000 patients. So we repeated the rows of the fictitious data, assigned new patient IDs, created a 100,000-row (14.7 MB) file and measured it.

Process Time (5 runs)
Loading and checking 912–950 ms
Counting (default settings) 52–67 ms
Counting (up to Line 5) 83–98 ms

Counting is the process that runs every time you focus or change a setting. At around 50 ms, the chart changes as soon as you click.

6. What we handed to Claude Code, and what people decided

Handed to Claude Code Decided by people
The program that generates the fictitious data and the answer file The disease (rheumatoid arthritis), the treatment groups, and the shape of up to Line 5
Tests comparing the answers with the tool’s totals Setting the answers before the data
Preparing the deliberately broken input and tests to check the behaviour The eight ways of breaking it, especially including IDs rounded by Excel
Reviewing the checks, and the answers for the checks recounted by a separate program The rules for raising warnings, such as separating blanks from “values that cannot be read”
Implementing the rule-based proposals Where to draw the line between differences to catch (shape only) and differences not to catch
Calling the AI five times each and checking against the answers Keeping the AI’s answers as proposals
Measuring the speed on 100,000 rows What size of data to plan for

Once written, comparisons and tests can be run any number of times. Whenever we rebuilt the screens, added AI features or revised the rules for the checks, we could confirm each time that the totals had not changed.

7. Summary

  • A chart appearing on the screen is not the same as the patient counts being correct. We checked at four levels: the whole, breakage, name variants and size
  • The whole: with fictitious rheumatoid arthritis data (3,000 patients) whose answers were set first, all 145 items matched the answers
  • Breakage: with deliberately broken input (160 rows), columns were matched by name and 21 warnings were raised. We also found and fixed discrepancies in the checks themselves
  • Name variants: the rule-based proposals gave only the two planted pairs, and the AI was correct in all 10 runs. Even so, a person confirms
  • Size: a 100,000-row file loads in about 0.9 seconds, and counting takes around 50 ms
  • Setting the answers first, deciding how to break the input, and keeping the AI as a source of proposals were decided by people

If you prepare the answers first, you do not have to stop at “roughly right”. And the limits of having made the answers yourself should be written down too.

Next steps

Part 4 covers building in the AI: what to give it and what not to, how we limited it to the fictitious data in the public edition, and how to confirm that nothing is sent to the AI when you use your own files.

Sources

  • The screenshots and figures on this page were produced with fictitious rheumatoid arthritis data (3,000 patients) and deliberately broken input. They have no connection with any real patients, facilities, drugs or companies