[Claude Code] Turning Visual Checks into Automated Checks, Part 2: Verifying the Results — Pixel Matching and an Answer-Keyed Dummy TLF

1. Introduction — “it ran” is not the same as “it is right”

The page-break detection described in Part 1 is simple if you look only at the steps. Convert the page to a greyscale image and count the white pixels along the edges. That is exactly why getting results on screen and those results being correct are two different things.

• A slightly different greyscale formula changes how many pixels fall on the white side of 245
• A different downscaling method changes the values of pixels at the edges
• Confusing “greater than” with “greater than or equal to” changes the verdict for a page at exactly 0.98

None of these is visible just by looking at the screen. A tool for checking TLFs is pointless if it is careless about checking itself.

This post shows how we verified the page-break detection in TLF Visual QC at three levels.

1. Parts: compare greyscale conversion and downscaling pixel by pixel against a standard tool
2. Whole: count hits and misses on a dummy TLF whose broken pages and answer key were fixed in advance
3. Real: see how it behaves on a real submission-format TLF

• Every pixelgreyscale and downscaling results match ImageMagick
• 8 / 8deliberately broken pages that were flagged CHECK
• 0CHECK flags on pages that were not broken (dummy TLF)
• 4pagesthe submission-format TLF we tried (R Consortium submission pilot)

2. Parts — matching greyscale conversion and downscaling pixel by pixel

2-1. Why start with the components

The page-break detection values depend heavily on the results of greyscale conversion and downscaling. The white cut-off is a single line, “brighter than 245,” so a tiny difference in the greyscale formula produces pixels that cross the line, such as 244 versus 246. Downscaling, too, changes the brightness of pixels at character edges depending on how neighbouring pixels are blended.

So we compared the greyscale conversion and downscaling written for the browser, pixel by pixel, against the same processing done with ImageMagick, a widely used image processing tool.

2-2. Results

Feeding in the same page images, for 29 pages of the dummy TLF, the pixel values after greyscale conversion and downscaling matched ImageMagick on every pixel (a difference of 0). Three of those pages are included in unit tests, so we re-check every time we make a change. With not a single mismatch, no component-level drift can creep into the white ratio calculations that follow.

3. Whole — an answer-keyed dummy TLF

3-1. Fix the answer first

Even if the components are correct, the whole they make up is not necessarily correct. To verify the whole, you need an answer key saying which pages are broken.

With a real TLF, people decide the answer after the fact. There is a risk that, having seen the results, judgement wavers: “you could say this page is broken.” So we took the approach of deciding which pages to break first, and then building the TLF.

The dummy TLF is 31 pages from a fictitious study. Which pages were broken, and how, is written to an answer file (JSON) at the same time as the TLF is generated.

3-2. Contents of the dummy TLF

Type	Orientation	Pages	Answer
Cover	Portrait	1	Not checked
Demographics table (full page)	Portrait	2–4, 6, 9, 11, 27, 28, 30	OK
Demographics table (ends two thirds of the way down)	Portrait	7	OK
Last page of demographics table (only 2 rows left)	Portrait	5, 29	CHECK
Columns shifted left, right half empty	Portrait	8	CHECK
Nearly blank (title and one note line only)	Portrait and landscape	10, 21	CHECK
Adverse event listing (full page)	Landscape	12–14, 16, 17, 19, 20, 22, 23, 25	OK
Adverse event listing (ends two thirds of the way down)	Landscape	26	OK
Last page of adverse event listing (only 1–2 rows left)	Landscape	15, 24	CHECK
Listing columns shifted left, right half empty	Landscape	18	CHECK
Short efficacy table with 2 footnote lines at the bottom of the page	Portrait	31	OK

Besides the broken pages, we also included pages with a lot of white space that are not broken: tables ending two thirds of the way down (7, 26) and the page with a short table and footnotes (31). If the only goal were to catch broken pages, flagging everything CHECK would score every hit. Not flagging pages that are not broken matters just as much.

Mixing portrait tables with landscape listings also brings the dummy closer to a real TLF bundle. On landscape pages, the same amount of text yields a different white ratio.

3-3. Results

Of the 30 pages checked (2–31), 8 were flagged CHECK: 5, 8, 10, 15, 18, 21, 24 and 29. All matched the 8 answer pages, and none of the 22 unbroken pages was flagged.

	Answer CHECK	Answer OK
Verdict CHECK	8	0
Verdict OK	0	22

Here are the values for the OK pages with a lot of white space.

Page	Content	Bottom edge white ratio	Verdict
7	Table ending two thirds of the way down (portrait)	0.944	OK
26	Listing ending two thirds of the way down (landscape)	0.859	OK
31	Short table and 2 footnote lines	0.977	OK (near threshold)

Pages 7 and 31 stop just short of the 0.98 threshold. Page 31 in particular is only 0.003 away. The margin between broken and unbroken pages is not that large.

4. Real — trying a submission-format TLF

4-1. The R Consortium submission pilot

As a real TLF, we used the one from R Submission Pilot 1, published by an R Consortium working group. In this initiative, a submission package assembled in R was submitted to the FDA on a trial basis in eCTD format and the FDA reproduced the analysis; the full submission is public on GitHub. The study is the CDISC pilot study, and the TLFs are not from any real company.

From this, we loaded the PDF that compiles the TLFs for submission (4 pages). According to the README, the table formats in this PDF are deliberately not harmonised, in order to preserve differences in how each organisation produces them.

4-2. Results

Page	Bottom edge	Right edge	Overall	Verdict	Assessment
1	0.857	0.895	0.862	(not checked by default)	No cover; the table starts on page 1
2	0.960	0.922	0.917	OK	Large empty area below the table, but a footnote sits in the bottom half of the measured area
3	1.000	0.897	0.885	CHECK	A short table ends near the top. Not a page-break problem
4	0.896	0.913	0.876	OK	—

4-3. What we learnt

It is a small example of only 4 pages, but it shows the nature of the method clearly.

• CHECK lands on “pages with a lot of white space”: page 3 is not broken; a short table simply ends near the top. Its bottom edge is still pure white, so it is flagged CHECK. The flag means “please look at this by eye,” not “this is broken”
• Footnotes can cause misses: page 2 has a large empty area below the table, but because a footnote sits in the bottom half of the measured area, its bottom edge white ratio stops at 0.960 and it is OK. This is the limitation described in Part 1, appearing in a real TLF
• Some PDFs have no cover: the table starts on page 1, so the default range (from page 2) skips page 1. The tool’s About section advises setting the range to Custom to check from page 1

On the dummy, all 8 pages were hits, but on a real TLF, both flags on ordinary pages with a lot of white space and misses caused by footnotes can happen. Not claiming “it finds everything” on the strength of the dummy results alone is the most important thing we confirmed at this level.

5. What we left to Claude Code, and what people decided

Left to Claude Code	Decided by people
Tests matching pixels against ImageMagick	Verifying at three levels: components, whole and reality
The program that builds the dummy TLF and its answer file	The kinds of breaks, and including unbroken pages with a lot of white space
Comparing verdicts against the answer, and reporting the results	Fixing the answer before the TLF
Summarising the results of loading the submission pilot TLF, with an assessment of each page	Which public TLF to use, and showing only values without images

The dummy TLF was generated by Claude Code from fictitious data. What people decided was “what to break” and “what to include unbroken”; we wrote the list of break types and the shape of the answer into the instructions before handing it over.

Once written, matching and tests can be run any number of times. Even when we added a page to the dummy or rebuilt the screens partway through, re-running the component matching and the dummy verdicts confirmed each time that the results had not changed.

6. Summary

• Getting results on screen and those results being correct are different things. We verified at three levels: components, whole and reality
• Parts: greyscale conversion and downscaling matched ImageMagick on every pixel
• Whole: on a dummy TLF (31 pages) whose broken pages were fixed in advance, CHECK matched the 8 answer pages and did not appear on any of the 22 unbroken pages
• Real: on the R Consortium submission pilot TLF, a short-table page was flagged CHECK and a page with a footnote was missed. The flag means “a page to look at by eye,” not “a broken page”
• Fixing the answer first, and including unbroken pages with a lot of white space, were decisions made by people

If you prepare the answer in advance, you do not have to stop at “roughly right.” And the limits of an answer you prepared yourself also need to be checked against a real TLF.

Next steps

Part 3 covers the other check: figure colours. It shows how we measure and flag the difference in brightness when printed in black and white, the difference in colour, and how figures appear under different types of colour vision, with results for seven example colour pairs.