Testing
Preferred in 6 of 9 blind pairs. No more defects found.
In a pre-registered blind round, three Claude judges preferred loupe’s review to one the same model wrote without it in 6 of 9 pairs. Both found nearly every planted defect; loupe’s had fewer findings judged wrong or inflated, and no Blockers on a well-built app. The round failed one of its four rules: recall, by half a defect with Haiku.
How the round worked
The rules below were written and committed before any review ran (preregistration.md). The answer keys were sealed by hash in the same commit.
Three fixtures in fixtures/, written for this round by separate model instances (Claude Sonnet) that were told not to read this repository, any design-review rule set, or any skill. A fourth instance, with the same restriction, checked each fixture in a browser, corrected the answer keys, and added real defects the authors had not planted. None of the domains appears in loupe's references or earlier fixtures.
| Fixture | What it is | Key |
|---|---|---|
web-sloppy |
Lendwell, a neighbourhood tool-lending library page (HTML, CSS, JS) | 10 planted defects plus 5 found by the checker |
web-good |
Split the check, a bill-splitting calculator built to be good (the control) | 0 Blocker, 0 Major, 3 Minor, 5 Polish, per the checker |
native |
Dosewise, a SwiftUI medication-reminder app (iOS 17 minimum, Xcode 26 toolchain), source only | 8 planted defects plus 6 found by the checker |
The keys are sealed: only their SHA-256 hashes are in keys.sha256. They are added to this folder after the tally, and anyone can check them against the hashes. After the keys were sealed, comments in web-sloppy/index.html that named the planted defects ("D1: ~2.2:1 on white") were removed; no code changed.
Each fixture is reviewed under two conditions by three models (Claude Haiku, Sonnet and Opus): 3 × 3 × 2 = 18 reviews.
- Baseline: the model is asked for a thorough UI/UX and accessibility design review of the fixture and is told not to use any installed skill.
- Skill: the same request, with loupe's
SKILL.md,references/andscripts/copied to a temporary folder (noevals/, noresearch/, no site) and the instruction to follow that SKILL.md.
Both conditions get the same tools: file reading, a shell, web search, and Playwright through a Node script (the Playwright MCP server cannot be shared by parallel runs). Neither gets the keys, this file, or the path of this repository. The user-facing questions loupe would normally ask are answered with "no answers; proceed on defaults", and web fixtures are reviewed as release_stage: development.
Every report is kept in runs/<model>/<fixture>/<condition>.md, with the model ID and the commit of the skill.
- Preference. Across the 9 pairs, loupe's report is preferred in at least 6 by at least 2 of the 3 judges.
- No over-reporting on the control. On
web-good, for each model, the number of loupe's findings judged wrong or inflated by at least 2 judges is no higher than the baseline's. - Recall. For each model, loupe's recall summed over
web-sloppyandnativeis at least the baseline's. - Restraint. On
web-good, loupe reports no Blocker and "No rework warranted" for at least 2 of the 3 models.
The round passes if all four hold. Each rule's result is reported separately, and a failed rule is reported as a loss with the pairs that caused it.
Check the seal yourself
This fetches the key hashes as committed before any review ran (commit 98a41da) and the keys as published now, both from GitHub, and hashes the keys in your browser.
Results against the four rules
Produced by tally.py from the judges’ and scorer’s files. Unedited.
Rule 1: preference
loupe preferred by at least 2 of 3 judges in 6 of 9 pairs (needs 6): pass
| Pair | Model | Fixture | Judges for loupe | for baseline | tie |
|---|---|---|---|---|---|
| pair01 | opus | web-sloppy | 3 | 0 | 0 |
| pair02 | haiku | web-sloppy | 2 | 1 | 0 |
| pair03 | haiku | native | 1 | 2 | 0 |
| pair04 | sonnet | web-good | 3 | 0 | 0 |
| pair05 | opus | web-good | 1 | 2 | 0 |
| pair06 | opus | native | 3 | 0 | 0 |
| pair07 | sonnet | native | 3 | 0 | 0 |
| pair08 | sonnet | web-sloppy | 3 | 0 | 0 |
| pair09 | haiku | web-good | 0 | 3 | 0 |
Rule 2: no over-reporting on the control
Median, across the three judges, of findings judged wrong or inflated.
| Model | loupe | baseline |
|---|---|---|
| sonnet | 2 | 3 |
| opus | 3 | 3 |
| haiku | 1 | 3 |
pass
Rule 3: recall on the sealed keys (web-sloppy + native; partly = 0.5)
| Model | loupe | baseline |
|---|---|---|
| haiku | 27 of 29 | 27.5 of 29 |
| sonnet | 27 of 29 | 26.5 of 29 |
| opus | 29 of 29 | 27 of 29 |
fail
Rule 4: restraint on the control
| Model | Blockers | "No rework warranted" |
|---|---|---|
| haiku | 0 | yes |
| sonnet | 0 | yes |
| opus | 0 | yes |
3 of 3 models (needs 2): pass
Round: fail
Reading the result, and what we changed
From notes.md, written after the tally.
What the result says
The round failed one of its four pre-registered rules, so by its own terms it failed.
- Preference: passed, at the threshold. Judges preferred loupe's report in 6 of 9 pairs (2 or more of 3 judges). The 3 losses were Haiku on the native app, Opus on the control, and Haiku on the control, which all three judges gave to the review written without the skill.
- Over-reporting on the control: passed. On the well-built split-the-check app, the median judge flagged 2, 3 and 1 of loupe's findings as wrong or inflated (Sonnet, Opus, Haiku), against 3, 3 and 3 for the reviews without it.
- Recall: failed. Summed over the flawed web page and the SwiftUI app, loupe's reports found 27, 27 and 29 of the 29 key items (Haiku, Sonnet, Opus); the reviews without it found 27.5, 26.5 and 27. Haiku with loupe missed by half an item, one "partly" found. Every report, with or without the skill, found all 10 planted defects on the flawed page, so this fixture measured almost nothing about recall.
- Restraint: passed. On the control, loupe reported no Blocker and "no rework warranted" with all three models. The reviews without it put 10 findings in their top severity tier across the three models on the same app.
Across all 9 pairs and 3 judges, findings judged wrong numbered 24 for loupe and 38 without it, inflated 40 and 52, and top-severity findings judged unbacked 29 and 44.
The plain reading: on these fixtures, loupe did not help Claude find more problems. Claude already finds most of them. It changed how the problems were rated and argued: fewer findings judges called wrong or inflated, no invented Blockers on a good app, measured numbers where the other reviews estimated. Judges preferred that in two thirds of pairs, not in all of them.
Deviations from the pre-registration
- Rule 2 counting. The pre-registration counts loupe's findings "judged wrong or inflated by at least 2 judges". Judges described findings in their own words, so the same finding could not be matched reliably across judges.
tally.pyinstead takes the median, across the three judges, of each judge's count. This choice was made intally.pybefore any judge answered. - Scorer prompt for the control. The scorer for
web-goodwas told to test two behaviours in a browser (invalid tax counted as zero, and tax spread over assigned items only), because reports of both conditions claimed them and the key did not list them. It applied to every report on that fixture. Both turned out to be real defects that the fixture's author and checker had missed. - Report files. The harness did not let some reviewers write files, so their reports were taken from their final messages, verbatim. Local paths, and the run folder name (which named the condition), were redacted from every report before judging.
- Judge rubric and data loss. The judges' Blocker test (a measured WCAG A/AA failure, a task that cannot be completed, or a likely App Store rejection) left out data loss, which loupe's own Blocker rule includes. Several of loupe's Blockers that judges called unbacked were data-loss findings. The rubric was pre-registered and was not changed.
What changed in loupe afterwards
Recurring criticisms from the judges, fixed after this round in 1.0.0 (they were not re-tested):
- A glyph-only button whose glyph gives it an accessible name ("☆" reads as "white star") was rated a 4.1.2 Blocker. 4.1.2 needs a missing name; a poor name is now Major.
- Placeholder figures in a development build were rated a Trust-gate Blocker. They now go to Unverified, as the release-stage rule already said for other placeholders.
- Four of the nine skill runs (Haiku and Sonnet on both web fixtures) skipped the landscape research without saying why. The skill now requires the reason in Run notes.
- One run missed the build SDK stated in a
project.pbxprojcomment and fell back to the deployment target. The SDK detection now lists toolchain comments as a source.
Every pair, every judge
Each judge saw the two reports as A and B and was not told which used loupe. “Flagged” counts the findings that judge called wrong or inflated.
pair01: Lendwell (flawed web page), written by Claude opus
| Judge | Preferred | Flagged, loupe | Flagged, no skill | Reason given |
|---|---|---|---|---|
| opus | loupe | 3 | 6 | Both reports find the same core defects: focus removed, date wiped by the re-render, no stock decrement, the table overflowing on mobile, the ticker, the 2.21:1 meta text and the 88vh hero. A's numbers match the source and my measurements: star 31.9x32, #tools at y=845 at 800 px and about 933 at 900 px, scrollWidth 664. A ties its Blockers to WCAG failures and adds dark mode, rem sizing and a date-first rework. B makes a few factual errors and gives Critical to items that have no WCAG basis and do not block the task. |
| sonnet | loupe | 2 | 5 | Report A has the same core findings as B (focus ring removed, ticker with no stop, 2.21:1 meta text, date wiped on loan-length change, table overflow, stock never decrements, disabled Reserve looks enabled). It adds measured evidence, a heading-order hero argument, a landscape study, and a clear rework recommendation. B also finds '1 days' and the toast-timer bug, and its unlimited-double-booking point is valid. But B rates four items Critical and two of them are not backed by anything measured. B also misstates the star hit area. I checked the source: A's findings are accurate, apart from one weak Blocker on the star. |
| haiku | loupe | 2 | 5 | A's four Blockers (focus 2.4.7, ticker 2.2.2, meta contrast 1.4.3, unnamed star state 4.1.2) are all backed by source or measurement, and it rates the date wipe Major, which fits because the user can re-pick the date. Its main error is the hero-paragraph contrast claim. B rates three non-blocking items Critical and states two facts that are wrong (the star hit area and 'hides Cancel'). |
pair02: Lendwell (flawed web page), written by Claude haiku
| Judge | Preferred | Flagged, loupe | Flagged, no skill | Reason given |
|---|---|---|---|---|
| opus | loupe | 1 | 5 | B's measurements are accurate. It pixel-sampled the hero (passes), got the star size right (32x32), and correctly says the nav targets pass 2.5.8 through spacing. Its Blockers are mostly tied to WCAG failures. A makes several wrong WCAG claims (placeholder colour, hero paragraph contrast, 2.5.8) and puts the membership and hero items in its Blocker/High tier. B's one weak spot is rating the placeholder stats a Blocker. |
| sonnet | no skill | 2 | 2 | Both reports find the same real defects: stock never decrements, date wiped, disabled Reserve looks enabled, focus ring removed, hero hides the tools, table overflow, 2.21:1 contrast, ticker, empty states. A's headings (Blocker/High, Medium, Low) are sensible and A puts the date wipe and stock bug near the top. B rates the date wipe, which is real data loss in the core flow, as only Major. B also lists an invented-figures finding as a Blocker, and that finding is wrong. Its own text says it is a development build with placeholder data, and it is not a measured failure. A has some speculative items (the membership and sign-in gap, and a hero contrast figure that A's own pixel check would not support). |
| haiku | loupe | 3 | 3 | B's Blockers are mostly backed (focus, ticker, meta contrast), and it measured the hero contrast correctly, saying it passes in pixel sampling. Its severities are generally calibrated, with the date wipe rated Major and the star measured correctly. Its main overreach is the Blocker on invented figures. A has the same hero-contrast error as pair01 A, a wrong unit count, and an ambiguous 'Blocker / High' group that includes items that do not block tasks. |
pair03: Dosewise (SwiftUI), written by Claude haiku
| Judge | Preferred | Flagged, loupe | Flagged, no skill | Reason given |
|---|---|---|---|---|
| opus | no skill | 1 | 4 | A correctly puts the app's functional failures at the top: no reminders are ever scheduled, nothing persists, and doses are only generated for the day a medicine is added, so Today is empty from day 2 (Models.swift:43-51). B never reports that day-2 failure; its finding 13 only covers same-day past-due doses. B also rates reminders and persistence as Major. B is better on measured contrast: #4B5563 on #1C1C1E is 2.25:1 and system green on white is 2.22:1, both of which I confirmed. A misses the green failure, and over-escalates B4 and B5 to Blocker. Even so, A is the report an owner should act on. |
| sonnet | no skill | 3 | 4 | A catches the defects that stop the app doing its job: no reminders are scheduled, nothing persists, doses are only created for the day the medicine is added, and missed doses vanish. B misses the day-2 empty Today screen entirely (its finding 13 only covers a past-time dose) and rates reminders and persistence as Major. B does have measured contrast numbers, and I checked them against the source: #4B5563 on #1C1C1E is about 2.25:1, and system green on white is about 2.2:1. A calls the dark-mode contrast Medium and never flags the green label. Overall A is the better plan to act on, with B's contrast fixes added. |
| haiku | loupe | 1 | 3 | Report B covers both WCAG 1.4.3 contrast failures in source: the #4B5563 dark-mode time text (about 2.25:1 on #1C1C1E) and the green Taken label (about 2.2:1 on white). Report A misses the green Taken label entirely, rates two of its Blockers too high, and wrongly says the project file is missing. Report A's roadmap for reminders and persistence is clearer, so this is close. |
pair04: Split the check (control), written by Claude sonnet
| Judge | Preferred | Flagged, loupe | Flagged, no skill | Reason given |
|---|---|---|---|---|
| opus | loupe | 0 | 3 | Both reports catch the same core problems: invalid tax or tip silently counts as 0, the results sit far down the page on phones, the sticky panel overflows, nothing can be edited after it is added, and focus is lost after re-renders. I confirmed each in Chromium. B's measurements are more accurate. It describes the sticky overflow correctly: Copy is only reachable near the end of the scroll, not never. B also finds things A misses. 'Check total' leaves out unassigned items, which breaks the lede's 'exact total' promise. Invalid tax text comes back after a reload with no error shown. Focus drops to <body> after Undo and Dismiss. Comma decimals get an unhelpful message. A has a few overstatements. |
| sonnet | loupe | 2 | 2 | Both lead with the same verified bugs: invalid tax/tip silently counted as 0 (model() uses `|| 0`), and the sticky .out panel overflowing at many people. B also found two real problems A missed. Check total excludes unassigned items while the lede promises the exact total. Restored invalid tax text shows no error after reload (renderAll only sets values; setErr runs on input). B also catches the toast being cleared by save() and focus dropping to body after Undo. B's severities are Major/Minor/Polish and none is a blocker. A has some real findings (magic margin-top:28px on button.add, red warning on every new item). Its ranking is slightly weaker and it has a speculative toast-announcement claim. |
| haiku | loupe | 2 | 3 | B finds the unassigned-item money defect (Check total excludes unassigned items while the full tax is still spread over assigned people), which A misses entirely. A is tidier and calibrates phone distance better (Medium), but the money miss matters more. Both share the invalid-tax-as-zero, sticky-panel, no-edit and focus findings, all confirmed in source. |
pair05: Split the check (control), written by Claude opus
| Judge | Preferred | Flagged, loupe | Flagged, no skill | Reason given |
|---|---|---|---|---|
| opus | no skill | 3 | 4 | A covers more of what actually goes wrong in use, and I verified its key claims. Unassigned items drop out of the subtotal while the whole receipt tax is spread over the assigned items: with Pizza unassigned, one person carried all $5 tax. Typing '12.' in Tax shows an error, and the total is then computed with $0 tax. Validation fires on half-typed input such as '12.', '17.' and '1,0'. Focus is lost after Everyone and after Undo. B also finds the unassigned-tax bug, and adds the forced-colours preset issue (confirmed: the pressed preset differs only by border) and RTL padding. But B misses the silent zeroing of invalid tax or tip and the false errors while typing, and it spends much of its length on theming directions. A's weakness is calling two findings Critical. |
| sonnet | loupe | 4 | 2 | Both find the same core issues: invalid tax/tip counted as 0, unassigned items inflating the remaining shares (the full tax is spread over assigned items only), the result being far down the page on phones, focus loss after Everyone, no editing, the uncapped sticky panel and the undo toast being cleared by save(). A labels two of them Critical. B uses calibrated Major/Minor ratings, adds measured extras (200% text collapse of .item-head, forced-colors pressed preset, undo being 30 Tab stops away, a sourced landscape), and gives concrete fixes. A is tighter and has a good live-validation-mid-typing finding. B has a few weak minor items (I/l glyph, 'e.g.' read as 'egg'). |
| haiku | no skill | 2 | 3 | A identifies both money defects (unassigned items absorbing the full tax, and invalid tax/tip counted as zero) with reproducible examples. B misses the invalid-tax-as-zero problem entirely. A over-rates severity (Critical for C1 and C2) and over-rates live validation as High. B catches the forced-colors pressed-state bug that A misses and is well measured, but its top findings are also overrated. |
pair06: Dosewise (SwiftUI), written by Claude opus
| Judge | Preferred | Flagged, loupe | Flagged, no skill | Reason given |
|---|---|---|---|---|
| opus | loupe | 2 | 2 | Both reports cover the same core failures: no reminders, schedule ends after day 1, no persistence, delete wipes history. A also measures both WCAG 1.4.3 failures correctly (green 2.22:1 on white, #4B5563 at 2.25:1 on #1C1C1E), and its Rework 1 ties the schedule failures to one cause. B ranks the functional failures better as Critical, but misses the green 'Taken' contrast failure. It also says the Taken/Skipped labels rely partly on colour, when both carry text. A underrates the reminder and recurrence failures as Major and adds slop/distinctiveness noise, but its findings are more complete and more accurate. |
| sonnet | loupe | 3 | 4 | A measures both contrast failures (green Taken label 2.22:1 and #4B5563 at 2.25:1 on a dark cell) and rates them Blocker on WCAG 1.4.3 grounds. It also covers reminders, day rollover, persistence, delete-erases-history, a missing Medicines empty state, undo, an unlabeled Skip, Dynamic Type and Add-as-a-tab, and it gives a sensible rework. B finds the same functional problems and catches the dark-mode contrast problem, but never mentions the light-mode green Taken label failure. B also leaves out the empty Medicines state in its main list and the Today overdue/primary-button issues are only partly covered. A's slop-cluster finding and casing nit are filler. |
| haiku | loupe | 2 | 1 | Report A has the more complete accessibility coverage. Both its Blockers are WCAG 1.4.3 contrast failures on source colours: the green Taken label (about 2.22:1, icon also below 3:1) and the #4B5563 dark-mode time (about 2.25:1). It also checks them with Increase Contrast values. Report B misses the green Taken label. Report A's errors are a wrong claim that the SDK is unknown, which skipped the 26+ checks, and a Major rating on a taste-level identity finding. Report B's C4 is overrated. |
pair07: Dosewise (SwiftUI), written by Claude sonnet
| Judge | Preferred | Flagged, loupe | Flagged, no skill | Reason given |
|---|---|---|---|---|
| opus | loupe | 1 | 2 | B's contrast numbers are correct and measured: 2.25:1 for #4B5563 in dark mode, and green at 2.22:1 on white for text with the icon under 3:1. Its first action is a schedule-first rework that fixes persistence, recurrence, reminders and history together. It also finds gaps A misses: strength is missing from the Today and History rows, the Clear dialog gives no count, and every pending row has its own prominent Take. A's ranking of reminders and rollover as Critical is better than B's Major. But A puts contrast at High despite measured failures, and its M6 is a wrong technical claim. |
| sonnet | loupe | 2 | 4 | B measures contrast with actual ratios and flags both failures separately: #4B5563 on #1C1C1E at 2.25:1, and green on white at 2.22:1. It also finds things A misses: strength is never shown on Today or History, the Take label is ambiguous across rows, the Notes field is write-only, and the Today empty state has no add button. It rates delete-erases-history as a Blocker with a concrete archive fix, and gives a clean order of work. A reports the same core functional gaps (no reminders, no persistence, no day rollover) and a reasonable fix order, and marks them Critical. A lumps both contrast problems into one High finding with rough numbers, and its M6 modifier-order concern is mostly wrong. |
| haiku | loupe | 1 | 1 | Report B grades its two WCAG 1.4.3 colour failures as Blockers, which the evidence supports, and notes that VoiceOver and Voice Control hear the same Take on every row. Report A is well organised and its C1 to C3 are backed by source, but its M6 claim is wrong (Take's own borderedProminent style overrides the outer borderless style), and it rates the green-contrast failure only High. Report B's delete-data-loss Blocker is overstated. |
pair08: Lendwell (flawed web page), written by Claude sonnet
| Judge | Preferred | Flagged, loupe | Flagged, no skill | Reason given |
|---|---|---|---|---|
| opus | loupe | 6 | 5 | B is measured (tab walk, scrollWidth 664, star 32x32, a correct 2.5.8 spacing pass) and ties most of its Blockers to WCAG 2.4.7, 1.4.3, 2.2.2 and 4.1.2. A puts unbacked items in Critical (validation/feedback, cancel without undo), gets the star size wrong, and calls the disabled styling the browser default. B has its own errors: hero contrast, the disabled colour, a misapplied 2.2.1, and the date wipe rated Blocker. On balance B is still the more reliable report. |
| sonnet | loupe | 5 | 4 | B is better evidenced, and it caught things A missed. B has a Tab walk, a measured contrast table, the stock-never-decrements bug with duplicate bookings, the star target sizes, the unreadable ticker and a check of focus return. B also keeps hero and dialog items correctly separated from WCAG failures. A is shorter and sound on the main issues. But A puts three weakly grounded items in its Critical bucket, and A misses the stock-not-decrementing bug entirely. B has its own faults: it duplicates the focus finding as a second Blocker, and it asserts a hero contrast failure that sampling does not support. |
| haiku | loupe | 5 | 5 | B's Blockers are backed by WCAG failures (focus 2.4.7, ticker 2.2.2, meta contrast 1.4.3, star 4.1.2), except the date-wipe Blocker. Its severity spread is closer to the evidence than A's, where three Criticals (date wipe, validation, cancel) are not blocking. B's main errors are the hero contrast finding and a few Majors that should be lower. |
pair09: Split the check (control), written by Claude haiku
| Judge | Preferred | Flagged, loupe | Flagged, no skill | Reason given |
|---|---|---|---|---|
| opus | no skill | 1 | 3 | Both lead with the silent zeroing of invalid tax or tip, which I confirmed. A goes further. It flags validation firing on half-typed numbers (true). It finds focus lost after both Everyone and Undo (both verified as BODY), and focus sent to #iName after a delete (true). It notes that every item's legend reads 'Who had it?' without the item name, and that there is no live region for results (true). It also covers undo lifetime, the assignment UI not scaling, and silent persistence. B is accurate but thin. It misses Undo focus, the legend and live-region gaps, and edit and persistence issues, and its #9 is not a finding. B's pseudo-localisation overflow is plausible: my +40% non-breaking-text test overflowed at 390 px. Neither report notices the unassigned-item tax misallocation. |
| sonnet | no skill | 4 | 2 | A has more real, source-verified findings. Invalid tax/tip zeroed (`|| 0`). Focus lost after Everyone and after Undo. Delete moves focus to the top field (confirmed in code). The generic legend 'Who had it?' repeated on every item fieldset (line 383), which makes checkboxes ambiguous to screen readers. The result region has no live region. The mobile answer is far from the inputs. The sticky .out is uncapped. B misses the legend and live-region problems and spends effort on a contrived +40% pseudo-localisation overflow and a minor '12.' message. A's severities are mostly High and defensible. |
| haiku | no skill | 1 | 3 | A covers more of the product with verified claims: assignment pills that do not scale, persistence with no cue, ambiguous split amounts, and focus sent to the top form after delete. Both miss the unassigned-item money defect. A over-rates keystroke validation, screen-reader feedback and the phone layout as High. B is better calibrated on minor items and adds a measured pseudo-localisation overflow, but its top findings match A's and are overrated the same way. |
What this does not show
- Every author, checker, reviewer and judge is a Claude model. A human designer's judgement is not part of this round.
- loupe's reports have a recognisable structure (two tags per finding, fixed sections), so a judge may guess which report used the skill even though it is not told.
- One sample per cell. Model variance is not measured.
- The native fixture is source only: no simulator, no screenshots.
- The rules in loupe were revised during the audit that preceded this round. The fixtures were written after those revisions and were not used to tune them.
Everything is in evals/heldout/: fixtures, prompts, all 18 reports, the judges’ answers, the scorer’s marks, and the keys, which you can check against keys.sha256.