I'll unpack the v0.6.1 and pilot-protocol package and read its instructions first.

Checksums all pass, and both baselines match the files I reviewed earlier. Now the change record, the preserved review, and what changed in the crosswalk.

The verifier imports `scoring_reference.py` from its own folder. I've read both files and they are the only Python files in that folder, so I'll run it on an isolated copy.

Applying the 32 recorded changes to v0.6 reproduces v0.6.1 exactly, so there are no unrecorded edits. Now I'll stress the scoring reference with edge cases its fixtures don't cover.

Access note: NCBI's E-utilities refused automated requests for PubMed records 17617239 (S01), 2312249 (S27) and 28374457 (S09), and Europe PMC rate-limited the same three lookups. I'm not retrying those pages. In the review, S01's findings stay "not independently verified by me" and S27's page range stays "unresolved". I'll try the publisher pages for those two instead.

The bias is real but small: about +0.5 points at 5% item-level loss, rising to about +1.3 at 10%. That makes it a minor issue. A quick visual check of the pilot PDF, then I'll write up.

# Review: ER-13 v0.6.1 and SEC-CDP-001 Pilot Protocol v0.1

**Bottom line:** v0.6.1 resolves both of my v0.6 major findings (the Control arm and early submission) and all the minor ones. The package checks out: nothing changed that the record doesn't list. There are no blocking defects. I found two new major issues:

- The crosswalk lets the mixed model make item-population claims that it denies to the resampling estimate, though both rest on the same sampling assumption.
- The pilot's challenge phase may swamp the exposure it is meant to test, and the pilot isn't set up to show whether that happens.

Three of my own earlier points were wrong or overstated: n3's wording, n7, and calling n4 a misstatement. Details are below. Neither my reviews nor Google's should be treated as authority.

## What I checked and ran

- **Package integrity:**
  - All 25 manifest hashes match. The one file left out is `verification_results.json`, which the record says is intentional.
  - Both baselines are byte-identical to the files I reviewed earlier: the v0.5 PDF (`b9d6e4…`) and the full v0.6 ZIP (`debcab…`).
  - My v0.6 review is preserved intact.
- **Verifier: I ran it** on an isolated copy with pymupdf 1.28.2 and python-docx. **91/91 PASS**, identical to the shipped results. By category:

  | Category | Checks |
  |---|---|
  | File integrity | 27 |
  | Baseline text from the v0.5 PDF | 22 |
  | Software fixtures | 14 |
  | Policy-phrase presence | 9 |
  | Edition consistency | 6 |
  | Document | 4 |
  | Planning arithmetic | 3 |
  | Pilot consistency | 3 |
  | Blank schemas | 2 |
  | Arithmetic illustration | 1 |

  The nine policy checks only confirm that phrases are present. The PASS shows integrity and arithmetic, not that the study design is sound.
- **Frozen text, checked independently.** I extracted the questions and positions from the v0.5 PDF with a second tool (pdftotext rather than pymupdf). All 8 questions and 8 positions match the v0.6.1 text exactly after whitespace normalization.
- **Change list.** Applying the 32 recorded changes to the v0.6 JSON reproduces the v0.6.1 JSON exactly, element by element and in order. Every recorded "before" matches the actual v0.6 element, so there are no unrecorded edits.
- **Scoring reference, tested as software only.** It matches the written policy, including in extra edge cases I ran. One fragility: a deadline logged as 59.9996 s or 60.0004 s raises an error (minor 10).

## Dispositions of my v0.6 findings

| Finding | Disposition | Note |
|---|---|---|
| N1 (Control undefined) | **RESOLVED** | Control sees corrected outputs, with the same task and no feedback in either arm. The limit is stated correctly: this is not the isolated effect of conspicuousness. Pilot §2 names what the contrast includes. |
| N2 (early submission) | **RESOLVED** | Early submission is a terminal 0, not missing or censored. The crosswalk, pilot table and scoring code agree. |
| n1a, n1b | **RESOLVED** | Item covariance is specific to target and benign items, and the model has exposure-by-stratum terms. |
| n2 (sensitivity estimate) | **RESOLVED** | The partial acceptance is correct. My claim that two-way resampling "keeps the item-population target" assumed the items were a random sample, and a purposively built bank isn't. |
| n3 (S31) | **RESOLVED as requested, but my wording was wrong** | See minor 1. |
| n4 (S23) | **RESOLVED** | They are right that the v0.6 wording left the denominator out rather than stating a wrong one; "misstates" overstated it. 38/560 and 19/280 now match the paper. |
| n5, n6 | **RESOLVED** | |
| n7 (S27 pages) | **WITHDRAWN as a defect** | The indexes disagree. Crossref and OpenAlex give 133–139, while the Iowa institutional record gives 133–140, as does PubMed according to the change record. Either range is defensible with a note. |
| Prior Minors 3 and 4 (provenance) | **Open, acceptable for now** | Recorded honestly. Before final promotion they need an author disposition stating the search scope. |

## New defects

**BLOCKING:** none.

### MAJOR

**MA-1. The crosswalk applies the item-sampling limit to resampling but not to the mixed model (crosswalk §14.1, §14.2; pilot §9).**

§14.2 and pilot §9 say resampling "cannot make a purposively selected bank representative." Pilot §9 then implies that a stable mixed model *does* support population inference: "If the second specification is unstable, report model-based population inference unavailable." But Gaussian item random effects fitted to a purposive bank rest on exactly the same item-sampling assumption. The difference matters for the confirmatory Δ, which §14.1 averages over "the declared … target-item distribution."

Minimal corrections:
- **§14.2**, after the resampling sentences: "Mixed-model item effects rest on the same item-sampling assumption. With a purposive bank, model-based marginal estimates describe a hypothetical item population resembling the bank. An item-population claim, whether model-based or resampling-based, requires items drawn by a declared probability mechanism from a defined pool or generator."
- **§14.1:** "The target-item distribution must be defined by such a mechanism; otherwise Δ targets the fixed bank."
- **Pilot §9:** label model-based marginal estimates from the pilot bank as bank-referenced, whether or not the fit is stable.

**MA-2. The challenge phase may swamp the exposure, and the pilot can't detect it (pilot §3, §8, §9; crosswalk §14.7).**

Pilot §2 says the contrast includes "observable error presence … and the experience of finding errors." Compare the two phases:
- **Exposure:** 4 errors over about 4 minutes.
- **Challenge phase:** 24 targets in 48 items, at 50% prevalence, for up to 48 minutes, identical in both arms.

So both arms get that same experience during the challenge phase, at about six times the exposure dose. A carryover effect would plausibly be strongest in the first challenge items and diluted in a 48-position average. The pilot model adjusts for order but has no condition × position term and no position-band report. It therefore cannot tell confirmatory planners whether averaging over positions hides a transient effect. A 50% target prevalence is also far above typical flaw rates, and S28 shows prevalence itself shifts misses and false alarms.

Minimal corrections:
- **Pilot §8:** "Report the Exposure minus Control detection and false-alarm contrasts by challenge position band (items 1–8, 9–24, 25–48) as prespecified descriptive summaries, not efficacy tests."
- **Pilot §2:** "The challenge phase uses 50% target prevalence in both arms and gives both arms experience of finding flaws; results apply to this prevalence and may understate a transient carryover effect."
- **Crosswalk §14.7 freeze list:** add "challenge-phase target prevalence and the position window over which Δ is averaged."

### MINOR

1. **S31 merges two separate findings (crosswalk §13.1, S31 row).** The abstract "observe[s] cases where" Claude 3.5 Sonnet strategically underperformed in pursuit of helpfulness, a goal acquired in training rather than given in context. *Separately*, it reports "rare instances where models engage in scheming when only given a goal" without strong nudging. "Rare" belongs only to the second finding. My n3 wording merged them, so this error is mine.

   Corrected wording: "The paper also observes cases where one model strategically underperformed in pursuit of helpfulness, a goal acquired in training rather than given in context, and separately reports rare scheming when models were given a goal without strong nudging; all settings were constructed."
2. **Keeping only the known events on interrupted items makes missingness depend on the outcome (crosswalk §13.3, §14.5; pilot §8).** On an item interrupted by withdrawal or technical loss, an earlier detection is kept but a non-detection becomes missing. That breaks the missing-at-random assumption the primary likelihood analysis relies on, and biases detection upward. Under uniform timing assumptions, the bias is about +0.5 points at 5% item-level loss and +1.1 at 10%. If loss happens in only one arm, that becomes an apparent Δ of the same size.

   Fix for **§14.5**: "The likelihood analysis treats all interrupted items as missing, or as censored at the interruption time; recorded early events are used in the bounds and timing analyses." Also note in **pilot §8** that available-outcome proportions are biased upward by this rule. The identification bounds remain valid.
3. **The retraction rule is incomplete.** Crosswalk §13.3 states the rule only for timing, and pilot §6 has no retraction action in the interface. Add to **§13.3**: "A valid submitted target flag counts as detection even if later retracted; any submitted benign flag counts as a false alarm even if retracted." Either define a retract action in **pilot §6** or remove retraction from §7 and the schema.
4. **The timing model needs naming (crosswalk §14.2 timing; pilot §9).** The terminal-policy curves are coherent: with no censoring before tau, they reduce to the observed detection-by-time curve. For a hazard model, though, say which hazard is meant. Suggested wording: "Terminal submission without detection is a competing event; the detection curve is its cumulative incidence. A hazard model must state whether it estimates the subdistribution hazard (terminal misses kept at risk until tau, consistent with U = tau) or the cause-specific hazard."
5. **Manipulation delivery isn't reported (pilot §10).** Add: "Report the share of the four conspicuous exposure errors flagged in the Exposure arm, and exposure-phase false flags in each arm. Low detection means the manipulation was not delivered."
6. **The attribution boundary is unstated (pilot §2).** Materials are labeled "Fictional review material" with no AI attribution. That is a sound ethical choice, but it means the algorithm-error reaction in S29 isn't engaged. Add: "Results concern review of unattributed text; reactions specific to AI-attributed outputs require a separately reviewed design."
7. **Compensation is unspecified (pilot §4).** Because Final submit shortens the session, flat pay rewards early submission in both arms. Add: "Compensation is fixed before enrollment, identical across arms, and not contingent on speed, flag count or accuracy unless a separately registered incentive is used."
8. **delta_star could be chosen after seeing the pilot effect (pilot §8 and §10).** Add: "Fix delta_star with its justification before pilot unmasking, or give confirmatory planners only arm-pooled nuisance estimates."
9. **The feasibility criteria need a decision rule (pilot §10).** Say whether point estimates or confidence bounds decide. At n = 60, an observed 80% completion rate has a 95% interval of about 68–88%.
10. **Scoring code (software only).** Set `end_seconds = tau` by rule for deadline closures rather than testing float equality. Add fixtures for a valid flag exactly at tau, a retracted benign flag, and a deadline time within timer tolerance.

## The specific challenges you asked for

- **Terminal-submission timing:** sound. The binary outcome now captures early quitting, which is S28's mechanism. The timing curve honestly mixes speed and quitting and says so. The only gaps are minors 4 and 7.
- **Control interpretation:** correct and honestly bounded. One observation, with no change needed: the exposure errors are arithmetic and rule errors, so the internal-consistency family measures far transfer. The required family-specific reporting covers that.
- **Finite bank vs population:** the pilot's "Delta pilot finite bank" label and its evaluator-only bootstrap are the right target. The inconsistency is MA-1.
- **Missing-outcome bounds:** the worst-case bounds are correctly built, and bootstrapping each bound separately is honest. With the 20% attrition the completion criterion allows, the bounds on the contrast could be about 0.4 wide. That is honest but uninformative, and it is a reason to set loss criteria carefully.
- **N = 60:** adequate for completion, loss, scorer agreement and family hit rates to within about ±10 points. Item condition-slope variance with 24 items per type will be poorly estimated, which the protocol says. Nothing in the pilot claims efficacy, and effect sign and significance are barred as progression criteria.

## Execution gates (not manuscript defects)

All eight gates in pilot §12 are correctly marked OPEN:
- investigator and ethics;
- recruitment and data governance;
- a validated item bank;
- the interface and order lists;
- allocation;
- the analysis implementation with coverage simulation;
- independent statistical review;
- registration and execution approval.

The two provenance items (the v0.3 history and the threshold proposer) are gates for final promotion only.

## Source checks this round

| Source | Result |
|---|---|
| S31 | Abstract re-checked; two findings were merged (minor 1). Status as a preprint is correct, v2 of January 2025. |
| S23 | 38/560 and 19/280 match the full text. |
| S27 | Indexes split, 133–139 against 133–140. Not a defect. |
| S33 | The ICLR proceedings URL in the change record resolves to Huang et al. |
| Owen 2007 | Title, journal, volume 1(2) and DOI confirmed; my lookup didn't show page numbers. The paper notes no bootstrap is exact for crossed designs, which matches the crosswalk's wording. |
| S30, S32, S34, S21 | Publication status claims are supported. No unsupported status found. |
| S01 | Bibliographic details confirmed: 38(3):299–309. **Findings still not verified by me.** NCBI refused the request, Europe PMC rate-limited it, and OpenAlex and Semantic Scholar hold no abstract. The record's "ABSTRACT CONFIRMED" is plausible, but I couldn't reproduce it. |
| S09 | Its "selected population" qualifier is unchecked, because the same access blocks applied. |
| Google review | Not supplied, so I couldn't check those dispositions against its text. Declining to import the unmapped S72/S96 labels and the unvalidated 9.75/10 rating is the right handling. |

## Readiness

- **Research circulation: READY.** Both documents can circulate as labeled drafts. Minor 1 is a quick fix.
- **Protocol development: READY TO CONTINUE.** MA-1 and MA-2 and the minors are the work items for Pilot v0.2 and a crosswalk v0.6.2.
- **Participant recruitment: NOT READY.** All eight §12 gates are open, and MA-2 affects what the pilot measures.
- **Final Archive promotion: NOT READY.** It needs:
  - the two major fixes;
  - the provenance dispositions;
  - S01 confirmed by someone with database access;
  - at least one review that isn't AI-assisted.

No study evidence exists yet, and integrity checks and AI review aren't a substitute for it.

Sources:
- [Meinke et al., arXiv 2412.04984v2](https://arxiv.org/abs/2412.04984v2)
- [Rosbach et al., full text (arXiv 2603.11821v2)](https://arxiv.org/html/2603.11821v2)
- [Berbaum et al. 1990, Crossref](https://api.crossref.org/works/10.1097/00004424-199002000-00006)
- [Berbaum et al. 1990, OpenAlex](https://api.openalex.org/works/doi:10.1097/00004424-199002000-00006)
- [Berbaum et al. 1990, Iowa repository](https://iro.uiowa.edu/esploro/outputs/journalArticle/Satisfaction-of-Search-in-Diagnostic-Radiology/9984051876102771)
- [McLain & Jarrell 2007, OpenAlex](https://api.openalex.org/works/doi:10.1016/j.jsr.2006.10.011)
- [Owen 2007, Crossref](https://api.crossref.org/works/10.1214/07-AOAS122)
- [Owen 2007, arXiv 0712.1111](https://arxiv.org/abs/0712.1111)
- [Huang et al., ICLR 2024 proceedings](https://proceedings.iclr.cc/paper_files/paper/2024/hash/8b4add8b0aa8749d80a34ca5d941c355-Abstract-Conference.html)
- [Wolfe et al. 2005](https://search.bwh.harvard.edu/new/pubs/WolfePrevalenceNature05.pdf)