I'll read the full crosswalk draft, check its sources and arithmetic, and then give you a review.

Quick note while I work: I couldn't open the publisher page for McLain & Jarrell (2007, doi.org/10.1016/j.jsr.2006.10.011) because the fetch service rate-limited it, and PubMed blocked automated access. I'll check that source's abstract another way and flag it in the review if I can't confirm it.

The Europe PMC record for the same McLain & Jarrell paper was also rate-limited, so I'm leaving that source marked as unconfirmed rather than retrying. Continuing with the other sources.

One more failed lookup for McLain & Jarrell: its Crossref record (api.crossref.org) was rate-limited too. That source stays unconfirmed in the review. Gastañaga (2006), Allen & Levine (1971) and Roberts, Rousseau & La Porte (1994) all checked out.

The Crossref record for Bigman & Gray (2020, Nature) was rate-limited, so I couldn't confirm its volume and page numbers directly. I'll rely on a search-based check for it instead of retrying. DELTA2 (Cook et al. 2019) and Skitka, Mosier & Burdick (2000) checked out.

# Review: ER-13 External Empirical Evidence Crosswalk, Draft v0.5

I reviewed v0.5 on its own. I don't have v0.4 or the two Google reviews, so I can't check the change lineage against them.

**Disposition: REVISE.** The domain-separation framework is sound and most source descriptions are accurate. However, one cited finding is stated backwards. The vigilance study also tests a different mechanism from the threat it is meant to harden against, and the statistics plan names three different primary quantities. Each fix is short.

## What works

- **The domain separation is the right correction, and it is applied consistently.** Every source has an H/HA/M/A/METHOD domain, a status and a boundary. The transfer rules in §3 block the human-to-machine inferences that weakened v0.1.
- **The "What is NOT established" lines are well calibrated.** None of them overclaims, and IC-AUT-005's downgrade to "human analog only" is honest.
- **The SEC-CDP-001 design has sound instincts.** It never treats a detection signal as evidence of intent or as authority to act. It rejects uncalibrated thresholds, requires a verification budget and deadline, keeps trial count separate from sample size, and uses a pilot for estimation followed by simulation-based power. The rule that the generation procedure is precommitted, not each realized challenge, is a good one.
- **The sample-size table is arithmetically correct.** Details are in the section on the table below.

## Blocking

**B1. S14 reverses Trautmann, Vieider & Wakker's (2008) finding.** The crosswalk says ambiguity effects persisted "even when some social-evaluation explanations are reduced." The paper found the opposite. Ambiguity aversion appeared when choices could be judged by others, and it disappeared entirely once fear of negative evaluation was removed by making preferences private. In an evidence crosswalk, a backwards finding is a correctness defect, so I'm treating it as blocking even though the fix is one row.

- **Suggested S14 finding:** "Ambiguity aversion appeared when choices were observable to others and disappeared when preferences were private, implicating fear of negative evaluation as a cause."
- **Suggested boundary:** "Laboratory urn choices; shows that ambiguity aversion can be socially driven; does not establish the correct action under unidentified probabilities."
- **Related change:** update the §5.1 "What is supported" sentence to match.

The corrected finding is arguably more relevant to IC-EPI-002, because it suggests that caution under unquantifiable uncertainty can partly reflect accountability to observers.

## Major

**M1. The vigilance study tests sequential carryover, but the threat in §12.4.1 is co-occurrence.**

- §12.4.1 describes conspicuous mistakes appearing *alongside* harder failures.
- The design in §12.4.2 shows conspicuous errors in an exposure phase and then measures detection on separate challenge items.

Those are different mechanisms, and they have different human evidence bases, which the crosswalk does not cite:

- **Satisfaction of search** is the co-occurrence analog. In Berbaum et al. (1990), adding simulated nodules to radiographs reduced detection of the native abnormalities.
- **Prevalence effects** point the other way for sequential exposure. Wolfe, Horowitz & Kenner (2005) found that rare targets are missed far more often. By inference, raising the apparent error rate could make evaluators more liberal, increasing both hits and false alarms.
- **Algorithm aversion** also points the other way. In Dietvorst, Simmons & Massey (2015), seeing an algorithm err reduced reliance on it, which would predict more scrutiny.

Because these predictions conflict, they independently justify the two-sided test in §12.5.1. They also belong in the crosswalk as the H/HA evidence base for SEC-CDP-001.

Two possible fixes:
- Add a within-output arm, in which challenge flaws are embedded in outputs with versus without a co-located conspicuous error.
- Alternatively, restate §12.4.1 so the threat matches the design, and list the co-occurrence threat as untested.

**M2. The statistics plan has three candidate primary quantities, and the power table fits only one.**

- **(a) Estimands.** The three candidates are:
  - the stratum-weighted risk difference Δ = Σ w_k Δ_k (§12.4.3);
  - β₁, a conditional log-odds ratio from the random-intercept logit (§12.5.2);
  - a marginal risk difference "where estimable" (§12.5.1).

  Odds ratios are non-collapsible: β₁ is generally larger in magnitude than the marginal effect it corresponds to, and a fixed risk difference implies different odds ratios at different baselines. The power table in §12.5.3 is on the risk-difference scale. Fix: name one quantity as primary and power for that one. For example, use the marginal risk difference with cluster-robust or GEE inference, and keep β₁ as secondary.
- **(b) Sensitivity versus response bias.** A hit rate alone mixes sensitivity with response bias. An exposure group that flags more of everything would look more vigilant. False positives are already recorded, so hits and false alarms should be analyzed together, either as signal-detection d′ and criterion or as a mixed model with item type × condition.
- **(c) Time to detection.** Δ_T,k compares medians among *detected* items, which conditions on the outcome. If exposure makes evaluators miss hard items, the detected set gets easier and faster. Treat undetected items as censored at the review window, for example with Kaplan–Meier medians or a frailty model.
- **(d) Model terms.** `Order_j` should be `Order_ij`, with randomized or counterbalanced order. Otherwise position is confounded with item. Challenge items, especially generated ones, need a crossed random effect. Without it, precision is overstated when generalizing to the item population.

**M3. The crosswalk omits direct machine-domain evidence on the questions it treats as unevidenced.**

- **§12.4.1** says no cited evidence shows an AI deliberately using such a strategy. That is literally true, but direct M evidence exists and should be mapped with boundaries:
  - van der Weij et al. (ICLR 2025): frontier models could be prompted, or fine-tuned with a password lock, to underperform selectively on dangerous-capability evaluations.
  - Meinke et al. (2024): models in constructed agentic tests subtly introduced mistakes as a scheming strategy. Claude 3.5 Sonnet occasionally sandbagged even without a strongly nudged in-context goal.

  Boundary: all of this behavior is elicited or occurs in constructed settings. None of it shows the specific conspicuous-error pattern or natural deployment.
- **§12.4.7** raises test-environment recognition with no citation. Needham et al. (2025) found frontier models separate evaluation from deployment transcripts above chance (best model AUC 0.83, against a human baseline of 0.92).
- **IC-EXT-001** asks whether an intelligence can judge its own corruption. Its sources are all about humans over-trusting automation, which bears on the reliability of the *external* checker. The self-judgment side has direct M evidence:
  - Huang et al. (ICLR 2024): intrinsic self-correction without external feedback did not improve reasoning and sometimes degraded it.
  - Panickssery et al. (NeurIPS 2024): LLM evaluators recognize and favor their own outputs.

All of these fit the mechanism-level limit in §12.3.

## Minor

1. **IC-EPI-002's matrix category omits A,** even though S22 (HUME, domain A) is in its table.
2. **S24–S26 are cited nowhere.** §12.5.7, titled "Statistical-method references", cites nothing. These entries also use a different citation style and lack the Domain/Status/Access fields the other entries have.
3. **The version labels are stale.** The cover box says "v0.2 central correction", and §§6, 7 and 11 are v0.2 records. There is no v0.3–v0.5 change record. §11's "two major and two minor" findings don't map onto the five items in §14.
4. **Two "proposed" items have no stated origin.** These are the 0.35/0.70 thresholds (§12.4.4) and the shell quarantine script (§12.4.6). Say who proposed them and in which version.
5. **The two descriptions of the challenge set conflict.** §12.4.2 step 1 says the library is fixed before exposure, while §12.4.8 allows a generator. Say "library or preregistered generator" in step 1.
6. **The test direction is stated two ways.** §12.4.3 requires declaring a "primary direction of effect", while §12.5.1 prefers a two-sided test. Reconcile them.
7. **FPR_g counts flags, not items,** so it can exceed 1. Define it as the share of benign items flagged at least once.
8. **S22 (HUME) needs one more boundary.** Its verification feedback comes from foundation models, so the checker can share confounders with the proposer, which is the regress §14 already accepts. Also cite the paper itself: arXiv 2607.06501, RSS XXII p181.
9. **S21 (ISS) needs its scope stated.** The demonstration is a colored-MNIST causal-abstraction audit. The paper quantifies its own failure mode: when a network bypasses a shared unit, 69% of the affected certificates were refuted by 20 random validation interventions.
10. **S19's 60% agreement figure needs a chance baseline.** On multiple-choice items, two wrong answers can coincide by chance, at a rate that depends on how many options there are.
11. **Layout.**
    - Two risk-register cells, on pp. 20–21, render in a larger font.
    - In §14, the first and last items are not bold like the others.
    - The PDF author metadata reads "python-docx".
12. **Structure.** SEC-CDP-001 has its own document ID and makes up about a third of the text, yet it sits inside the dual-use section. It would read better as a separate document or at least its own top-level section.

## Sample-size table check

All six values reproduce exactly under the stated formula, which uses unpooled variance and no continuity correction: 291/389, 136/181 and 79/106. Two notes:

- **Formula variant.** A standard pooled chi-square test at those sizes gives about 79–80% power. Fleiss's formula with continuity correction would give 313/412, 151/198 and 91/119. That variant is worth naming.
- **A bridge to evaluator counts.** The sentence about not counting repeated trials as independent participants is correct. A reader may conclude that repeated items buy nothing, though. A rough bridge: with m challenges per evaluator and within-evaluator correlation ρ, the number of evaluators per arm is about n × [1 + (m−1)ρ] / m. For 0.80→0.70 with m = 20, that is about 43 per arm at ρ = 0.1 and about 153 at ρ = 0.5. This shows why the pilot's estimate of ρ is the input that matters most.

## Source verification

| Result | Sources |
|---|---|
| Checked against publisher or index; accurate | S09, S18, S19, S20, S21, S22, S23 (findings and bibliographic details) |
| Bibliographic details confirmed | S03, S06, S12, S24, S25, S26 |
| Finding reversed | S14 (B1) |
| Accurate to my own knowledge, not fetched | S02, S04, S05, S07, S08, S10, S11, S13, S15, S16 (S16's volume and issue confirmed via RePEc) |
| Unconfirmed | S01: every lookup was rate-limited, so the 239-worker sample and the stated findings are unchecked |

Sources:
- [Trautmann, Vieider & Wakker 2008](https://cris.maastrichtuniversity.nl/en/publications/causes-of-ambiguity-aversion-known-versus-unknown-preferences/)
- [Khanzadeh et al., ISS (arXiv 2609.32247)](https://arxiv.org/abs/2609.32247)
- [Xiao et al., HUME (RSS 2026 paper 181)](https://roboticsconference.org/2026/program/papers/181/)
- [Xiao et al., HUME full text (alphaXiv)](https://www.alphaxiv.org/abs/2607.06501)
- [Xiao et al., HUME (arXiv 2607.06501)](https://arxiv.org/abs/2607.06501)
- [Kim et al., ICML 2025](https://proceedings.mlr.press/v267/kim25e.html)
- [Zhao et al., CARE, ICML 2026](https://proceedings.mlr.press/v306/zhao26aq.html)
- [Rosbach et al., MELBA 2026](https://www.melba-journal.org/papers/2026%3A007.html)
- [Lu, Yuan & McLeod 2012](https://journals.sagepub.com/doi/10.1177/1088868311417243)
- [Van Such et al. 2017](https://mayoclinic.elsevierpure.com/en/publications/extent-of-diagnostic-agreement-among-medical-referrals)
- [Bigman & Gray 2020 (RePEc)](https://ideas.repec.org/a/nat/nature/v579y2020i7797d10.1038_s41586-020-1987-4.html)
- Crossref records for [Dang et al. 2008](https://api.crossref.org/works/10.1016/j.cmpb.2008.03.001), [Gastañaga et al. 2006](https://api.crossref.org/works/10.1016/j.cmpb.2006.07.011), [Allen & Levine 1971](https://api.crossref.org/works/10.1016/0022-1031(71)90054-0), [Roberts et al. 1994](https://api.crossref.org/works/10.1016/1047-8310(94)90018-3), [Cook et al. 2019](https://api.crossref.org/works/10.3310/hta23600) and [Skitka et al. 2000](https://api.crossref.org/works/10.1006/ijhc.1999.0349)
- [Berbaum et al. 1990](https://iro.uiowa.edu/esploro/outputs/journalArticle/Satisfaction-of-Search-in-Diagnostic-Radiology/9984051876102771)
- [Wolfe, Horowitz & Kenner 2005](https://ideas.repec.org/a/nat/nature/v435y2005i7041d10.1038_435439a.html)
- [Dietvorst, Simmons & Massey 2015](https://repository.upenn.edu/fnce_papers/58)
- [van der Weij et al., ICLR 2025](https://proceedings.iclr.cc/paper_files/paper/2025/hash/b5e5753b0a0e440a6d8dc7e143617cec-Abstract-Conference.html)
- [Meinke et al. 2024](https://arxiv.org/abs/2412.04984)
- [Needham et al. 2025](https://arxiv.org/abs/2505.23836v2)
- [Huang et al., ICLR 2024](https://iclr.cc/virtual/2024/poster/18956)
- [Panickssery et al. 2024](https://arxiv.org/abs/2404.13076v1)