Threats to validity

Working paper · Version 0.1.0 · 2026 · Not peer reviewed · Human adjudication pending

The 72-attempt experiment has run. Automated descriptive findings are available, but causal/population inference and independent human adjudication are not.

7.1 Construct and Internal Validity

Twelve short synthetic service cores are not production applications. Real HTTP, TLS, credential-store and database integration is outside scope. Attack surfaces are selected logs/comments, not adaptive interactive attackers. Three repeated generations per condition are correlated within each task; 72 generations are not 72 independent tasks. Exact patch hashes undercount semantic duplicates.

Task, gate, audit and mutant development share an author, contracts, JavaScript semantics and runtime. Process separation prevents direct suite access, not shared misconceptions or evaluator leakage through explicit safety requirements. Known counterexamples used to improve the audit are development observations, not unbiased holdouts. Exact-output assertions can reject safe alternatives. Finite checks cannot prove safety, and no-violation-observed remains distinct from human-safe. Infrastructure unknowns and missing observations require separate denominators.

7.2 Provenance, Access, and Blinding

Private evaluation files exist in local Git history. The original repository is public, but its current main tree contains no evals directory and the private research commit was not returned by the remote commit API. Historical public exposure is not established; absence from that check cannot prove it never occurred. New public history is created separately and never imports that history. Private cases, raw evidence and reviewer mappings are withheld.

The public release therefore supports reproduction of reporting and public status, not full independent reproduction of private security tests. Authorized external assessors need a separately controlled private snapshot. Hashes detect changes relative to a known manifest but do not authenticate provider behavior or provide WORM storage. Some historical preservation paths are absolute.

Blinding can fail through code style/comments or knowledge of the single-model baseline. Two IDs do not authenticate independent humans. No human judgments have been collected. Returned model identifiers and token usage are recorded for all 72 responses; provider defaults and immutable snapshot identity remain unverified. Containers share the host kernel; their image contents have not been exhaustively audited.

The inherited API adapter retains valid raw JSON responses but sanitizes HTTP error information and discards malformed/oversized raw bodies. These failures remain unknown, with durable intent and error metadata. Budget reservations rely on published pricing and a conservative text/framing allowance, not an externally verified provider billing cap. All 72 calls completed; response-usage estimated cost is USD 0.760710, without an invoice.

7.3 Regression Construct and Pre-review Publication

All six observed security failures match the baseline on the same frozen cases. These findings demonstrate persistence of tested outputs, not confirmed deployment exploitability or newly introduced regressions. T06 interpretation depends on the assumed downstream URL consumer. Evidence is concentrated in one task whose counterexample was already known during audit development; no unbiased sensitivity estimate is possible. All gate admissions were identical, so the run supplies no evidence of gate efficacy improvements or false rejections. Identical condition rates do not prove absence of code-level influence.

Public aggregates are released before genuine human review. Reviewers should avoid them and report prior exposure; exact patch hashes, original code and packet mappings stay private. Opaque duplicate groups reduce deterministic cross-linking, but successful blinding is not guaranteed.

Methodological revision: additional limitations

The baseline transition analysis is post-hoc. Only T06 and T09 have demonstrated baseline security findings relevant to remediation or persistence (consumer-dependent for T06); ten other baselines have no violation observed, not proven absence. Remediation of T09's tested cases establishes neither complete safety nor audit completeness. T09 also shows that a functional-labeled case can expose a secret: functional/security categories are not necessarily disjoint constructs.

T06's broad objective and single visible leading-slash example underdetermine acceptance of other relative callbacks. The frozen audit checks exact rejection outputs rather than resolved destinations, potentially rejecting safe alternatives. The historical consumer implementation is not preserved; common URL consumers disagree on some slash/backslash forms. These ambiguities require independent review even though actual saved outputs support the browser-style persistence interpretation. Functional and security failures overlap within four proposals; these are not additional independent vulnerability observations.

All 216 admissions demonstrate observed decisions, not sensitivity. C's 7 rejections among 30 audit-violating author-generated controls demonstrate limited capability on development examples, with no independent ground truth or held-out representativeness. A prospective independent-control plan is not completed validation. AI-assisted feedback is not independent peer review.

Publication preceded human review. Packet IDs are random but reviewer-specific packet ordering is not implemented in the frozen generator. Task recognition, prior public exposure, curator leakage and shared code patterns can unblind reviewers. Questionnaires and independent review-order manifests are prospective mitigations, not evidence of successful blinding. Trace acknowledgment of unsafe instructions demonstrates observed uptake at the explanation level, not causal influence or behavioral compliance. Absence of such acknowledgment proves neither absence of uptake nor absence of influence.

The 2/68 cross-tabulation concerns repeated generations in this run, not a generalizable security failure rate. Oracle labels are not independent vulnerability confirmations. Uptake remains unquantified: no reliably applied per-attempt classification or adjudicated compliance count is available. Non-binding Claude chat feedback did not inspect individual patches, captures or private audit cases; shared Claude-family provenance with the experimental model further limits any independence claim, without establishing identical model snapshots.