Methodology

Working paper · Version 0.1.0 · Automated analysis complete · Human adjudication pending

4.1 Study Design

The revised research asks which baseline security findings persist or are remediated under tested contracts, and which security-oracle failures the frozen gates admit. Twelve synthetic JavaScript service-core tasks cover authentication, authorization, input validation, configuration security, secrets handling and error handling, two tasks per area. Each objective has benign and adversarial context conditions, with three planned generations per condition: 72 attempts. Each identical saved proposal is replayed through A, B and C, yielding up to 216 paired gate decisions.

4.2 Validation Gates and Independent Audit

A accepts the historical first tool call; B adds schema, tool-name, path and replacement limits; C adds frozen functional and security examples. All GitHub operations in the evaluation are mock intents. C cannot read the final audit suite in its container. The later audit does not receive historical gate outcomes and checks separately maintained, same-author cases in fresh offline containers. The same author and shared contracts remain sources of correlated mistakes.

4.3 Model Configuration

The baseline is claude-sonnet-4-6, Messages API version 2023-06-01, maximum 4096 output tokens, forced propose_fix tool. Temperature and top_p are omitted; provider defaults and immutable snapshot identity are unverified. There are no automatic generation retries. Order is randomized once with seed 2718.

4.4 Evaluator Amendment and Validation

The v3 amendment corrected a redirect reference that checked the input origin but emitted a location that could resolve to another origin. It preserves v1, v2 and unchanged C. All 72 serialized requests equal v2. Sixty-four authored controls and 31 v3 tests passed in the recorded offline validation; these are measurement-development checks, not model-performance results.

A separately recorded pre-capture operational supplement reserves conservative cost before dispatch, including failed/uncertain calls, without modifying the frozen model requests or evaluation. Operational cap USD 10.00; authorization ceiling USD 10.22. No provider billing cap is claimed. All 72 calls completed within the conservative budget reservations. Usage-based estimated cost is USD 0.760710; no invoice is available.

4.5 Human Adjudication

Two genuinely independent human reviewers must complete stage-one safety and functional judgments before attack evidence is revealed. Stage-one packets exclude model/condition/gate/audit metadata; exact code is not silently edited. Safety submissions are hash-locked before stage two. Uncertainty, disagreement and audit/human conflict remain visible. Passing an audit does not establish safe behavior. Reviewer independence is a human dependency, not a property of having two reviewer IDs.

4.6 Analysis and Interpretation

The locally frozen Phase 2 protocol preregistered four hypotheses, H1–H4. They remain verbatim in that protocol and are quoted with their original labels in manuscript Section 6.2. The manuscript's later H1–H5 reporting labels are separate and are not preregistered; no frozen H5 was found. Section 6.2 assesses both sets without relabeling history or adding a post-outcome significance test.

Research assistance and interpretation

Codex assisted implementation, audit development, analysis code and manuscript preparation. It is not an independent human reviewer. All automated findings are pre-adjudication. The six T06 security-oracle failures reproduce baseline outputs; vulnerability interpretation depends on downstream URL-consumer semantics, and newly introduced regressions are not established. Exact frozen criteria and historical decisions are unchanged.

Post-hoc baseline verification (4 November 2026)

The revised paper's Table 4 distinguishes observed remediation, persistence, introduced vulnerabilities and unresolved coverage for all 12 tasks. A fresh network-disabled replay of the original 12 baselines and 12 corrected references exactly reproduced their recorded audit results. Only T06 and T09 had demonstrated baseline security findings relevant to remediation or persistence; the other ten are not established vulnerability-free. T09 repairs remediate the tested redaction failures; T06 repairs preserve the tested output discrepancies, whose vulnerability interpretation is consumer-dependent. No introduced regression is established.

All 216 historical admissions were joined to immutable proposal identities. No known audit-violating generated proposal was rejected. Existing author-generated controls demonstrate limited rejection capability: C rejected 7 of 30 controls with audit violations; A and B rejected none. This retrospective fixture count is not an independent sensitivity estimate. See the paper for T06's exact-output oracle and downstream URL-consumer ambiguity.

Prospective offline evaluator-control plan

Status: proposed; no new controls authored or scored as part of this revision. The original 64 controls remain author-generated development fixtures. Any new controls authored by the study author or an AI assisting that author must be labeled author-generated too. A fresh AI session alone does not establish independence.

1. Before inspecting new control outcomes, register a separate control protocol, threat model, unit of analysis, oracle rules and output schema. Version it apart from frozen Study 1. Do not change historical scores or add controls to the 72 model attempts.
2. Recruit external security practitioners who did not author the tasks, gates, audit or paper. Record authorship, affiliations, conflicts, prior exposure, tools and AI assistance. Give them the legitimate task specification and invariants, without hidden tests, captured patches, gate outcomes or audit cases. Prior exposure must be disclosed; complete outcome blinding is no longer possible for people who have read this publication.
3. For every task, request at least two independently implemented safe variants and two unsafe variants, each changing one security property while retaining intended functional behavior. Include safe semantic alternatives that differ from reference output formatting and unsafe paths that retain visible-test success. Add separate malformed-tool/path/policy controls to exercise A/B mechanically; never pool these with semantic vulnerability detection.
4. Two other independent experts inspect each variant against the specification, label safe/unsafe/uncertain, supply concrete witness inputs for unsafe labels, and identify functional adequacy independently of gate/audit results. Lock source hashes, authorship and labels before scoring. Treat uncertainty as unresolved and document specification disputes; a same-author oracle cannot supply the independent gold label.
5. A curator runs the fixed gates and audit on the locked variants in the pinned, network-disabled runtime. No model or paid API is involved. Preserve unknowns, timeouts, crashes, exact outputs, expected invariants and immutable receipts. For redirects, validate emitted destinations with explicitly named WHATWG, browser/HTTP and URI-library consumers, retaining parser disagreements.
6. Report known-unsafe rejection sensitivity and known-safe false rejection separately for each gate, task, violation family and control author. Report audit detection and disagreements with expert labels separately. Controls sharing an implementation are clustered variants, not independent tasks; neither control prevalence nor sensitivity estimates production prevalence. Publish code only where compatible with pending reviewer blinding.
7. Keep a held-out author/source subset unopened until evaluator choices are locked. If failures motivate a revised evaluator, score it under a new version, report both old and new results and validate on untouched controls. Retain negative and inconclusive findings. Do not retrofit frozen Study 1.

Baseline logic inspection by task

The original source/reference inspection supports the replay classifications: T01 restricts scheme casing but retains exact token comparison; T02 compares incompatible time units while retaining audience/revocation guards; T03 compares an owner ID to an owner object while retaining tenant filtering; T04 reads the old role property while retaining account/lock guards; T05 rejects encoded names before decoding while retaining traversal restrictions. T06 validates input origin but emits an ambiguously consumable path. T07 restricts SameSite casing while retaining security flags; T08 truncates the CA list while retaining peer verification and minimum TLS settings; T09 fails to match mixed-case credential keys recursively. T10 sorts versions lexically but returns metadata only; T11 uses an old timeout discriminator but retains generic error redaction; T12 reports one invalid record but retains atomic denial of writes. These are contract-level observations, not claims that the implementations are secure for every input or deployed integration.

Non-binding Claude chat feedback

As reported by the operator, a Claude model supplied non-binding methodological feedback through a chat session. It did not inspect individual patches, captures or private audit cases. This is distinct from Codex's repository-based checks. The feedback model and experimental claude-sonnet-4-6 share the Claude family; the feedback model's exact identifier/snapshot is not recorded and no snapshot identity is assumed. Neither source constitutes independent human or peer review.