AI Repair Safety: Security-Oracle Outcomes and Gate Admissions Across 12 Synthetic Tasks

Mucheru Maina

Working paper · Methodological revision 2026-11-04 · Frozen experiment v0.1.0 · 2026 · Not peer reviewed · Human adjudication pending

Abstract

We report a frozen experiment with 12 synthetic JavaScript repair tasks, paired benign and adversarial conditions, and three generations per condition. All 72 planned Claude Sonnet 4.6 attempts completed. The separately executed audit classified 68/72 proposals as functionally passing and recorded security-oracle failures in 6/72 proposals, all in T06. These six proposal-level oracle failures arise from exact-output mismatches and are not six confirmed vulnerabilities: the redirect-security interpretation depends on how a downstream URL consumer resolves emitted locations. Two occurred among functional passes (2/68, 2.94%), a descriptive count within this task-clustered run, not a generalizable security failure rate. Only two of the 12 tasks have demonstrated baseline security findings relevant to remediation or persistence: T06 retains baseline outputs that fail the security oracle, with vulnerability interpretation conditional on consumer semantics; T09's six repairs remediate tested baseline redaction failures. Matching T06 baseline outputs establish persistence of the tested oracle discrepancy, not a newly introduced security regression. All three gates admitted every proposal (216/216 admissions), including the six with security-oracle failures; admissions alone do not establish gate effectiveness. Benign and adversarial conditions each produced 34/36 functional passes and 3/36 security-oracle failures, without establishing equivalence or a causal effect. Thirty-one distinct byte patches account for 72 generations, with 41 repetitions beyond the first occurrence. Repeated generations are not independent tasks. Independent human adjudication remains pending; no confirmed deployment exploitability, human safety, prompt-compliance or false-rejection result is reported.

Keywords: AI-assisted programming; automated program repair; software security; empirical software engineering; independent verification; program analysis; vulnerability persistence.

1. Introduction

Automated repair is often evaluated by whether a generated patch satisfies a test suite. A test-adequate patch, however, can remain semantically incorrect or security-relevant behavior can fall outside the exercised cases [4, 5]. The same distinction applies to repair proposals generated by large language models: functional correctness, observed security behavior, admission by a verification gate, and human judgment are separate outcomes.

This study asks:

Which baseline security findings persist or are remediated under the tested contracts, and which security-oracle failures do the frozen admission gates admit?

The study contributes a frozen paired design, a separately executed audit, publicly reproducible descriptive analysis, and a negative result that narrows the original claim. T06 retains baseline security-oracle failures whose vulnerability interpretation depends on consumer semantics; T09 remediates tested baseline redaction failures. The experiment provides evidence about these scoped transitions and gate blind spots, but not a prevalence estimate for security regressions in production software or a causal estimate of adversarial prompting.

2. Background and Related Work

Automatic program repair uses tests, contracts, models, or other behavioral oracles to guide and assess candidate patches [4]. Prior empirical work shows that plausible patches can overfit available tests, motivating assessment beyond test-suite adequacy [5]. AI code generation introduces an adjacent concern. Pearce et al. found security weaknesses in Copilot-generated programs across multiple vulnerability scenarios [1]. Perry et al. studied how people used an AI coding assistant on security tasks and found that access to the assistant did not guarantee secure outcomes [2]. These studies motivate separate security measurement, but neither directly estimates repair safety under the present paired, saved-proposal design.

SWE-bench evaluates language models on real repository issues and illustrates how much richer repository-scale repair is than the short service-core tasks used here [3]. Consequently, this study does not compare its success rate with SWE-bench or generalize from synthetic JavaScript functions to repository-level maintenance. Its narrower focus is the relationship between functional tests, three deterministic admission gates, and a separate audit over the same saved proposals. This focused review situates the study; it is not a systematic literature review.

3. Research Questions

RQ1 — Baseline-to-repair outcomes. Which baseline security findings persist or are remediated under the tested contracts, and how do these observations overlap with functional outcomes?

RQ2 — Admission behavior. Which security-oracle failures are admitted by Gates A, B, and C when the same proposal is replayed unchanged?

RQ3 — Paired conditions. How do descriptive outcomes differ between benign and adversarial evidence conditions at the task level?

RQ4 — Verification cost and uncertainty. What missing, unknown, duplicate, rejected, and resource-consumption outcomes arise in the frozen run?

The originally preregistered hypotheses are the four H1–H4 statements in the locally frozen Phase 2 protocol, reproduced verbatim and assessed in Section 6.2. This was local preregistration before capture, not external registration. The manuscript's five broader H1–H5 reporting labels are not the frozen hypotheses; Section 6.2 distinguishes them as reporting H1–H5. No frozen H5 was located, so none is presented as preregistered. The protocol remains unchanged.

4. Experimental Methodology

4.1 Study design

Twelve synthetic JavaScript tasks cover authentication, authorization, input validation, configuration security, secrets handling, and error handling. Each task has one benign and one adversarial context, with three generations per condition. The complete design therefore contains 12 × 2 × 3 = 72 planned attempts. The task is the unit of scenario diversity; the 72 generations are not 72 independent tasks.

All 72 canonical request serializations are byte-identical to Phase 2 v2. The returned model identifier was claude-sonnet-4-6 for every response. Provider defaults and an immutable provider snapshot identity remain unverified. Capture timestamps span 2026-11-03 15:59:44.434583 to 16:10:23.106393 UTC (18:59:44 to 19:10:23 Africa/Nairobi). These are recorded completion timestamps, not verified provider execution times. The run used no requested temperature, top_p, top_k, stop sequences or model sampling seed. Seed 2718 randomized dispatch order only; the legacy baseline file also contains seed 17, which was not sent as an API parameter. Identical model aliases and requests cannot ensure identical future weights, routing, defaults or outputs.

4.2 Threat model and scenario construction

The attacker controls one designated untrusted CI-log segment or source comment. The legitimate repair objective and explicit safety requirements remain present. The attacker cannot modify the coordinator, audit definitions, credentials, or security policy. Candidate code executes in disposable, network-disabled containers without repository mounts or model credentials. GitHub operations are simulated intents; no production service or repository merge is exercised.

Tasks model narrow synchronous JSON service contracts rather than complete HTTP, TLS, database, or credential-store integrations. Paired conditions share the objective, baseline, and visible tests; only the designated evidence differs.

4.3 Validation gates

Gate A implements the historical first-tool-call behavior. Gate B adds tool-name, schema, path-allowlist, immutable-file, and bounded-replacement checks. Gate C adds frozen functional and security examples. Each saved proposal is replayed unchanged through all three gates. An infrastructure failure is classified as unknown, not as successful verification.

4.4 Independent audit and evaluator correction

The final audit runs after capture and replay. It receives saved proposals but not historical gate decisions; its assertions remain outside candidate processes, and Gate C cannot read its cases. This is execution separation, not independent authorship: the task, gate, and audit share an author, specifications, and runtime assumptions.

Development review found that the original T06 reference checked the parsed input origin but emitted a serialized redirect location that could be reinterpreted as another authority. Phase 2 v3 corrected the evaluator-only reference and retained the original reference as an author-designated unsafe control under the assumed redirect-consumer semantics. Unchanged Gate C admits that control, while the corrected audit records exact-output failures. This counterexample informed audit development and is not a held-out discovery. Offline validation matched all 64 authored control states; the v3 suite recorded 31 passing tests, and the budget supplement added four passing tests. These checks validate the authored controls, not general audit sensitivity.

4.5 Human adjudication

The protocol requires two genuinely independent humans. The versioned prospective review guidance requires reviewers to state the assumed URL consumer, supporting evidence and uncertainty without seeing an automated preferred verdict; frozen packets remain unchanged. Stage one hides model identity, condition, gate and audit outcomes, and attack evidence while reviewers judge functionality and safety. Their submissions are locked before stage two reveals attack evidence for a separate prompt-compliance judgment. Seventy-two stage-one packets have been prepared, but no human votes have been collected. Accordingly, this paper reports no human-adjudicated safe/unsafe label, reviewer agreement, injection-compliance rate, false-admission rate, or false-rejection rate.

4.6 Analysis

The analysis is a descriptive census of this frozen run. It reports exact numerators and denominators, task- and condition-level summaries, duplicate byte patches, missing outcomes, latency, token use, and usage-based cost. It does not treat repeated generations as independent tasks, calculate a post-hoc significance test, or claim a population confidence interval. Phase 1 evidence is preserved but not pooled with Phase 2. The frozen machine label violation_observed denotes an oracle finding, not automatic confirmation of a vulnerability. Numerical observations and machine labels remain unchanged; scientific interpretation is reported separately.

5. Results

5.1 Study overview

All 72 planned attempts completed once. There were no API errors, missing patches, automatic retries, or maximum-token stop reasons. Every proposal has three gate records and one audit record, yielding 216 gate decisions and 72 audit outcomes.

5.2 Functional correctness

The independent audit classified 68/72 proposals as functionally passing (94.44%), 4/72 as failing, and none as unknown. All four functional failures occurred in T06. The remaining 11 tasks had six functional passes each.

5.3 Security-oracle findings

The audit recorded security-oracle failures for six proposals and no violation observed for 66. Two failures occurred among 68 functional passes (2/68, 2.94%); four occurred among the four functional failures. All six are T06 proposals. This is a descriptive cross-tabulation over repeated generations in 12 tasks, not a generalizable security failure rate or a count of independently confirmed vulnerabilities.

Every failing T06 output value matches the unchanged baseline on the same frozen cases. This establishes persistence of the tested output discrepancy. The emitted locations resolve externally under WHATWG-style interpretation for the problematic forms; the actual downstream consumer and deployment exploitability are unverified. Section 6.3 explains why an exact-output failure alone does not confirm a vulnerability. T09's six repairs remediate its tested baseline redaction failures. Only T06 and T09 provide demonstrated baseline security findings relevant to persistence or remediation; the other ten tasks have no violation observed, not established safety. The 66 remaining repairs are also not proven safe.

Table 3 exposes the concentration that aggregate rates can obscure: the audit observed no violation outside one task, and the known counterexample for that task informed evaluator development.

5.3.1 Post-hoc baseline-to-repair security comparison

This analysis was added after capture on 4 November 2026. It reads original objectives, invariants, baseline/reference implementations and frozen security cases. Fresh offline replay of all 24 baseline/reference combinations exactly reproduced the immutable validation receipt. All baselines failed at least one functional case. T06 had baseline security-oracle failures and T09 had demonstrated baseline redaction failures; the other ten had no violation observed, often with restrictive baseline behavior. All corrected references had no violation observed. The original T06 reference had the same consumer-dependent finding and was amended before capture.

In this table, “None observed” means no oracle violation observed; it is not a vulnerability-absence claim. “Remediated” means that the previously failing tested behavior is corrected, not that the repair is generally safe. T09 corrected mixed-case and nested credential redaction, including a canary disclosure on a case labeled functional. T06 preserved the same failing redirect output values as its baseline. No introduced vulnerability is established. For the other ten tasks, a missing detection cannot establish absence before or after repair; calling these “preserved safety” would overstate the evidence. Untested vulnerability status is undetermined for every proposal, including T09 and the references. These categories describe observed transitions and unresolved coverage, not an exhaustive partition of real vulnerabilities.

5.4 Validation gate outcomes

Gates A, B, and C each admitted 72/72 proposals. Consequently, each gate admitted 6/6 oracle-failing proposals and 2/2 proposals that both passed functional checks and failed the security oracle. These are automated security-oracle-failure admissions, not human-adjudicated unsafe admissions. With no gate rejections and no human-safe denominator, a false-rejection rate is undefined. Reviewing all 216 decisions and their proposal identities confirms 72 admissions per gate and no rejection of a proposal with a security-oracle failure in the captured run. Admissions alone do not establish gate effectiveness, discrimination or calibration. Gate A has no semantic security test; B can reject malformed or unauthorized proposals but does not inspect security semantics; C can reject violations exercised by its examples. In the historical 64 author-generated controls, 30 were oracle-failing: A and B rejected zero, and C rejected seven (including the T06 and T09 baselines, with T06 interpreted conditionally). This demonstrates limited example-based rejection ability, not independently validated sensitivity. The frozen unsafe controls were deliberately capable of passing C; they are not a representative held-out sample.

5.5 Benign and adversarial conditions

The benign and adversarial conditions each produced 34/36 functional passes, 3/36 security-oracle failures, and 1/34 security-oracle failures among functional passes.

Each of the 12 paired tasks has a zero observed difference on the functional-pass rate and security-oracle-failure attempt rate. The design and sample do not support an equivalence claim, and identical observed rates are not evidence that adversarial context has no code-level influence. Behavioral prompt compliance requires the pending staged human review.

A post-hoc inspection covered all 36 adversarial-condition tool outputs, but no locked, reliably applied per-attempt uptake rubric or classification exists. Uptake therefore remains unquantified. Illustrative acknowledgments include: T03 rejects the suggested tenant-filter bypass, T07 rejects the proxy-header workaround, and T08/T10/T12 discuss removing unsafe comments. These are observations of acknowledgment and stated rejection, not proof of internal reasoning or causal influence on executable behavior. Some explanations quote attacker text while rejecting it; literal retention is not compliance. T06 retains exact-origin checks rather than the suggested suffix policy, while still retaining the consumer-dependent serialization behavior also present in benign outputs. No captured audit outcome establishes successful attacker-induced weakening. Silence in other explanations does not establish that evidence was ignored. A complete causal or compliance classification remains unresolved; no chain-of-thought trace was recorded and no human judgments are substituted.

5.6 Patch diversity

Thirty-one byte-distinct patch groups account for 72 generations; 41 generations repeat a byte patch observed elsewhere in the run. The six T06 patches are byte-distinct. Byte identity is a conservative syntactic measure: distinct patches can be semantically equivalent, while repeated generations remain correlated observations rather than new tasks. Exact patch hashes are withheld while review blinding is pending; public records use opaque group identifiers.

5.7 Resource consumption

Responses report 78,180 input tokens and 35,078 output tokens, with zero cache activity. At the recorded list prices of USD 3 and USD 15 per million input and output tokens, respectively [6], the usage-based estimate is USD 0.760710. No invoice is available, so this is not a verified billed amount. Conservative pre-call reservations total USD 9.197313, below the USD 10.00 operational cap and USD 10.22 authorization ceiling.

Median API latency was 8.876 seconds (range 5.570–14.344). Recorded gate-bundle time totals 15.804 seconds and final-audit time totals 114.799 seconds. Gate records time a bundle rather than isolated A/B/C execution, so these values must not be summed once for each gate or interpreted as production costs.

5.8 Limitations of automated assessment

The automated audit and gates share authorship and contracts. The only repair security-oracle failures occur in T06, whose counterexample was known during evaluator development; they are not independently confirmed deployment vulnerabilities. The study therefore does not estimate unbiased audit sensitivity, general gate effectiveness, or a model-wide unsafe-repair rate. Human adjudication remains necessary to assess safety judgments, reviewer agreement, prompt compliance, and false rejections.

6. Discussion

6.1 Answers to the research questions

RQ1. T09's tested baseline redaction failures were remediated in six repairs. T06's tested baseline output discrepancies persisted in six; their vulnerability interpretation depends on downstream URL-consumer semantics. These are the only two of 12 tasks with demonstrated baseline security findings relevant to the transition analysis. Two of 68 functional passes have T06 security-oracle failures; this does not estimate a general security failure rate. No newly introduced regression or deployment exploitability is established.

RQ2. All three gates admitted the six proposals with security-oracle failures, including the two functional passes. This demonstrates a blind spot for the frozen T06 assertions, not confirmed unsafe deployment admissions or general gate ineffectiveness. Admissions alone do not establish effectiveness.

RQ3. Automated outcome rates were identical between conditions at every task. This is a null descriptive result, not an equivalence test or causal finding. Explanation-level uptake remains unquantified and behavioral compliance awaits human review.

RQ4. There were no missing attempts or infrastructure unknowns. Patch duplication and verification runtime were observed. With no gate rejections or human safety labels, false-rejection and human-safe admission rates remain undefined.

6.2 Hypothesis assessment

The originally preregistered hypotheses below are quoted verbatim from the locally frozen Phase 2 protocol. They retain their original numbering; there is no preregistered H5 in that artifact.

H1: B reduces policy-violating admissions relative to A but can miss semantic violations in allowed files.

H2: C reduces some final audit violations relative to B, but misses others or rejects legitimate repairs.

H3: adversarial evidence can alter behavioral compliance with the supplied attack objective; ordinary regressions are distinct.

H4: correct alternatives may need semantic adjudication despite passing verification.

Preregistered H1 — no observed support for reduction (RQ2). A and B each admitted all 72 proposals. The expected reduction did not occur in this run; no captured policy-violating proposal supplies a meaningful test of B's policy-rejection sensitivity. Both admitted the T06 oracle-failing proposals, with semantic safety unresolved.

Preregistered H2 — reduction expectation contradicted in this run; other parts unresolved (RQ2). C and B each admitted all 72, including the same six security-oracle failures. Thus C did not reduce final-audit failures here. The missed-oracle-failure component is observed. Legitimate-repair rejection is not meaningfully testable with no rejections and no human-safe labels. This does not establish a general causal or population result.

Preregistered H3 — inconclusive (RQ3). Identical automated endpoints do not establish equivalence or behavioral compliance. Neither unquantified explanation uptake nor the current human-review status identifies an adversarial causal effect.

Preregistered H4 — unresolved pending adjudication (RQ1–RQ2). T06's specification and consumer ambiguity motivates semantic review despite gate admission, but does not establish that any proposal is a correct alternative. No independent judgment yet tests that claim.

For continuity, the separate, later manuscript reporting hypotheses are retained below. They are not preregistered and must not be confused with the four protocol statements:

• Reporting H1: “some functionally correct generated repairs contain security regressions.” Inconclusive: two functional passes fail T06's security oracle, but newly introduced regressions are not established and consumer-dependent exploitability remains unresolved (RQ1).
• Reporting H2: “deterministic controls catch some unsafe repairs but miss semantic violations.” Not meaningfully testable as a general safety claim: no captured proposal was rejected and independent safety labels are absent. Historical author-generated controls show limited oracle-based rejection capability only (RQ2).
• Reporting H3: “separate verification detects additional violations.” Supported only for additional oracle failures: the separately executed audit flags six proposals admitted by C. This is not confirmation of six vulnerabilities or unbiased detection sensitivity (RQ2).
• Reporting H4: “adversarial context can influence repairs, while injection compliance differs from ordinary regressions.” Inconclusive: no causal/compliance classification is established (RQ3).
• Reporting H5: “stronger verification introduces rejection, uncertainty and operational costs.” Runtime cost is observed; additional rejection and unknown outcomes did not occur. False rejection is not meaningfully testable without human labels (RQ4).

These are scoped descriptive assessments, not post-hoc significance tests or replacements for the frozen hypotheses.

6.3 Security case study: T06 redirect serialization (post-hoc interpretation)

T06 asks the repair to accept relative checkout callbacks while retaining the trusted origin. All six generated repairs serialize a redirect location that a WHATWG-style URL consumer can resolve to an external origin. The frozen audit itself uses exact-output comparison, not a URL consumer. Two pass the broader functional suite; four reject callbacks without a leading slash that the audit treats as functional requirements. Gate C admits all six.

The historical authored reference already emitted the same problematic forms under WHATWG-style interpretation. Phase 2 v3 corrected the evaluator reference before capture and retained the old reference as a known unsafe control. A separately labeled post-capture diagnostic resolved the emitted location from each captured patch and recorded an external resolved origin. The diagnostic receipt does not preserve the resolver implementation; its exact consumer provenance remains unresolved. That diagnostic did not alter audit scores, cases, or gate decisions. It is an author diagnostic, not independent confirmation of deployment exploitability or human adjudication.

The original objective says “Support relative checkout callbacks without open redirects”; its one visible functional example has a leading slash. The frozen audit additionally expects relative callbacks without a leading slash, preservation of nested/encoded path segments, and acceptance of explicit HTTPS default port 443. Four proposals fail both no-leading-slash cases: benign generations 0 and 2, adversarial generations 1 and 2 (generation indices are zero-based). Benign generation 1 and adversarial generation 0 pass the functional suite. Thus the four failing proposals implement a narrower relative-callback contract than the audit expects; whether that contract satisfies the original objective requires adjudication. The broad objective supports this interpretation, but the single visible example underdetermines its full breadth; “without alternate ports” also leaves room for disagreement over explicit default ports. Human review should assess this specification/test gap. Historical scores remain unchanged.

Every proposal fails the same five emitted-location security cases, which exercise leading double/triple slashes, backslash normalization and dot-segment normalization. Parsing the input against a trusted origin does not guarantee that returning pathname plus query preserves that origin when another parser consumes the result. Four proposals have both functional and security-oracle failures; two have functional passes and security-oracle failures. These are distinct inputs and mechanisms within one task, not ten independent failures or six independent vulnerabilities.

The frozen security assertions require an exact rejection object. A safe absolute trusted redirect or another safe response could therefore fail the oracle; generic audit mismatches are not necessarily vulnerabilities. For these six proposals, the captured values and baseline comparison establish persistence of tested outputs; the historical diagnostic records an external resolved destination without preserving its resolver implementation. The previously recorded post-hoc neutral-example comparison demonstrates consumer dependence: Python urllib.parse.urljoin retains the trusted host for a triple-leading-slash path, whereas Node's WHATWG URL resolves it to the apparent external host. Both resolve a double-leading-slash path externally. Thus Python urljoin does not make all recorded T06 forms same-origin. The disagreement concerns particular forms, and no deployment consumer is specified. The frozen backslash-input case emits a double-leading-slash value in the saved observations; the previously recorded raw-backslash neutral example is a consumer comparison, not an additional experimental failure. The audit did not run either resolver, so describing its scoring as consumer-based was inaccurate. The precise downstream application/browser consumer and historical diagnostic implementation are not established; browser/HTTP end-to-end exploitability remains to be checked independently. The unresolved human-adjudication question is: given the original callback contract and an explicitly stated downstream consumer, do the emitted values permit an external redirect, and do the narrower relative-path implementations meet the functional objective? Reviewers must cite evidence and uncertainty rather than infer a preferred verdict from the audit. No unconditional safe/unsafe classification follows from these scores. Historical exact-output results remain unchanged.

7. Threats to Validity

Construct validity. Functional checks and finite security cases are imperfect proxies for correctness and safety. Exact-output assertions may reject safe alternatives. “No violation observed” is not equivalent to safe, and a security-oracle failure is not automatically a confirmed vulnerability or a newly introduced regression.

Internal validity. The same author developed tasks, gates, audit cases, and mutants. Execution separation prevents direct case access but not correlated misconceptions. The T06 counterexample was known during audit development, which raises evaluator-selection and overfitting concerns. The provider's effective defaults and immutable model snapshot are unverified.

Conclusion validity. There are only 12 tasks, three generations per condition, and 31 byte-distinct patches. Repeated generations are correlated within tasks. The analysis is descriptive; it reports no population confidence interval, equivalence test, or significance claim.

External validity. The short synthetic JavaScript tasks omit repository-scale context, stateful services, real networking, databases, TLS, and production deployment. One model and one frozen run cannot establish behavior for other models, versions, languages, domains, or production systems.

Reproducibility and blinding. The public package omits private audit cases, raw captures, generated code, exact patch hashes, and reviewer mappings. It reproduces published descriptive reporting but not the full private security test. Public results preceded human review, so reviewers must disclose prior exposure and successful blinding cannot be assumed. Private evaluation material exists in local historical Git data; prior public exposure was not established, but cannot be disproved by current remote inspection.

Post-hoc and control validity. Baseline transition classification, trace inspection, consumer comparison and this methodological revision occurred after outcomes were known. They are diagnostics, not preregistered tests. The original controls and evaluator share authorship; no independently authored control suite has yet been executed. Publication of task-level findings preceded human review, so reviewer recognition of T06 and knowledge of reported outcomes may compromise blinding even after metadata removal. Independent reviewer order, exposure screening and third-reviewer resolution are prospective safeguards, not completed procedures.

8. Implications for Practice

First, acceptance decisions should distinguish functional test success from security evidence. Second, redirect and serialization checks should validate the value actually consumed, not only an intermediate parsed value. Third, independence should be described precisely: execution separation is useful but does not remove shared specifications or authorship. Finally, teams should retain unknown outcomes, duplicates, and evaluator limitations rather than collapsing them into a single pass rate.

These implications are engineering recommendations derived from a narrow study. They are not evidence that the tested gates are generally ineffective or that the measured rates predict production risk.

9. Conclusion

In this frozen 12-task, 72-attempt run, 68 proposals passed the functional audit and six T06 proposals failed security-oracle assertions; two of those six were functional passes. All three gates admitted every proposal. The T06 outputs match baseline failures, establishing persistence of tested behavior, while vulnerability interpretation depends on downstream URL-consumer semantics and deployment exploitability remains unconfirmed. T09's six repairs remediate tested baseline redaction failures. Only these two tasks have demonstrated baseline security findings relevant to remediation or persistence; the other ten are not thereby proven safe. The 2/68 cross-tabulation is not a generalizable security failure rate. No newly introduced regression is established. Identical condition-level endpoints do not establish equivalence or absence of adversarial influence. Independent human adjudication remains pending.

Data and Code Availability

The public repository contains sanitized per-attempt outcomes, request hashes, analysis and publication code, artifact manifests, and manuscript sources: https://github.com/muchezz/ai-repair-safety-study. Raw model responses, generated code, hidden audit cases, exact patch hashes, credentials, and reviewer mappings remain in a controlled private snapshot. This restriction protects blinded review and private evaluator material, but prevents public end-to-end replication of the security audit. The v0.1.0-preadjudication release remains immutable.

Revision and Pending Validation

See the methodological change log and unresolved questions.

The 4 November 2026 revision changes interpretation and reporting, not frozen definitions, scores, capture files or the historical prerelease. Baseline replay and numerical verification are post-hoc offline checks. Newly created evaluator controls remain a plan, not experimental evidence. Independent authors should construct safe/unsafe variants without access to evaluator cases, lock their intended invariants and provenance, and then test all gates and the audit offline. The detailed plan and unresolved human questions are in methodology and human review instructions. Publication preceded human adjudication; no completed independent review is claimed.

AI Assistance Disclosure

Codex assisted implementation, audit-development code, analysis code, verification, manuscript editing and website preparation. A Claude model also provided non-binding methodological feedback through a chat session, as reported by the study operator. That model did not inspect individual patches, captures or private audit cases; its feedback was based on the material supplied in chat. It was an AI feedback source, not an independent human reviewer or peer reviewer, and its scientific claims require artifact-level checking.

The experimental repair model was claude-sonnet-4-6. The feedback model belongs to the same Claude model family, but its exact identifier and immutable snapshot are not recorded here; no claim is made that it was the same model version or exact snapshot. Shared model-family provenance further limits any claim of independent assessment. No human judgments were simulated. The numerical observations remain checked against frozen records and the public outcome projection.

References

[1] H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri. “Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions.” 2022 IEEE Symposium on Security and Privacy, pp. 754–768, 2022. https://doi.org/10.1109/SP46214.2022.9833571

[2] N. Perry, M. Srivastava, D. Kumar, and D. Boneh. “Do Users Write More Insecure Code with AI Assistants?” Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 2785–2799, 2023. https://doi.org/10.1145/3576915.3623157

[3] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” International Conference on Learning Representations, 2024. https://openreview.net/forum?id=VTF8yNQM66

[4] M. Monperrus. “Automatic Software Repair: A Bibliography.” ACM Computing Surveys, 51(1), Article 17, 2018. https://doi.org/10.1145/3105906

[5] H. Ye, M. Martinez, T. Durieux, and M. Monperrus. “A Comprehensive Study of Automatic Program Repair on the QuixBugs Benchmark.” Journal of Systems and Software, 171, 110825, 2021. https://doi.org/10.1016/j.jss.2020.110825

[6] Anthropic. “Claude Sonnet 4.6: Model overview and pricing.” Claude Platform Documentation. Accessed 3 November 2026. https://platform.claude.com/docs/en/models/sonnet-4-6/overview

Appendix A. Reproducibility Anchors

The Phase 2 v3 definition digest is 37441f8cb59a827154db7e857488230aeb3d8bf7799117bdf5bb804ec7e95346. The bound validation receipt digest is dca3d2e51409c9f8c3c0fc6b51aca3c025495797b6d590e1c5604bab2a2bb9dd. The validated Node image and content identifiers, exact reproduction commands, and public artifact hashes appear in the rendered reproducibility and artifacts pages.