Reproducibility and artifact scope

Working paper · Version 0.1.0 · Public-safe reproducibility package

This public package is based on release version 0.1.0-preadjudication: protocol, validation aggregates, sanitized per-attempt outcomes, reporting code and draft manuscript. Raw Phase 2 model outputs remain private. Private audit cases, hidden tests, raw captures, keys and reviewer mappings are excluded from both files and Git history. A full private snapshot is preserved separately. This restriction prevents public end-to-end security-test replication; external review requires controlled access. Do not describe this release as a complete open empirical benchmark.

Reproduction Procedure

Python 3.12 is sufficient for the static build and public checks:

python3 scripts/build.py
python3 scripts/check.py
python3 scripts/analyze.py data/status.json
python3 -m unittest discover -s scripts -p test_analysis.py

The build uses relative URLs under /ai-repair-safety-study/. No service secrets, API calls, JavaScript dependencies or container execution are needed. The PDF is generated from research/PAPER.md with the pinned optional publishing tools:

python3 -m venv /tmp/research-pdf-env
/tmp/research-pdf-env/bin/pip install -r requirements-publishing.txt
/tmp/research-pdf-env/bin/python scripts/paper_pdf.py

Verify artifacts against PUBLIC_ARTIFACT_MANIFEST.json. It hashes public release inputs and generated assets; the manifest itself is excluded to avoid recursive hashes. Git records the exact public source revision; the private historical revision alone does not identify uncommitted frozen evaluator files.

Provenance Anchors

Historical private source revision: c28c04a724f91db324429e0e456f2136f760029f. V3 definition SHA-256: 37441f8cb59a827154db7e857488230aeb3d8bf7799117bdf5bb804ec7e95346. Frozen validation receipt SHA-256: dca3d2e51409c9f8c3c0fc6b51aca3c025495797b6d590e1c5604bab2a2bb9dd.

Requested model: claude-sonnet-4-6; max_tokens 4096; forced propose_fix tool; API version 2023-06-01. Temperature/top_p omitted. Every response returned claude-sonnet-4-6. Effective provider defaults and an immutable model snapshot remain unverified. All 72 request hashes are published without prompt bodies. Request serializations are identical to v2.

Pinned image: node@sha256:8f693eaa7e0a8e71560c9a82b55fd54c2ae920a2ba5d2cde28bac7d1c01c9ba5

Validated image content ID: sha256:33ed5ef90e835c40007b658b2d7cb1354b282bffcb53feceb305e9feff1d98e1

Actual validation used non-root, network-disabled, read-only-root containers, dropped capabilities, no-new-privileges, restricted tmpfs and bounded resources. No private host mounts or model credentials are passed to patch processes.

Controlled Private Continuation

The operator's authorization covers 72 attempts and USD 10.22, with a USD 10.00 operational cap. All 72 attempts completed once. The independent-human-review dependency remains. A separate pre-call budget supplement must run before dispatch. Frozen v3 capture alone lacks cumulative budget enforcement. Replay, audit, packet generation and scoring have consumed the immutable captures; no generation was retried. No public command claims to run withheld private tests.

Published Findings and Private Correspondence

The public observation projection reproduces all descriptive metrics; each request hash matches the planned manifest. It includes gate/audit outcomes and opaque duplicate groups, not raw generated code, exact patch hashes, hidden cases or packet IDs. All six T06 security-oracle failures also occur in the baseline; their vulnerability interpretation depends on URL-consumer semantics. This post-capture validity comparison is documented separately from frozen scoring. Public summaries are cross-checked with the frozen private scorer.

Methodological revision verification

The frozen experimental release remains v0.1.0-preadjudication. The manuscript revision is dated 4 November 2026; current artifact manifests describe revised publication inputs and do not replace the historical release archive/manifest. The original outcome projection, scoring code and scientific-content lock are unchanged. The new aggregate verification is explicitly post-hoc: sanitized methodological verification.

For an authorized operator with the private source tree and locally provisioned pinned image (never downloaded automatically):

python3 scripts/verify_methodology.py /path/to/private/source

This command verifies hash-chained artifacts and the completion manifest, checks all 72 proposal/audit and 216 gate identities against public outcomes, and freshly replays original baselines and corrected references in offline containers. It exports only sanitized aggregates into a new file, without modifying private evidence. It cannot be run from public files alone.

All captures report requested and returned model claude-sonnet-4-6. Recorded completion timestamps run from 2026-11-03T15:59:44.434583+00:00 through 2026-11-03T16:10:23.106393+00:00. The request contains only max_tokens=4096, a user message, propose_fix tool definition and forced tool choice besides the model identifier; API version is 2023-06-01. Temperature, top_p, top_k, stop sequences and sampling seed were omitted. Dispatch seed 2718 affects ordering, not model randomness. Provider defaults, backend routing, immutable weights and provider execution timestamps are unverified. Request hashes support input identity and saved-response replay, not deterministic future regeneration.

A neutral-example consumer comparison used local Node WHATWG URL and Python urllib.parse.urljoin. They disagree on triple-leading-slash and backslash paths; this is post-hoc diagnostic evidence, not a change to exact-output audit scoring. The historical redirect diagnostic receipt records a resolved origin but does not retain the consumer implementation. End-to-end browser behavior and exact historical consumer provenance remain unresolved.

The neutral comparison is reproducible with python3 scripts/check_url_consumers.py; its versioned output is consumer comparison.

The existing post-hoc aggregate retains its original field names, including preserved_observed_vulnerability; for T06 this is a legacy oracle-derived label, not an independent confirmation of deployment exploitability. The artifact and frozen observations remain unchanged. Updated prose separates these machine labels from adjudicated scientific conclusions.