Corrections and research scope

Supported public claim

The current public description is limited to the peer-reviewed 2024 ReproScreener study. That study evaluated keyword and GPT-4 screening of reproducibility-related signals in paper abstracts. Source code added later does not by itself establish a validated capability or published result.

2025 local-model experiment

A later forensic audit found that the unpublished 2025 local-model experiment cannot support model-performance claims. Many responses omitted required fields, the archived parser converted omitted fields into negative labels, prompt examples overlapped with evaluation items, and the exact model build and execution environment were not retained. Repeated malformed responses also made output consistency unsuitable as evidence of detector reproducibility.

The repository experiment had an additional identity defect. Repository URLs with missing values were shortened independently from the paper-ID list and then paired positionally. All 22 archived repository folder labels disagreed with their stored Git origins. The downstream analysis also used key presence instead of the saved model score. These per-paper repository results are invalid and have not been relabeled or reinterpreted.

The full-manuscript local-model branch had no completed evaluation run. The archived work did not systematically compare several open-weight models or isolate model, prompt, context, and temperature effects.

Artifact disposition

On 2026-08-31, the active documentation stopped publishing the local-model notebook, its Llama figures and comparison plots, and the per-paper repository notebooks, tables, and figures affected by the identity join. Git history remains the historical record; those artifacts must not be cited as performance evidence.

The repository downloader now creates paper-ID and repository-URL pairs from the same source rows and rejects duplicate paper IDs. This prevents the original positional drift but does not repair the archived experiment. Any new evaluation requires a new protocol, stable manifests, strict output-schema validation, held-out examples, immutable model provenance, and predeclared metrics.