ReproducibiliTea is an international community of journal clubs dedicated to advancing Open Science and improving academic research culture. 

ReproducibiliTea at Stanford was launched at 26 October 2022 and welcomes new members. Information on upcoming meetings is presented below, and you can find us on our slack channel, and join our mailing list reproducibilitea@lists.stanford.edu. The meetings are held every second Wednsday of the month, at 12:00.

To help us prepare for the meetings, and order lunch for everyone (free lunches provided) please register for the next meeting using the registration form. 

The next meeting is June 10th, 12:00  to 1 pm in LK306 Seminar Classroom (see map plan of the building).

Topic: Scaling Reproducibility: An AI-Assisted Workflow for Large-Scale Replication and Reanalysis

Speaker: Yiqing Xu

Abstract: Computational reproducibility is central to scientific credibility, yet verifying published results at scale remains costly. We develop an AI-assisted workflow for automated full-paper replication -- retrieving materials, reconstructing environments, executing code, and matching outputs to point estimates reported in regression tables. We define a universe of all empirical and quantitative papers from the three top political science journals (2010--2025) and measure stated data availability using automated extraction. For a stratified sample of 384 studies, we apply the workflow to conduct full-paper replication, totaling 3,523 empirical models. We find that journal verification requirements, combined with data archiving mandates, drive reproducibility: the share of fully or largely reproducible papers rises from 20.8% before DA-RT adoption to 82.5% after, and conditional on accessible replication packages, 92.1% of papers are fully or largely reproducible (234/254). As a secondary application, we apply standardized IV diagnostics to 84 studies (597 IV specifications among 1,910 replicated models), illustrating how automated execution enables systematic reanalysis across heterogeneous empirical settings.

See the preprint at: https://arxiv.org/abs/2602.16733