Analysis of Verified Replication Packages from the American Journal for Political Science

Project Description

Since 2015, the American Journal for Political Science (AJPS) has adopted a Replication and Verification Policy (https://ajps.org/ajps-replication-policy/) that requires researchers to deposit replication materials (including data and code) required to reproduce analytic results reported in published articles. In cooperation with the Odum Institute for Research in Social Science, AJPS has implemented a process for third-party verification. Replication packages deposited since the adoption of this policy should have a higher success rate for computational reproducibility than packages that were not verified.

This internship will explore the rates of reproducibility for packages submitted before and after the policy adoption and implementation of the Odum workflow. Results should be comparable to those reported by Kingi et al for the American Economic Journal. The intern will download a subset of packages from the AJPS Dataverse and attempt to reproduce the reported results (with reasonable effort) using the Whole Tale platform, when possible.

Necessary Prerequisites:

Basic understanding of quantitative methods in political science
Ability to run and troubleshoot code in a variety of languages (R, Python, Matlab, Stata)
Basic programming skills including the ability to download and process published data packages

Desirable Skills / Qualifications:

Interest in data curation
Interest in open, reproducible science and computational reproducibility

Expected Outcomes:

Assessment report on issues encountered during replication attempts (e.g., computing systems, data, code, documentation quality)
A set of tales in WT representing the AJPS replication packages
Requirements for the Whole Tale system for unsupported replication packages

Primary Mentor: Craig Willis, National Center for Supercomputing Application, University of Illinois at Urbana-Champaign

Secondary Mentor(s): TBD