Reproducibility in Applied Security Conferences: An 11-Year Review on Artifacts and Evaluation Committees
Daniel Olszewski, Allison Lu, Anna Crowder, and 10 more authors
In Proceedings of the 3rd ACM Conference on Reproducibility and Replicability, 2025
Reproducible research ensures a foundation for credible and progressive scientific advancement. Calls for reproducibility in the computer security and privacy community resulted in ACM WiSec and ACSAC in 2017 creating artifact evaluation committees (AECs) to foster more reproducible outcomes. Upon acceptance to ACM WiSec and ACSAC, authors may submit experimental artifacts to the committee to demonstrate that their artifacts effectively support their paper. While this may seem to address reproducibility, it is unknown whether AECs have actually improved reproducibility. Furthermore, AECs only assess artifacts submitted to the committee, and thus, may not holistically reflect the status of reproducibility (e.g., artifacts that did not go through the AEC). We conduct an indirect (i.e., availability) and a direct (i.e., running code) reproducibility studies to measure and evaluate the potential and realized reproducibility by classifying and running artifacts associated with individual papers. We collect over 2,000 papers and 550 artifacts to measure the state of reproducibility within four established applied security conferences between 2013 and 2023, two of which have AECs and two comparable conferences without AECs. We find that only 10% of research provides runnable artifacts, and only 3.9% of published papers produce a reproducible artifact. More importantly, our work shows that including an AEC alone does not necessarily result in more reproducible artifacts; specifically, ACM WiSec artifacts are more or less in line with the two control conferences, whereas ACSAC ultimately achieves 90% of artifacts participating in the AEC, with 40% of artifacts running. As such, our work shows that while the security community is developing mechanisms to foster reproducible research, there are significant improvements needed to progress reproducibility within computer security and privacy research.