At VISSOFT 2019, Christian Collberg and coauthors received a Most Influential Paper Award for their 2003 article “A System for Graph-Based Visualization of the Evolution of Software”. In the acceptance address, “A (Short) Exercise in Reproduction”, Christian describes the difficulties he encountered in trying to rebuild the now fifteen-year-old software to match what was used for the original article and suggests how he and Computer Science in general can do better.
Over the last few years we have given a few invited talks on repeatability, reproducibility, and sharing. The last keynote presented at the CPs-IoTBench Workshop held in Montreal was entitled “Dare to Share: Risks and Rewards of Artifact Sharing in Computer Science”. The slides are here: materials. This is the third such talk, others having been presented at the ACM SIGCOMM 2017 Reproducibility Workshop, Reproducibility’17, and the 2017 Annual Computer Security Applications Conference, ACSAC 33.
FindResearch.org is a resource for sharing links to research artifacts, the code and data that back up the results in published research papers. Such sharing is a necessary prerequisite for colleagues being able to repeat and reproduce published research.
A common approach to conducting applied Computer Science research is to build a computing system (the “research artifact”) using a specific methodology, to study and measure the behavior of the artifact, and to draw conclusions as to how well the methodology is able to construct correct/scalable/secure systems.