BoF: Practial Reproducibility by Managing Experiments Like Software

Chairs: Ivo Jimenez, Mike Heroux, Kate Keahey, Jay Lofstead, Carlos Maltzahn

When: Wednesday Nov 15, 12:15pm-1:15pm

Where: Room 601

Links:

Abstract

This BoF is for HPC community members who want to make sharing and re-executing scientific experiments more practical. As a starting point, we will introduce the Popper Protocol (http://falsifiable.us), a set of guidelines for managing scientific explorations as software projects. We will moderate discussions, and conduct a survey among attendants on tools/services that can be used to “Popperize” scientific explorations, i.e. fully automate/document experiments by scripting all aspects of their execution/analysis and version-control their evolution using DevOps tools. Our goal: create a list of tool-chain templates that domain-specific communities can use to incentivize researchers/students to generate easily re-executable/shareable experiments.

Agenda

               
12:15   Introduction. Brief introduction to scope the topic of the BoF. In short, our goal is to focus on replicability / re-usability of artifacts associated to experiments (scientific explorations).
12:25   Lightning Talks. Kate Keahey, Carlos Maltzahn and Ivo Jimenez will give a 5-minute lightning talk each, describing their point of view on this topic.
12:35   Discussion. What domain-specific tools do you use? Can they be scripted? Are you scripting your explorations? What works? What needs improvement? How do we incentivize the generation of replicable experiments?
1:05   Survey. Attendees answer a brief survey

How to Cite

If you would like to cite this BoF, this is the suggested format:

Birds of a Feather session on Practical Reproducibility by Managing Experiments Like Software, in International Conference for High Performance Computing, Networking, Storage and Analysis (SC ‘17), Denver, Colorado November 2017. URL: https://falsifiable.us/events/sc17_bof.