Title: If Correct Science is the Goal, is Reproduction the Answer?

Abstract: Science has become computational, introducing a second dimension to science that has mostly been ignored. The original dimension is the conventional scientific dimension focused on a question we want to answer. The research practices for addressing such questions are well-understood. The second dimension concerns the authenticity and veracity of data and computation applied to that data. Unfortunately, rather than recognizing this methodological dimension in its own right, we have tried to shoehorn it into existing practices, confusing the key principle underpinning the scientific method. That is, because we think it should be easy to reproduce computational artifacts, we strive for reproducibiity rather than replicability. Perhaps, we have made our jobs harder than we needed to?

I’ll begin by identifying the challenges we face in computational science, which are even more pronounced in the realm of AI. Then I’ll discuss the confusion around reproduction and replications. Taking a step back, I’ll attempt to derive, from first principles, what we need to properly validate computational experiments, and I’ll conclude with some ideas on a path forward.

Biography: MARGO I. SELTZER is Canada 150 Research Chair in Computer Systems and the Cheriton Family chair in Computer Science at the University of British Columbia. Her research interests are in systems, construed quite broadly: systems for capturing and accessing data provenance, file systems, databases, transaction processing systems, storage and analysis of graph-structured data, and systems for constructing optimal and interpretable machine learning models.

She is the author of several widely-used software packages including database and transaction libraries and the 4.4BSD log-structured file system. Dr. Seltzer was a co-founder and CTO of Sleepycat Software, the makers of Berkeley DB, the recipient of the 2021 ACM Software Sytems award and the 2020 ACM SIGMOD Systems Award.

She serves on the Computer Science and Telecommunications Board (CSTB) of the (US) National Academies. She is a past chair and vice-chair of the Computer Science Committee of the National Academy of Engineering and a past President of the USENIX Assocation. She served as the USENIX representative to the Computing Research Association Board of Directors and on the Computing Community Consortium.

She is a member of the National Academy of Engineering and the American Academy of Arts and Sciences, a Sloan Foundation Fellow in Computer Science, an ACM Fellow, a Bunting Fellow, and was the recipient of the 1996 Radcliffe Junior Faculty Fellowship. She is also recognized as an outstanding teacher and mentor, having received the Phi Beta Kappa teaching award in 1996, the Abrahmson Teaching Award in 1999, the Capers and Marion McDonald Award for Excellence in Mentoring and Advising in 2010, the CRA-E Undergraduate Research Mentoring Award in 2017, and a UBC Killam Teaching prize in 2023.

Professor Seltzer received an A.B. degree in Applied Mathematics from Harvard/Radcliffe College and a Ph. D. in Computer Science from the University of California, Berkeley.