Truth and transparency in expertise research

Abstract

Across research areas, general issues of low statistical power, publication bias, undisclosed flexibility in data analysis, and researcher degrees of freedom, can be recipes for irreproducibility. To address the problem, a reform movement known as the “credibility revolution” emphasizes the need for greater transparency in how research is conducted. In this article, we describe a general approach to creating a culture of openness—tailored for expertise researchers—and describe how and why practices such as preregistration, open notebooks, open data, open materials, and open communication, might be applied to research on experts. We argue that adopting these practices helps to connect end-users with the entire research lifecycle and helps reconnect researchers with the process of gaining knowledge. By sharing notes about our predictions and plans along the way, we are forced to confront their merits. By documenting design and data analytic decisions ahead of time, and by sharing data and materials, we make errors and insights more discoverable. And by inviting research partners, expert practitioners, and the public into the lab, we stand the best chance of successfully translating research into practice.