Methodology and Statistics
Tilburg School of Social and Behavioral Sciences
Tilburg University

On 22 March 2024 Olmo van den Akker defended his thesis: Preregistration in psychology: Past, present, and prospects at the university of Tilburg.

Summary
While preregistration has been lauded as one of the solutions to the replication crisis in psychology, not much empirical evidence is available about its effectiveness. In this set of studies, we aimed to assess whether preregistrations in psychology are sufficiently producible (i.e., they can be conducted based on the information provided in the preregistration) and sufficently in line with the corresponding publications. We also assessed whether preregistered studies include a lower proportion of positive results than non-preregistered studies, which would be an indication of a preventative effect on questionable research practices like p-hacking and HARKing.

We assessed 459 preregistered studies that either won a Preregistration Challenge prize or earned a Preregistration Badge. We custom-made checklists to assess preregistration producibility and preregstration-study consistency. More than 30 coders used these checklists to assess the studies in our sample. We selected our control group of non-preregistered studies based on the ‘related records’ function of Web of Science. We found that there is room for improvement for preregistration in psychology. Hypotheses, statistical models, and inference criteria were typically not very well described in preregistrations. Moreover, we found that the consistency between preregistrations and papers was low, mainly for data collection procedures and statistical models. More comprehensive preregstration templates did lead to more producible preregistrations. When comparing preregistered and non-preregistered studies we found no difference in the proportion of positive results, but preregistered studies were typically of higher quality and had more impact than non-preregistered studies.

In all, there is room for improvement with regard to the effectiveness of preregistration in the field of psychology. Although it could be that preregistrations, especially when they are based on a comprehensive template, prevent some questonable research practices, the practice of registered reports may be more promising.

Supervisors
Prof. dr. J.M. Wicherts, Prof. dr, M.A.L.M. van Assen, Dr. M. Bakker

Financed by
ERC

Period
September 2017 – March 2024