University of Tilburg
Department of Methodology and Statistics

Project

(Data-dependent) choices and (statistical) consequences in psychology

My PhD project focuses on how seemingly arbitrary decisions in the research process affect statistical outcomes, in the context of psychology. That apparently innocuous choices in the research process can be important has received increasing attention ever since Simmons, Nelson and Simonsohn (2011) demonstrated how the existence of multiple alternative analytic options can lead a researcher to find “evidence” for almost any claim. Such researchers’ degrees of freedom (Wicherts et al., 2016), include amongst many possibilities choices about primary outcome, scoring of items or scales, or which statistical model to use. 

Despite increasing awareness (e.g., Steegen et al., 2016; Orben & Przybylski, 2019) that what seem like innocent choices in the research process can have large consequences for final estimates, it is often unclear which choices are important. In addition, when a behavior is recognized as potentially problematic (as with reporting p­-values as “marginally significant”), it can be unclear how widespread the behavior is (although see John, Loewenstein, & Prelec, 2012), and hence how concerned we should be. As such, we believe further research on the prevalence and consequences of flexible (usually, data-dependent) decision-making in the research process is timely, and has the potential to contribute to increased replicability, transparency and research integrity in psychological science.  The PhD project consists of several sub-projects, of which three have been designed so far. First, an examination of how often researchers in psychology report p-values between .05 and .1 as ‘marginally significant’ (published; Olsson-Collentine, van Assen & Hartgerink, 2019). Since researchers tend to use an (implicitly) predefined alpha level, later reporting results as marginally significant is an example of an implicit change in the decision rule. How problematic this is depends on the extent to which the decision rule has been altered. Second, an examination of the consequences of seemingly minor changes in experimental location and settings in direct replications in social and cognitive psychology (in review). We found that the consequences are typically small and hence an unlikely explanation for when there are large differences in outcome between direct replications. Third, study level choices can create uncertainty in meta-analytic outcomes that are not currently accounted for. We explore how flexible decision-making at the study-level can affect meta-analytic summaries. We expect this third sub-project to be published in 2020, and further sub-projects will be planned in the course of the PhD-project.

Supervisors

Prof. dr. Jelte Wicherts, dr. Marjan Bakker

Financed by

Tilburg University

Period

2018-10-01 – 2022-10-01