Project
Motives for Statistical Reporting Practices

Multiple studies point towards a lack of understanding of statistical inference in psychological research. For instance, Hoekstra et al. (2006) found that p-values were often misinterpreted in the literature. Hoekstra led another study to look into misinterpretations of confidence intervals, identifying that they were also inadequately understood (2014). Moreover, these comprehension problems are not restricted to the frequentist approach to statistics as Wong and colleague’s (2022) study on Bayes factor shows. Hence, it potentially is a conspicuous problem that permeates multiple statistical inferential approaches throughout psychological research.

However, the interpretation of statistical results differs even among experts (van Dongen et al., 2019). Therefore, it could be expected that researchers may have distinct, and even valid, interpretations that might have been considered wrong under the particular paradigm of a researcher. Furthermore, we still do not know if the aforementioned reporting practices are a result of a lack of understanding, or are instead caused by reporting standards, space restrictions in journals, or implicit context.

The first study will explore how researchers write reports about statistical, inferential studies to describe the degree to which hypotheses and results are intertwined in their argumentation. This should aid us in shedding light on how researchers reflect when writing about inferential results. Besides, if researchers do not link statistical results to theory beyond, e.g., a significant p-value, this could be, at least, a sign of difficulties in logical interpretation and, thus, in understanding deductive statistical inference.

The second study aims to interview researchers whose articles were analyzed in a study conducted by Tendeiro and colleagues (2024). The authors collected multiple studies and reported issues in reporting practices, concretely about Bayes factors. Surveying the authors of the analyzed articles should be conducive to understanding the why of such reporting practices; for example, space restriction in journals. Our interviews will also be informed by our first study since we will be probing authors about deductive statistical inference. In other words, we will have hypotheses about how researchers reflect that could be directly asked to other authors.

The third study will be about the perspective of peer-reviewers and editors. We will survey and interview them about what is needed to curate statistical misreporting and create a milieu that promotes statistical reasoning. Other specifics, such as space restriction in journals and the promotion of slow science (Stengers, 2017), will also be investigated.  

Finally, the fourth investigation will be a study testing our then collected hypotheses. In it, we could, for instance ask an experimental group to write a hypothesis, result, and discussion of a simulated study with all the present-day conventions, e.g., reporting a significant p-value. Another group will not have any restrictions on how to write their results. However, their statistical test will require researchers to reflect explicitly on the result. For instance, they will have to report Bayes factors and posterior odds (Tendeiro et al., 2024), or instead of p-values, they will report s-values (Rafi & Greenland, 2020). Finally, the last group will have the restriction frequently used in peer-reviewed journals, but their statistical results will require them to muse over more the result.  

This project is about how we can improve the understanding and interpretation of statistics in psychological research. We will look into the necessary training people may need to use new and existing methods. Furthermore, we think that IOPS provides useful training for our project as well as a network that can help expand it.    

Supervisors
Prof. Dr. Don van Ravenzwaaij
Prof. Dr. Henk Kiers
Dr. Rink Hoekstra

Financed by
University of Groningen

Period
1 December 2024 – 30 November 2028