
Project
How trustworthy are simulation studies? A meta-research perspective
Monte-Carlo simulation studies are ubiquitous in the scientific literature. For example, in a random sample of papers from the journal Statistics in Medicine, 75% of research articles contained at least one simulation study (Morris et al., 2019). These simulation studies are, for example, used for evaluating the properties of statistical methods or for determining the required sample size by means of a power analysis in complex research designs. The design, conducting, reproducibility, and reporting of the results of simulations studies have hardly been studied from a meta-research perspective. The few studies that did investigate the current state of simulation replicability, reproducibility, and reporting standards, often argue for reform of standard practices of simulation studies to improve their quality (Luijken et al., 2024; Morris et al., 2019; Pawel et al., 2024). Studying this is especially relevant because researchers who conduct the simulation studies have a clear conflict of interest (Lohmann et al., 2022). Researchers generally want to showcase that their newly developed method performs well and/or outperforms existing state-of-the-art methods. Although the full extent of this problem is unclear, we have circumstantial evidence that the problem and its effect on the literature is substantial.
Topics and open questions that fall within the scope of the project include but are not limited to:
- Are simulation studies in the social sciences reproducible? Can the results of a simulation study be reproduced if computer code is shared or are there still barriers for reproduction?
- Are simulation studies at risk for being biased? That is, are the results (and conditions) of simulation studies more favorable to the new methods that are introduced in the same paper as the simulation studies?
- Do simulation studies that are set up by different independent researchers yield different results?
- How can a simulation study be preregistered? What recommendations can be made with respect to designing, conducting, and reporting the results of simulation studies?
This project fits within IOPS because it aims to evaluate and improve the way statistical methods (including psychometric and sociometric methods) are introduced into the literature by means of simulation studies.
Supervisors
Prof. Dr. Marcel van Assen
Dr. Robbie van Aert
Dr. Lieke Voncken
Financed by
Tilburg University
Period
1 September 2024 – 31 August 2028