Project
From Degrees of Freedom to Robustness: Strengthening the Evidence Base for Psychological Interventions

The main goal of this project is to systematically assess the methodological and statistical rigor of psychotherapy studies. Clinical psychologists and researchers evaluate the effectiveness of psychological interventions by conducting randomised controlled trials – the gold standard for effectiveness research (Hariton & Locascio, 2018). According to the Declaration of Helsinki, which sets the ethical standards for clinical trials in medicine and other disciplines, the interventional research of psychotherapy should register their RCTs (prospectively) in publicly accessible databases, with the intention to improve the transparency and the reliability (De Angelis et al., 2004). However, the majority of trials are still retrospectively registered or not registered at all, and frequently the registration and publication are not completely consistent (Bradley et al., 2017; Stoll et al., 2020). In addition, despite current standards in designing and reporting RCTs (Schulz et al., 2010), the robustness of the conclusion drawn from such studies can still be undermined by the researcher degrees of freedom in analysing the RCTs (Wicherts et al., 2016).
Therefore, this project plans to conduct four studies to investigate the methodological validity and statistical variability of mindfulness-based intervention studies to treat depression:
Study 1 is a systematic review that summarizes the design and the analysis plan and potential registration of published RCTs testing mindfulness-based interventions for depression. We aim to (1) evaluate the strictness of the registration, (2) to identify potential discrepancies between the registration and publication, and (3) to map which statistical models are used to test the mindfulness-based intervention effects. This systematic review also depicts an overview of the measurement models used in such RCTs.
Study 2 is a simulation study that aims to assess whether the statistical models identified in Project 1 provide unbiased estimates of a true intervention effect and whether power analyses with these models would result in substantially different minimum sample sizes than the required sample size reported in the studies from Project 1.
Then, we will request the individual patient data (IPD) from the original RCTs and conduct Study 3. In Study 3, we will assess the effect of statistical variability of published intervention effects by re-analysing the IPD, also conduct IPD meta-analyses on the effects and the individual differences of mindfulness-based intervention for depression. In addition, with the IPD, we can evaluate the psychometric properties of measures used in the original studies and perform sensitivity analyses to explore the impact of using measurement models on the meta-analytic findings.
In Study 4, we intend to synthesize the findings of previous projects by providing a set of guidelines to increase the validity of psychotherapy studies and registrations.
This project is relevant to IOPS because by systematically evaluating and comparing the statistical methodology used in clinical psychology studies, this project not only aims to strengthen the evidence base of mindfulness-based intervention studies, but also to develop guidelines that may further increase the statistical validity of psychological intervention studies in general, with a focus on measurement of outcomes.

Supervisors
Prof. dr. J.M. Wicherts
dr. P. Lodder
dr. M.B. Nuijte

Period
28 August 2023 – 1 September 2027