Ben Kretzler

Meta-Research Center
Methodology and Statistics
Tilburg School of Social and Behavioral Sciences
Tilburg University

Email

Project
Testing Causal Effects Variation

If we open a psychology journal and randomly select an empirical article reporting a significant result, chances are that a direct replication would not yield the same outcome (Open Science Collaboration, 2015). This replication problem is often attributed to a fragmented theoretical landscape, underpowered studies, and undisclosed flexibility in data analyses (Nosek et al., 2021). Recent discussions, however, suggest that treatment effects are inherently heterogeneous, highlighting the need for research designs and analyses that account for this variability (Bryan et al., 2021).

Our PhD project aims to (i) add to our current knowledge about the extent of heterogeneity in causal effects, (ii) help evaluate the validity of our current state of knowledge about such heterogeneity, and (iii) develop new tools for psychology’s toolbox to detect heterogeneity.

  • The first project consists of a meta-meta-analysis on gender moderation effects. We synthesize data from a representative selection of psychological meta-analyses and consider alternative ways to test gender moderation across sets of randomized experiments in different research lines. This will allow us to establish benchmarks for gender moderation effect sizes and to study power to detect gender moderation in both primary studies and meta-analyses (cf. Wallach, 2016, for a related project about medical randomized controlled trials that concluded that such analyses suffer from intolerable Type I and II error rates).
  • The second project re-analyzes data from the ManyLabs studies to estimate the gender moderation effects not subject to selection biases. In turn, these benchmarks could inform theory development (e.g., regarding gender differences/similarities; Hyde, 2005), research designs (e.g., by providing input for power analyses), and potentially help to assess whether research conducted so far was able to detect gender moderation effects with commonly used sample sizes and methods.
  • The third project will explore a new modeling approach for heterogeneity based on the covariance between different multiverses, including partially overlapping subgroup analyses. Heterogeneity is commonly tested using the Q test, which is underpowered with standard sample sizes (Borenstein et al., 2009). We will test whether our alternative approach bears improvements in detecting heterogeneity.
  • Finally, we will evaluate our new approach in comparison with both older (e.g., the Q test) and newer (e.g., covariance regression; Bloome & Schrage, 2019) approaches to detect heterogeneity. This could inform researchers and ultimately enhance the reliability of moderation effects identified in the literature.

This research aligns closely with the IOPS framework: it evaluates the feasibility of current moderation testing methods common in psychological research, provides effect size benchmarks and power estimates, and likely expands psychology’s methodological toolkit by introducing and assessing new approaches for detecting heterogeneity. Moreover, our project complements ongoing IOPS projects on moderation effects in meta-analyses and the validation of the multiverse model for identifying p-hacking and other sources of variability.

Supervisors
Prof. Dr. Marcel van Assen
Prof. Dr. Jelte Wicherts
Dr. Robbie van Aert

Financed by
Vici grant (VI.C.221.100) “Examining Variation in Causal Effects in Psychology (EVINCE Psychology)”

Period
1 September 2024 – 31 August 2028