Marvin Neumann

Behavioral and Social Sciences
Psychometrics & Statistics
University of Groningen

Project

Bridging the scientist-practitioner gap in judgment and prediction

My PhD project is concerned with the scientist-practitioner gap that exists in test-use and decision-making in, for example, the context of personnel selection and admission to higher education. I will focus on the important gap that exists with regard to the use of clinical vs. actuarial prediction methods. Previous research has demonstrated that, on average, actuarial prediction outperforms clinical prediction (Kuncel, Klieger, Connelly, & Ones, 2013). The superiority of actuarial prediction is not a new research topic (Meehl, 1954; Sawyer, 1966). However, implementation of actuarial prediction procedures, and evidence-based assessment in general is greatly lacking in practice. This creates a strong need for future research that helps practitioners to understand the value of evidence-based testing and decision-making. Therefore, in this PhD project, I will conduct several experiments that investigate how to best communicate the psychometric properties of assessment procedures and advantages of evidence-based assessment (actuarial prediction and standardized testing) to practitioners. Variables of interest are utility information in the form of Taylor-Russell tables (success ratio, selection ratio, base rate, predictive validity), practitioners’ attitudinal measures (confidence in prediction, satisfaction with prediction), factors associated with resistance of practitioners (e.g. autonomy, knowledge, method of communicating evidence) and actual implementation in practice. Previous studies showed that the lack of predictive validity that results from clinical prediction is often due to the inconsistency with which judges weight different information across subjects. Therefore, increasing the judgment consistency of practitioners is a main topic in this project.

People show higher intentions to use evidence-based testing and decision making (structured interviews and actuarial prediction) when these procedures offer more rather than less autonomy potential (Nolan & Highhouse, 2014). However, no research has simultaneously looked at the potential validity loss that could result if assessment and decision-making procedures offer more autonomy potential. This tradeoff between autonomy potential and validity will be further explored in my first study. Differences in predictive validity of different judgment procedures that vary in autonomy potential will be investigated, together with participants’ intention to use and actual use of these judgment procedures. In several studies, I will describe and test interventions that present the psychometric properties of evidence-based assessment in a simple and understandable manner to students and practitioners in the field (such as human resource managers and admission officers). This entails the presentation of all sorts of psychometric information that is relevant in the context of selection. Furthermore, the lens model (Kuncel et al., 2013; Yu, 2018) serves as an overarching framework that allows to model, and to compare the subject’s clinical judgment with a mechanical judgment based on regression analyses. This project results in guidelines and best practices that aid the implementation of evidence-based test use and assessment practices and contributes to narrowing the scientist-practitioner gap that exists in this context.

Supervisors
Prof. dr. R.R. Meijer, dr. A.S.M. Niessen, dr. J.N. Tendeiro

Financed by
University of Groningen

Period

1 September 2018 – 1 September 2022