Project
Addressing Validity Threats in Models with Response Times for International Large-Scale Assessment

With the advance of computerized assessments, process data such as response times are now commonly available when respondents take psychological or educational tests. Recently, psychometric models have been developed that make use of response times to improve the precision with which ability is measured (van der Linden, 2007; Bolsinova & Tijmstra, 2018). While these models allow for an impactful improvement in measurement quality if their assumptions are met, there are important threats to validity when assumptions of the model about the behaviour of response times do not hold (Tijmstra & Bolsinova, 2018). For example, it is often assumed that there is a linear relationship between speed and ability, which is also sometimes assumed to be the same for every group that is studied. When these assumptions are not met, researchers may unknowingly draw the wrong conclusions based on their analyses.

Unfortunately, these threats are commonly ignored in practice when models that include response times are used, despite there being plenty of reasons to expect impactful violations in practice (Bolsinova, Tijmstra, Molenaar & De Boeck, 2017). With international large-scale assessment programs such as the Program for International Student Assessment (PISA) moving towards adopting such response time models, it is becoming crucially important that tools are developed for assessing the validity of these models. Since the results of these programs have far-reaching practical consequences for educational policy and the comparison of countries, these tools will need to be developed before a wide-scale adoption of these methods can be defended.

This project focuses on the development and application of exactly these tools. Ideally, such tools should focus not just on formal model misfit (see for example Bolsinova & Tijmstra, 2016) but should actively consider the validity implications of applying the model despite the presence of violations. Since no response time model will be fully applicable to any real-life situation, let alone to the complexities of international large-scale assessment, this project focuses on providing tools that will help practitioners decide whether the use of response time models should be considered defensible in their specific context. Through both empirical analyses and simulation studies, the impact of using miss-specified response time models on the inferences of interest (e.g., country rankings) will be explored, and tools will be developed that aim to capture the sensitivity of these outcomes to various forms of model misspecification. With this, the project aims to provide a contribution to safeguarding and improving the quality of measurement in the field of international large-scale assessment.

Supervisors
Prof. Dr. Jeroen Vermunt
Dr. Jesper Tijmstra

Financed by
Tilburg University

Period
1 October 2024 – 1 October 2028