Methodology and Statistics
Social and Behavioural Sciences
Utrecht University

Academic webpage

Summary

This dissertation focuses on the intersection of two scientific fields: natural language processing and measurement theory.
Natural language processing is a field of study that focuses on the ability of machines to manipulate natural languages. Examples of natural languages ​​include English, German, and Mandarin Chinese, which have evolved naturally over time through use and repetition by humans. They exhibit rich, diverse, and often irregular structures that are difficult for machines to handle. This distinguishes them from artificial languages ​​such as programming languages ​​and mathematical notation, which are designed with explicit rules. Research in natural language processing ranges from simple tasks, such as counting word frequencies in documents, to more complex tasks that require machines to “understand” human text and have meaningful conversations with humans.
Measurement theory, on the other hand, is concerned with scientifically measuring properties of objects, such as determining the length of a bar and assessing a person’s personality. It involves two key concepts: construct validity and reliability. The first concerns the question of whether a measure actually measures what it is supposed to measure, while the second concerns the question of whether a measure produces consistent results under different measurement conditions in which more or less the same results might be expected.
In this dissertation, I argue that many of the key challenges in natural language processing research are related to the problem of measurement, and that measurement theory, particularly from a social science perspective, prescribes both conceptual and practical tools that can be used to deal with measurement-related problems in natural language processing research. The main motivation behind this dissertation is therefore to identify measurement-related challenges in natural language processing research and to explore the use of measurement theory to address these challenges.

Project 
Content Validity of High-Dimensional Measurement

“Incidental data” from sources like smartphones, social media and search queries are  abundant. They are collected continually and record human behaviour in natural  environments. Therefore, such data can be ideal for measuring social phenomena. In fact,  some high-impact studies have made successful predictions of variables like human values  and personality from incidental data.  

However, when the research goal is not to predict but to explain (i.e. test a theory or  estimate a causal model), which is often the case in the social sciences, at least two major  challenges arise with the use of incidental data. First, in explanatory modelling, constructed  scores (of latent constructs) must not only correlate but also be reliable and valid.  Unfortunately, incidental data tend to suffer from measurement error problems because, by  definition, they are not collected for the purpose of scientific research. Therefore,  methods like latent variable measurement models are needed to estimate and correct for  measurement error in incidental data. Second, in incidental data, indicators (e.g. social  media posts and clickstreams) are often high-dimensional and most have low relevance to  the target concepts. This raises the problem of high-dimensional measurement, which  existing latent variable measurement models cannot deal with. 

The goal of my PhD project is thus to improve measurements of theoretical (latent)  constructs (like human values and personalities) based on high-dimensional incidental data,  with a focus on content validity. Specifically, my project entails: 1) developing a variable  selection procedure for high-dimensional incidental data by leveraging knowledge from  machine learning and computational linguistics; 2) developing a method for valid cross country comparisons with incidental data; 3) developing user-friendly R packages and JASP  modules for applied social scientists to use the developed methods; 4) applying the  developed methods to study human values in Germany and the Netherlands.

Supervisors
Dr. Daniel Oberski; Dr. Dong Nguyen

Financed by
NWO Talent Programme Vidi

Period
1 June 2020 – 31 May 2024