Utrecht University
Faculty of Social Sciences
Methods & Statistics

Supervisors:
– prof.dr. Joop Hox (Utrecht University)
– dr. Barry Schouten (Statistics Netherlands)

On October 10th 2014 Thomas Klausch defended his thesis entitled

Informed Design of Mixed-Mode Surveys: Evaluating mode
effects on measurement and selection error
(Thesis not available yet)

Summary
A mixed-mode survey is an innovative method for collecting data in social research, which has increased greatly in world-wide popularity among survey researchers in the past decade. In a mixed-mode survey, researchers use multiple modes of communication – for example online, telephone, and in person – to ask questions to respondents. An advantage of mixed-mode surveys is that more people are reached, for example those who do not have an internet or a telephone connection or those who prefer a certain mode. However, choosing modes for mixed-mode surveys is not a simple task for researchers. One threat is that answers under some modes may have lower quality and it is questionable whether these should be used for mixed-mode designs. This dissertation contributes methodology that survey researchers can use to evaluate the effects of mode choice for the size of errors of statistics produced using mixed-mode surveys. In collaboration with Statistics Netherlands (Centraal Bureau voor de Statistiek / CBS), a large experiment was conducted, in which four different single-mode and three mixed-mode designs were compared with regard to so-called measurement and selection error for the case of the Crime Victimization Survey (CVS), a survey of safety in The Netherlands. It is found that the same types of respondents participate in different survey modes but that they may provide very different answers, depending on mode and question. Especially answers provided under the telephone and face-to-face modes differed from the web and paper modes in this respect. This result challenges the comparability of data collected using interviewers and self-administered questionnaires. It is recommended not to combine these modes in the CVS.

Project

Nonresponse and response bias in mixed-mode surveys
Mode bias is a nuisance in surveys using more than one survey mode (mixed-mode surveys) and longitudinal surveys that need to switch modes in the course of their lifetime. Sources of mode bias include mode-specific response propensity distributions of the population (causing mode-specific nonresponse error) and mode-, survey- and item-specific measurement distributions for each population unit (aggregating to mode-specific measurement errors). Mode biases are the aggregated net effects of these errors when comparing estimates from two or more modes. To date, both singular and generalizable knowledge on the size of these errors is scarce, but is keenly needed in order to assess the relative effects of mode-switches in mixed-mode and longitudinal surveys. Developing a common theory of the errors underlying mode bias and how they interact is the first goal of the research. Consequently, we will review and develop methods useful to assess the size of the errors based on empirical data from a parallel multi-mode experiment.

This project was financed by Utrecht University and  Statistics Netherlands (CBS).