Jurian Meijering

Foto JurianBiometris
Wageningen University

Supervisors
Prof. Adri Van den Brink & Dr Hilde Tobi

On December 13th 2016, Jurian Meijering will defend his thesis entitled

Probing the Power of Apollo: Methodological Challenges and Opportunities of the Delphi Method for Developing Rankings

Cover PhD thesis Jurian MeijeringAbstract thesis

The Delphi method is a structured data-collection process aimed at facilitating experts to achieve a certain level of agreement on a complex problem. In several subsequent rounds experts give their opinion on the problem of interest, usually by means of a standardized questionnaire. After each round experts receive controlled opinion feedback in the form of a summary of the findings. Based on this feedback experts are allowed to reconsider their opinion in the next round.

Although the Delphi method seems to provide opportunities for developing rankings, research into its application to the development of rankings seems to be non-existent. Furthermore, the Delphi method has several unresolved methodological issues, in particular regarding the measurement of agreement and the provision of controlled opinion feedback. Therefore, the general objective of this PhD project was to find out what the methodological challenges and opportunities are of the Delphi method for developing rankings.

To achieve this objective, four studies were conducted. In the first study, simulations were performed to find out how various agreement indices behave within and across the rounds of a Delphi study. In the second study, the Delphi method was applied to develop a ranking of research domains in landscape architecture. An experiment within the study investigated the effect of two types of controlled opinion feedback on the drop-out rate, experts’ degree of opinion change, and the level of agreement among experts. The third study investigated the methodological characteristics of six urban sustainability rankings, which resulted in a recommendation to define and operationalize the concept ‘urban sustainability’ using the Delphi method. Therefore, in the fourth and final study the Delphi method was applied to find out which components experts find most relevant for defining and measuring urban sustainability. An experiment within this study provided insight into the effect of feeding back experts’ initial ratings on the degree of opinion change and the level of agreement.

Overall, this PhD project showed how the Delphi method may be used to: (1) obtain a ranking of objects on a ranking attribute and (2) obtain a definition and operationalization of a complex ranking attribute. It was also shown that these applications of the Delphi method do not come without challenges. First, selection criteria and search strategies need to be developed by which sufficient numbers of different types of experts may be found. Second, for the first Delphi questionnaire a parsimonious list of items (e.g. ranking objects or components of a ranking attribute) needs to be drawn up that largely covers the potentially great diversity of existing items. Third, careful considerations need to be made about which types of information (i.e. summary statistics, rationales, experts’ own initial ratings) to feed back to experts after every round as this may influence various Delphi outcome measures. Fourth, the choice for a particular consensus, agreement, or association index needs to be carefully made and justified as it determines the level of agreement among experts that is obtained. Finally, project showed that by means of a simulation study and two real-world Delphi experiments new knowledge about the functioning of the Delphi method could be acquired. More of these studies are needed to establish evidence-based guidelines and to uncover the full potential of the Delphi method for developing rankings.

Project

The Delphi method: methodological issues and its application to the development of rankings
The Delphi method is a structured data-collection process aimed at acquiring a certain level of agreement among experts on a given topic. The method seems to provide opportunities for developing expert-based and indicator-based rankings. However, research into its application to the development of rankings seems to be non-existent. Furthermore, the Delphi method has several unresolved methodological issues. Therefore, this PhD project aims to improve the methodology of the Delphi method and investigate its application to the development of rankings by conducting four studies. In the first study, simulations were performed to find out how various agreement indices behave within and across the rounds of a Delphi study. In the second study, the Delphi method was applied to acquire an expert-based ranking. An experiment within the study investigated the effect of two types of controlled opinion feedback on the drop-out rate, experts’ degree of opinion change, and the level of agreement among experts. Additionally, the study provided insight into the opportunities and challenges of the Delphi method for developing an expert-based ranking. In the third study, content analysis and interviews were conducted to identify and evaluate the methodological characteristics of six indicator-based European green city rankings. In a subsequent and final study the Delphi method will be applied to gain insight into the opportunities and challenges of the method for selecting an weighing indicators for an indicator based ranking.

Financed by
Wageningen School of Social Sciences