Pairwise comparisons within education
The method of pairwise comparison is a data collection and scaling method for preference data. The method is used to create a rank order of objects, which is achieved by comparing pairs of objects on a trait. The method of pairwise comparison has been known for a long time, but it has only recently received attention in education. In education, administering a complete design of pairwise comparisons (e.g., of student assignments) is often unfeasible for judges (e.g., teachers) due to the large number of comparisons to be executed. Therefore, comparisons are selected based on an adaptive algorithm, but available algorithms may not be optimal. This research investigates the application of the method of pairwise comparison in education. First, we plan to develop an adaptive pair-allocation algorithm with optimal outcome quality taking the measurement error of the ordering into account. Second, we apply paired comparison judgment to peer evaluation of non-cognitive traits. Third, we model the evaluation of scripts using a scale rather than a dichotomous choice. Lastly, we develop a generic model to estimate a rank order of objects based on either paired comparison data, rank order data or rubrics data. Because we want to take the uncertainty of the parameters into account, we use a Bayesian modeling approach, which is well suited for uncertainty modeling. To conclude, we contribute to the data collection procedures and modeling approaches of pairwise comparison data in education. In this research project, we investigate available procedures for pairwise comparison in education, develop new designs for data collection and enhance modeling approaches using a Bayesian framework.
Prof. Klaas Sijtsma & Dr Anton Béguin
Tilburg University & Cito
1 September 2016 – 1 September 2020