VU University Amsterdam
Prof. Henk Kelderman, Prof. Pim Cuijpers, Dr Niels Smits
On June 17th 2015, Marjolein Fokkema defended her thesis entitled
Psychometric Contributions to Improving the Efficiency and Fidelity of Clinical Assessment and Research
In clinical practice, assessments are often performed for making decisions: for example, assessment may be aimed at deciding which treatment should be provided to a patient. By taking the decision for which assessment is performed into account, the accuracy and efficiency of assessment can often be improved (Cronbach & Gleser, 1965).
In Chapter 1, I discussed and applied curtailment, an algorithm that allows for early stopping of item administration when questionnaires are used for binary classification decisions. Application of curtailment to three mental health questionnaires was found to result in substantial test length reductions, without reducing diagnostic accuracy.
In Chapter 2, I presented a new algorithm for assessment length reduction of tests batteries: CART-SC. CART-SC combines a classification tree (CART) with application of stochastic curtailment (SC) in every node of the tree, allowing for reduction of both the number of tests administered within a battery, as well as the number of items administered within tests. By simulating application of CART-SC on an existing dataset of responses to a psychological test battery, substantial assessment length reductions could be obtained, without reducing diagnostic accuracy.
The CART algorithm is a powerful method for selecting relevant attributes for decision-making in clinical practice, because CART trees provide sequential testing plans. In addition, CART can deal with a large number of potential predictor variables, can be used for subgroup detection, and allows for automatic detection of interactions between predictor variables. These characteristics are shared by all recursive partitioning methods (RPMs). Therefore, RPMs seem preeminently suited for improving the efficacy of clinical assessment, and more appropriate than many of the data-analytic methods traditionally used in clinical research. In Chapters 3 and 4, I further discussed the potential of RPMs for improving the efficacy of clinical assessment.
Clinical decisions are often based on predictions of future behavior. Therefore, in Chapter 3, I show that by deriving rules from decision trees, we can create a prediction rule ensemble. By reanalyzing a dataset from an earlier study, I show that a prediction rule ensemble allows for more efficient decision making in clinical practice, while providing predictive accuracy similar to that of logistic regression analysis, which was originally applied to the dataset.
In Chapter 4, I introduce an algorithm that can automatically detect patient characteristics that are predictive of treatment outcome in datasets in which the results of several clinical trials have been pooled. The results of the algorithm can be represented as a tree, and in a simulation I show that the algorithm performs very accurately in recovering patients characteristics that are predictive of treatment outcome.
Finally, the accuracy of predictions of treatment effects depends not only on the data-analytic method used, but also on the accuracy with which treatment outcomes are measured. Biased measurement of treatment outcomes will likely introduce bias in the estimation of a predictive model. Therefore, in Chapter 5, I discussed response-shift bias: changes in the measurement models underlying total scores on self-report inventories. As an illustration, I examined the measurement models underlying a self-report inventory from an influential trial comparing the effects of four treatments for depression. Results show that, compared to before treatment, after treatment, item scores overestimate depressive symptomatology, measurement errors are smaller and there is a stronger association between the underlying constructs. These effects were more apparent in psychotherapy groups, than in pharmacotherapy groups. These changes indicate that response shifts have occurred, and that observed-score comparisons over time yield confounded measures of treatment efficacy.
Measurement Issues in Clinical Psychology
A considerable problem in mental health testing is the multitude of questionnaires used for clinical assessment. This has negative effects, such as the unwillingness to participate in internet therapy. In this project we develop a method for short clinical examination, fast adaptive diagnostic assessment (FADA), which unites two methods for reducing assessment time. Computerized Adaptive Testing is used to shorten the administration of each questionnaire. Decision trees are used to select a short sequence of questionnaires which is most informative for predicting diagnostic class. In four projects, the hybrid model is gradually refined, to come to an optimal model for FADA.
This project was financed by VU University Amsterdam