OBJECTIVES: In health preference research, respondents’ comprehension of information and interpretation of questions is critical for obtaining meaningful preference measures and improving response efficiency. This study reports a review of selected health preference studies to describe how comprehension was assessed in the survey instrument, evaluate the responses to comprehension questions, and describe whether the responses were used in analysis of the preference questions.
METHODS: We reviewed 25 stated preference surveys that used comprehension questions to assess respondent comprehension and retention of information provided in the survey. Questions were classified into three groups: attribute comprehension, general information comprehension, and choice question context. We calculated the percent of respondents who answered each comprehension question incorrectly and summarized by question type and respondent type. The review also examined whether and how comprehension question responses were used in analysis.
RESULTS: Sixty percent of the surveys included more than one comprehension question, for an average of 2.5 questions per survey. The percentage of respondents providing incorrect responses ranged from 1% to 69% across the surveys. Surveys of the general population had the highest average percentage of incorrect responses (28%) and surveys of caregivers had the lowest average (11%). In total, 46% of the surveys used the responses to comprehension questions in the analysis of the preference data, usually to examine subgroup differences.
CONCLUSIONS: These results contribute to a growing understanding of the performance of comprehension questions in health preference studies. Responses to comprehension questions may provide an indication of data quality that can be incorporated into the statistical analysis of preferences. Future research could further evaluate the effect of incorrect responses on preferences and investigate the correlation between respondent characteristics and comprehension.
