Making valid decisions using five sources of validity evidence

Melanie Hawkins, PhD
Swinburne University of Technology

I am grateful to the 2021 ISOQOL conference committee and reviewers for recognising the usefulness of my work by presenting me with the ISOQOL 2021 New Investigator Oral Award.

Data from patient-reported outcome measures (PROMs) are increasingly being used to guide decisions in health care. The term “valid” refers to how these data interpretations are appropriate, meaningful and useful for this decision making. This differs from the way it is commonly used in health sciences, to refer to the properties of a measurement instrument, such as a questionnaire.

In my conference presentation, I explained and provided PRO-related examples for the five sources of validity evidence, and demonstrated how a combination of evidence informs an argument about the extent to which PRO data are valid for intended decision making in a specific context.

The culmination of decades of work about validity, validation, and validity testing is presented in the Standards for Educational and Psychological Testing, which outlines a framework of evidence based on five sources:

1. the content of a questionnaire including the response options and scoring
2. response processes or the ways in which respondents interpret, think about, and respond to questions, as compared with the construct being measured
3. the internal structure of a questionnaire, which is about the ways and extent to which the questionnaire responses relate to each other and to the construct on which the score interpretations are based
4. relations to other variables, which is about the extent to which relationships are consistent between the questionnaire scores that exemplify the construct being measured and external variables (e.g., other questionnaires measuring related or different constructs, factors such as age and health conditions, and longer-term outcomes)
5. the consequences of collecting and interpreting the data, which is to do with the extent to which the data-informed decisions are appropriate, meaningful and useful within the measurement context, as well as the extent to which unintended negative consequences – or potential harm – are minimised.

We used the multidimensional Health Literacy Questionnaire (HLQ) as an example PROM to understand what helps people access, understand, appraise, remember, and use health information and services within the contexts of their everyday lives, especially for people experiencing disadvantage.

In our study, we have brought and applied the theoretical rationales of Samuel J Messick and the Standards, as well as Michael T Kane’s argument-based approach to validation, to the field of PRO measurement. See our references in the full journal article here. The framework of the five sources of validity evidence enables PRO researchers to plan organized and logical testing of new PRO instruments and of PRO instruments in new contexts. The framework also enables structured reporting of validity evidence to help prospective users determine if an instrument has the potential to generate data that are valid for the decision making needed in their study context.

This newsletter editorial represents the views of the author and does not necessarily reflect the views of ISOQOL. 

How to Submit a Newsletter Editorial
Do you have something to share about health related quality of life and patient-centered outcomes? We want to hear from you!
Learn More