Sponsored by FACITgroup

Description:

With the increasing effort to implement patient-reported outcome measures (PROMs) and other patient-centered outcome measures in clinical practice and to generate real word evidence, the field is facing various challenges in terms of continuous assessment (i.e. measurement challenges during life transition phases; e.g. transition from infancy to childhood, adolescents to adulthood or late adulthood to old age). This plenary session focused on theoretical approaches, methods and applications to tackle challenges in measuring health related quality of life (HRQL) across the life course, addressing aspects of standardization and domain-based measurement.

Speakers:

Christopher Forrest, MD PhD, Children’s Hospital of Philadelphia, Philadelphia, Pennsylvania, United States

Dr. Forrest serves as the Secretary-Treasurer on the ISQOOL Board of Directors. He is a pediatrician who leads several national collaborations that are advancing our knowledge on how to improve individual and population health and well-being. He is the founder of PEDSnet, a national network that has created an analysis-ready database for >12 million children, with data from 2009 to present and updated quarterly. His research group has developed >50 pediatric patient-reported outcome measures. He co-edited the new Handbook of Life Course Health Development, which has been downloaded over 700,000 times from the Springer web site.

Richard Gershon, PhD, Northwestern University, Chicago, Illinois, United States

Realizing that the outcomes tools he was using in clinical practice failed to cover the needs of his patients, Dr. Gershon focused his career on the development of modern assessment tools. He is PI for NIH Toolbox for the Assessment of Neurological and Behavioral Function, and NIH Infant and Toddler Toolbox. He is MPI for Advancing Reliable Measurement in Alzheimer’s Disease and cognitive Aging, and Environmental influences on Child Health Outcomes PRO Measurement Core. He is former PI for NIH Roadmap Patient Reported Outcomes Measurement Information System Technical Center, and consults on 100+ assessment projects in healthcare, education, and certification.

Melissa Tinsley, Executive Masters, Public Administration, Agency for Clinical Innovation, Sydney, Australia

Mel Tinsley, is the Associate Director, Integrated Digital Enablement Accelerator at the Agency for Clinical Innovation. Mel leads a diverse and dynamic team with a focus on driving system transformation by bringing people, process, and technology together – to improve the outcomes and experiences for people accessing and receiving care, as well as improve experiences for healthcare teams delivering care. Currently Mel’s priority program areas include Patient Reported Measures, e-enabling Clinical Quality Registries, Virtual Care, Leading Better Value Care eMR builds and clinical applications – all contributing to the Value Based Healthcare approach.

Sponsored by EORTC

Description:

The Cutting Edge Research plenary session featured some of the highest-ranked, innovative research from ISOQOL abstract submissions. In particular, these abstracts reflected research that truly “pushes the ISOQOL envelope” in providing new and different ways to look at quality of life.

Speakers:

Berend Terluin, MD PhD, Amsterdam University Medical Center, Amsterdam, the Netherlands

Minimal important change (MIC) estimated using longitudinal structural equation modeling

I am a retired family physician with an interest in mental health. As a senior researcher I am affiliated with the Department of General Practice at the Amsterdam University Medical Centers in Amsterdam, the Netherlands. I am particularly interested in the estimation of the “minimal important change” (MIC). I am fond of Monte Carlo simulation as a method to investigate and improve the MIC-methodology. I have authored several articles about this subject over the past decade. I love to collaborate with my colleagues in the Netherlands and others in Denmark, England, France, and other countries.

Paul Schneider, MD MSc, University of Sheffield, Sheffield, United Kingdom

The Online elicitation of Personal Utility Functions (OPUF) Tool: a new method to construct (patient) preferences for small groups and on the individual level

Paul Schneider is a PhD student in the Wellcome Trust Doctoral Training Centre for Public Health, Economics, and Decision Science at the University of Sheffield, UK. His research focuses on the valuation of health and wellbeing for economic evaluations, using advanced preference elicitation methods.

Paul has an MD and a doctoral degree in medicine (health service research) from the University of Witten/Herdecke, Germany, and a research MSc in Health Sciences from Maastricht University, the Netherlands. He is a keen advocate of open science practices.

Tolulope Sajobi, PhD, University of Calgary, Calgary, Alberta, Canada

Tree-based structural equation modelling for assessing measurement invariance in patient-reported outcome measures

Dr Tolu Sajobi is an Associate Professor of Biostatistics in the Departments of Community Health Sciences and Clinical Neurosciences at the University of Calgary Cumming School of Medicine in Canada. His research program explores novel methodologies for the analysis of longitudinal patient-reported outcomes and design of randomized clinical trials. He leads a nationally-funded study exploring the novelty of machine learning methods for understanding population heterogeneity in patient-reported outcomes. As a research methodologist, Dr. Sajobi is a seasoned collaborator and statistical leader on several research studies in several chronic disease populations.

Carolyn Schwartz, ScD, DeltaQuest Foundation and Tufts University School of Medicine, Concord, Massachusetts, United States

Listening to the elephant in the room: Response-shift effects in clinical trials research

Dr. Carolyn Schwartz is President & Chief Scientist at the not-for-profit DeltaQuest Foundation; and Adjunct Research Professor of Medicine and Orthopaedic Surgery at the Tufts University School of Medicine. The 2016 recipient of the ISOQOL President’s Award, her interdisciplinary and methodological research focuses on understanding what patients can do to have an impact on the course of their disease and their well-being. In addition to development of theory and methods for response-shift research in quality of life, her work has spanned a number of diseases and conditions, and she has developed over 20 PRO measures.

Description:

Over the last couple of years, we have seen various efforts to implement patient-reported outcomes (PROs) sustainably in clinical practice. While there are a couple of unique projects, extensive implementation of PROs in the health care sector has not yet been achieved. This plenary session focused on application and implementation of PROs in clinical practice. Speakers shared their experiences and lessons learned when implementing PROs in clinical practice: what works, where they face challenges, new approaches and future perspectives.

Speakers:

Annette Ladefoged de Thurah, RN MPH PhD, Aarhus Universitetshospital, Aarhus N, Denmark

Annette de Thurah, RN, MPH. PhD in epidemiology in 2010. Professor of health services research in rheumatology at Department of Rheumatology, Aarhus University Hospital and Department of Clinical Medicine, Aarhus University. Extended knowledge in leading projects within telehealth care and implementation of PROs into daily clinical practice. She has collaborated with AmbuFlex for almost 10 years and has implemented several AmbuFlex solutions within rheumatology. She has been supervisor in two AmbuFlex PhD studies (epilepsy: co-supervisor, renal failure: main supervisor) and are teaching mater classes in the use of PROs.

Morten Atke Nüchel, BS, SUMOstudio, Aarhus, Denmark

I was born in town in Jutland – Denmark. After finishing business school I was in retail apprenticeship. After completion I was headhunted to help start up Denmark’s first private owned telecommunications company. From tele I pivoted to IT and from IT to tech. I have started + 10 online and tech companies and are currently owner/partner of 5 tech companies. Privately I married to Mille and have a son age 22.

Adesola Odole, PhD, Department of Physiotherapy, College of Medicine, University of Ibadan and Department of Physiotherapy, University College Hospital, Ibadan, Nigeria

Dr Adesola Odole is a Reader/Associate Professor at the Department of Physiotherapy, College of Medicine, University of Ibadan, Nigeria and Honorary Consultant in musculoskeletal Physiotherapy at the University College Hospital, Ibadan, Nigeria. She is the Coordinator, E-learning Committee and Assistant Coordinator of M.Sc. Biomedical degree programme, College of Medicine, University of Ibadan. She is a co-chair, developing nations special interest group and member of ISOQOL membership committee. Her research and clinical interests include musculoskeletal health outcomes, patient-reported outcomes assessment, rating scales development/cross-cultural adaptation of existing ones, evidence-based practice, telemedicine, and biomedical education.

Panelist:

Joanne Greenhalgh, PhD, University of Leeds, School of Sociology and Social Policy, United Kingdom

Sponsored by Clarivate

Description:

In the field of patient-centered outcomes research, meaningful change has been an important topic in recent years with multiple presentations at previous annual conferences. This plenary session discussed different theoretical concepts (minimally important difference, valuable/meaningful change, responder definition) and methodological approaches and how different stakeholders (patients, regulatory authorities, payers/health technology assessment bodies, clinicians, researchers) evaluate different approaches of establishing meaningful change or responder criteria in relation to the setting and contextual factors. This plenary provided an opportunity to debate current challenges and gaps assessed by different stakeholders, best practices and future directions.

Speakers:

Sonya Eremenco, MA, Critical Path Institute, Tucson, Arizona, United States

Sonya Eremenco is Executive Director of the Patient-Reported Outcome (PRO) Consortium at the Critical Path Institute (C-Path). Sonya has 25 years of experience in clinical outcome assessment development, with a focus on multicultural development, linguistic validation, and electronic implementation. She chaired an ISPOR PRO Task Force on use of mixed modes of PRO data collection in clinical trials. She currently serves on the Steering Committee of ISOQOL’s Translation and Cultural Adaptation Special Interest Group. Sonya holds a Bachelor of Arts in Cultural Anthropology from Duke University and a Master of Arts in Multicultural Communication from DePaul University in Chicago, Illinois.

Beate Wieseler, Dr. rer. nat., Institute for Quality and Efficiency in Health Care (IQWIG), Cologne, Germany

Beate Wieseler is Head of the Department of Drug Assessment at the German Institute for Quality and Efficiency in Health Care (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, IQWiG). At IQWiG she is responsible for the scientific assessment of pharmaceuticals, the development of assessment methods as well as in the Institute’s collaboration with German and international external parties. Prior to joining IQWiG in 2005, Beate Wieseler worked in clinical research and regulatory affairs for about 10 years. Beate Wieseler holds a Diploma in Biology from the University of Bonn and a Dr. rer. nat. from the University of Freiburg, Germany.

John Peipert, PhD, Northwestern University, Chicago, Illinois, United States

Devin Peipert is an Assistant Professor in the Department of Medical Social Sciences at Northwestern University Feinberg School of Medicine in the United States. There, he focuses on the application of patient reported outcomes (PROs) in patient focused drug development and in clinical monitoring to optimize patient management. He also develops new approaches to quantify and manage drug intolerability, particularly in oncology. Methodologically, he is focused on developing and evaluating ways to capture whether and how individual patients change on PROs, particularly in response to interventions.

Description:

A repeated finding is that response shift is not a self-evident or easy to understand term and many researchers attach different meanings to it. With the Response Shift – in Sync initiative we aimed to contribute to a common language regarding definition, theory, and methods based on a synthesis of the response shift research to date. This initiative turned out to be a great learning experience and we wanted to share the lessons we have learned. We also wanted to invite the roundtable participants to provide their perspectives and discuss how response shift can enrich our research into quality of life.

Hosts:

Mirjam Sprangers, PhD
Amsterdam University Medical Centers,
Amsterdam, Netherlands

Véronique Sébille Rivain, PhD
University of Nantes,
Nantes, France

Description:

Implementing PROMs presents a significant challenge, particularly in the pediatric patient population. During this roundtable we aimed to discuss different challenges and potential solutions related to using PROMs in pediatric care.

Host:

Lotte Haverman, PhD
Emma Children’s Hospital – Amsterdam University Medical Centers,
Amsterdam, Netherlands

Description:

At this roundtable, we invited participants to discuss considerations for mitigating missing data in clinical research studies, including reasons for missing data, as well as handling missing data in the context of longitudinal analysis of patient-reported outcomes. Examples of reasons for missing data were shared as well as statistical models for handling missing data and the estimand framework. Participants were encouraged to bring case studies to the interaction for discussion during the roundtable.

Host:

Libby Floden, PhD MPH
Clinical Outcomes Solutions
Chicago, Illinois, United States

Description:

This session discussed the implementation of wearable sensors, and digital health tools more broadly, into research activities. It is easy to “chuck in a fit bit”, but harder to design a study which both measure patient relevant concepts of interest and analyse them in a way which will elicit meaningful and actionable results. This discussion does not start with all the answers – they do not yet exist, but attendees worked together to discuss ways to further this field, including starting ISOQOL research initiatives.

Host:

Pip Griffiths, PhD
IQVIA
Paris, France

Description:

This roundtable presented the basics of computerized adaptive testing (CAT), differences to standard static questionnaires, advantages and challenges of using CAT for PRO assessment, and steps for the development of CAT and item banks. Participants were encouraged to share specific questions and experiences of PRO-CAT assessments hopefully resulting in fruitful discussions and exchanges of experience.

Host:

Morten Aagaard Petersen, MSCi
Bispebjerg & Frederiksberg Hospital,
Copenhagen, Denmark

Description:

In this session, participants discussed the challenges as well as opportunities that COVID-19 has presented with respect to quality of life research and clinical practice.

Host:

Kyle Kemp, PhD
University of Calgary and Health Quality Council of Alberta,
Calgary, Alberta, Canada

Moderator:

Bellinda King-Kallimanis, PhD, LUNGevity Foundation, Bethesda, Maryland, United States

Individual Presentations:

Patient-focused communication of patient reported outcomes using data visualization techniques

Katherine Bevans, PhD, Janssen Pharmaceuticals, Horsham, Pennsylvania, United States

Co-designing a resource sheet to aid Prostate Cancer patient’s interpretation of PRO data from a clinical trial

Matthew Reaney, IQVIA, Reading, United Kingdom

More than words: A targeted review of qualitative data visualization in PROM development studies

Manraj Kaur, PT PhD, Brigham and Women’s Hospital/Harvard Medical School, Boston, United States

Visualizing Psychometric Results – The first step towards standardizing interpretable figures

Pip Griffiths, PhD, IQVIA, Paris, France

Moderator:

Claire Snyder, PhD, Johns Hopkins, Baltimore, Maryland, United States

Individual Presentations:

Tools to optimize use of PROs in clinical practice

Norah Crossnohere, PhD, The Ohio State University, Columbus, Ohio, United States

Strategies for implementing and disseminating the tools to optimize PROs in clinical practice

Michael Brundage, MD MSc, Queen’s University, Kingston, Ontario, Canada

Moderator:

Felix Fischer, PhD, Charite – Universitätsmedizin Berlin, Berlin, Germany

Discussant:

Angela Stover, PhD, Assistant Professor of Health Policy and Management at Chapel Hill, Chapel Hill, North Carolina, United States

Individual Presentations:

Multidimensional Assessments and Reliability Estimates

Aaron Kaat, PhD, Northwestern University, Chicago, Illinois, United States

Analysis of meaninful within-individual change – comparison of longitudinal item response theory and traditional statistical methods

Jakob Bjorner, MD PhD, QualityMetric, Johnston, Rhode Island, United States

Modeling general populations’ reference data for PROMIS item banks (Physical Functioning, Upper Extremities, and Pain Interference) in multiple countries using quantile regression

Constantin Yves Plessen, Charité University Medicine Berlin, Berlin, Germany

Using graphs to improve the interpretation of test scores and their measurement precision

Muirne Paap, PhD, University of Groningen, Groningen, the Netherlands

Moderator and Discussant:

Leah McClimans, University of South Carolina, Columbia, South Carolina, United States

Individual Presentations:

Epistemological goals and the use of HRQL and patient-reported outcome measures

Jan Boehnke, School of Health Sciences, University of Dundee, Dundee, United Kingdom

Where Does “Validity” End and “Fit for Purpose” Begin? Towards a Typology for Implementation Validity

Rebecca Jackson, MA, Indiana University Bloomington, Bloomington, Indiana, United States

Moral Considerations in Determining the Sufficiency of Validation Evidence

Kevin Weinfurt, PhD, Duke University School of Medicine, Durham, North Carolina, United States

Moderator:

Christopher Sidey-Gibbons, PhD, UT MD Anderson Cancer Center, Houston, Texas, United States

Individual Presentations:

The first step in patient-centered outcomes research: deciding what to measure.

Conrad Harrison, University of Oxford, Oxford, United Kingdom

Big data and computerized adaptive testing: everyone benefits when assessment systems are optimized around the patient. *

Christopher Sidey-Gibbons, PhD, UT MD Anderson Cancer Center, Houston, Texas, United States

Improving healthcare outcomes by combining PROs, the Electronic Health Record, and data science *

Christopher Sidey-Gibbons, PhD, UT MD Anderson Cancer Center, Houston, Texas, United States
Contributor: Christine Swisher, PhD , Project Ronin, San Diego, United States

Leveraging patient-reported outcome data to enhance predictive modeling using digital twins

André Pfob, MD, Breast Unit, University Hospital Heidelberg, Heidelberg, Germany

*Note: The two abstracts marked with * will be presented as one under the title “Beyond psychometrics – using data science to optimize modern measurement.” 

Moderator:

Dylan Trundell, Roche, London, United Kingdom

Speakers:

Michelle White, PhD, QualityMetric, United States
Stacie Hudgens, MA (AbD), Clinical Outcomes Solutions, United States
Cathi Ybarra, BSN, Bristol Myers Squibb, United States

Discussant:

Tara Symonds, Clinical Outcomes Solutions, United States

Contributor:

Ashley Slagle, Aspen Consulting, LLC, United States

Daily diaries are a useful method for collecting patient-reported outcome endpoints that fluctuate greatly from day to day. This symposium aimed to discuss the challenges of daily diary data collection in terms of the following: thoughtful construction that is user-friendly, operationalization in terms of administration, compliance monitoring, psychometric considerations for endpoint derivation and interpretation, and communication of outcomes. Panelists presented on the development of daily diary measures for derivation of endpoints in clinical trials, the operationalization of daily diary PRO data collection in industry-sponsored clinical trials, psychometric methodologies currently in the literature, ongoing work streams for scoring, deriving, and interpreting daily diary PROs, and offered perspective on the regulatory implications and review process for considering daily diaries as fit-for-purpose as endpoints in clinical trials for medical product development. A discussant offered summations and commented on the implications for future research.

Moderator:

Donald Bushnell, Evidera | PPD, United States

Speakers:

André De Champlain, PhD, Astra Zeneca, United States
Pip Griffiths, PhD, IQVIA, France
Myriam Blanchin, PhD, Nantes Université, France
Jinxiang Hu, PhD, University of Kansas Medical Center, United States

The advent of daily diary data as well as wearable device/sensor-based digital measures, has contributed vast amounts of heretofore unavailable information. However, a gap may still exist between these intensive longitudinal data (ILD) and our analytic frameworks. For example, is a simple mean value of a multitude of data points the best way to understand and interpret longitudinal data or can methods proposed for ILD be adapted to assess the psychometric properties of our measures in a way that may be less restrictive than what traditional frameworks can afford? In our symposium, the strengths of various novel ILD models were discussed and compared to traditional methods, including the time-varying effects and continuous Rasch models. The main focus of our symposium was to illustrate how these modelling techniques can be used to assess the measurement properties of ILD in a manner that fully exploits the richness of such data.

Moderator:

Antoine Regnault, PhD, Modus Outcomes, A Division of THREAD, France

Speakers:

Amélie Anota, Centre Léon Bérard, France
Libby Floden, Clinical Outcomes Solutions, United States
Francesco Cottone, PhD, Daiichi-Sankyo, Italy

Contributor:

Mallorie Fiero, US Food and Drug Administration, United States

The objective of this symposium organized by Statistics Special Interest Group (SIG) was to provide attendees with an overview of the current research related to the analysis of PRO data using time-to-event (TTE) methods and discuss the practical and methodological questions related to the application of these methods in clinical trials.

Four expert statisticians having hands-on experience with the TTE analysis of PRO data from clinical trials presented on several facets of the topic: analysis using complex event definitions (composite PRO events), application of the estimand framework to TTE PRO endpoints in cancer clinical trials, sensitivity analyses for TTE PRO endpoints, and TTE analysis accounting for events that can preempt the occurence of the PRO event (“competing event”).

The symposium was concluded by a general panel discussion about the challenges of applying TTE methods to PRO data, with a specific focus on outstanding questions and challenges.

Workshop level: Advanced

Workshop Goals:

The morning session introduced the challenges with the analysis of PRO data collected over time and equipped attendees with an understanding of the nature of missing data and key statistical models used for comparing PRO scores between randomized groups where the missing data is likely to be informative (related to the concept the PRO is measuring).

The afternoon session was then a practical session where participants could practice applying the models using a clinical trial dataset and one of three software packages. Workshop facilitators were present to support this hands-on analysis session.

Intended Audience

Statisticians or researchers involved in the statistical analysis of clinical trial PROs. A basic understanding of regression modeling was recommended for the morning and afternoon sessions. Prior knowledge of at least one statistical analysis software package (Stata, R, SAS) was recommended for the afternoon session. No prior knowledge of missing data handling was required. Participants brought their own laptop with statistical software installed.

Overview/Outline

The analysis of PRO data collected over time is not currently standardized. There are many models that could be used. However, to ensure the assumptions are met and an appropriate model selected, those making the selection require a good understanding of missing data on both its nature and impact. In this workshop, we first described what is considered missing data, how it can be mitigated prior to analysis, and discussed the role of intercurrent events. Next, participants gained knowledge of the types of missing data and how to explore these data. Recent research on the relationship between missing data rates and bias could help participants understand ‘how much is too much’. In the final morning session, a suite of models were then taught that could be applied to the analysis of longitudinal PRO data from a randomized clinical trial setting. The afternoon practical session allowed participants to work at their own pace, using a software package they were familiar with, analyzing a dataset with support from the facilitators.

Learning Objectives

1. To understand what constitutes missing data, potential impact and how to explore the missing data. The theoretical background to the field of missing data was presented, to help participants appreciate the importance of considering this and how to mitigate missing data where possible. The distinction between missing data and intercurrent events was also clarified. The different mechanisms and patterns of missing data were introduced, covering practical approaches for exploring and visualizing the missing data prior to fitting analytical models. How the missing data rate impacts bias and efficiency and showed that, under certain conditions, researchers can make valid between-arm comparisons in randomized clinical trials with high missing data rates. Workshop participants learned why they should document reasons for missing data, use all available data, and conduct analyses that make less restrictive assumptions about the missing data whenever possible.
2. To familiarize participants with the most commonly applied longitudinal models incorporating missing data assumptions. The final morning session presented a suite of longitudinal models for missing data. The mixed model for repeated measures, which assumes data is missing at random, was covered with the following extensions: multiple imputation, joint models and pattern-mixture models. A focus of the presentation was estimating the effect of treatment in a randomized clinical trial and equipping participants with the necessary tools for the afternoon session.
3. To gain practical experience of exploring missing data and fitting analytic models to randomized clinical trial data. Participants applied the knowledge gained in the morning sessions to data from a clinical trial administering PRO data over time. The workshop facilitators supported participants in conducting an analysis with either SAS, R or Stata software, depending on each participant’s preference and access to software. This session allowed participants to gain a greater appreciation of the decisions to be made in practice, and any problems that can arise during analysis. The practical session finished with a summary and discussion of the challenges found by participants and limitations of the models. The final session briefly highlighted other analysis strategies not covered in the workshop with a focus on future directions in the field.

Organizers:

Kim Cocks, PhD, Adelphi Values, Cheshire, United Kingdom

Jan R. Boehnke, PhD, University of Dundee, Dundee, United Kingdom

Bellinda King-Kallimanis, PhD, LUNGevity Foundation, Annapolis, Maryland, United States

Gina Mazza, PhD, Mayo Clinic, Scottsdale, Arizona, United States

Andrew Trigg, MSc, Bayer, Reading, United Kingdom

Workshop level: Advanced

Workshop Goals:

The primary goals of the workshops were to improve understanding of the linkage between PRO objectives, analysis and interpretation. In the first workshop, the emphasis was on learning through application how different analytic choices connect to different interpretations. The second workshop, emphasized the linkages through various analytic options when applying the estimand framework to PROs.

1. For PROs in clinical trials:
a.      Understand the importance of connecting design, analysis and interpretation
b.      Understand the implication of different analytic choices
c.      Illustrate the importance of the estimand framework
2. Provide clear demonstration of PRO estimands that can be applied in the regulatory setting

Intended Audience

The workshop was intended for academics, trialists and outcomes researchers who design or analyze patient-reported endpoints from clinical trials or other research projects.

Overview/Outline

Workshop 1

1. Background
   1. State of the field: current heterogeneity in PRO analyses
   2. PRO objectives: stating the obvious! (or: the importance of asking the right questions)
   3. Constructing a better PRO objective: critical elements and their impact.
2. Scenario 1: Substance Abuse Trial
   1. Main challenges to PRO data
   2. Sample analyses, interpretations and comparison
3. Scenario 2: Oncology Trial
   1. Main challenges to PRO data
   2. Sample analyses, interpretations and comparison
4. Breakout groups + hands-on application
   1. Discussion of main challenges
   2. Sample analyses and interpretations
5. Final remarks
   1. The goal was to translate a PRO objective into analysis and interpretation, and to understand the implications of analytic choices

Workshop 2

1. Background
   1. PRO objectives are often vaguely defined, leading to interpretation challenges and exclusion of results from the drug label
   2. The estimand framework provides a clear process for defining and interpreting PRO endpoints
2. Definitions of estimands, per ICH E9 addendum
   1. 5 components (with emphasis on intercurrent events in PRO setting)
3. Sensitivity and supportive analyses for PRO endpoints
4. Demonstration of estimands on PRO endpoints – Substance Abuse Trial Example
5. Demonstration of estimands on PRO endpoints – Cancer Trial Example
6. Breakout groups + hands-on application
   1. Identifying intercurrent events
   2. Proposed strategies
   3. Sensitivity analyses
   4. Interpretation and communication
7. Final remarks
   1. The goal was to demonstrate estimands by linking PRO objectives to analysis and interpretation

Learning Objectives

Workshop 1

1. Back to basics – what questions are we trying to address in clinical trials
2. Understand how different PRO analyses of the same data may lead to different results and/or different conclusions
3. Constructing a PRO objectives: what to consider
4. Given a PRO objective, define appropriate analytical approaches
5. Given a PRO objective, define a corresponding sensitivity analysis
6. Given a PRO objective, successfully communicate the interpretation of the analysis

Workshop 2

1. Understand how using the International Council for Harmonisation (ICH) addendum on estimands and sensitivity analysis in clinical trials provides a framework connecting the PRO objective, with the analytic approach, and interpretation
2. Describe the 5 components of estimands for PROs and how these relate to a PRO objective
3. Evaluate the strengths and limitations of different estimand strategies for trials
   1. Choice of population
   2. choice of summary statistic
   3. choice of intercurrent event strategy
4. Given a PRO objective, design 3 evaluable estimands for a clinical trial
5. Given a PRO estimand, define a corresponding sensitivity analysis
6. Given a PRO estimand, successfully communicate the interpretation

Organizers:

Libby Floden, PhD MPH, Clinical Outcomes Solutions, Chicago, Illinois, United States

Corneel Coens, PhD, EORTC, Antwerp, Belgium

Rachael Lawrance, PhD, Adelphi Values, Bollington, United Kingdom

Contributor:

• Jessica Roydhouse, PhD, University of Tasmania, Hobart, Australia

Workshop level: Advanced

Workshop Goals:

• Provide an introduction to common IRT models with an emphasis on polytomous models that are particularly well-suited to numerical rating scales (NRS) frequently used in PROs.
• Outline practical applications of IRT with PRO measures, including item calibrations and scaling, equating, differential item functioning as well as computerized adaptive testing.
• Allow participants to gain practical experience by completing several focused exercises to address common PRO IRT problems.

Intended Audience

Measurement scientists and practitioners who would like to gain a better understanding of IRT and its potential in supporting common PRO activities that are difficult, if not impossible, to address via classical test theory (CTT).

Overview/Outline

The widespread use of item response theory (IRT) models over the past decades attests to their importance in the development and analysis of PROs and items, both in clinical trial as well as real world settings. Unlike CTT, IRT yields item parameter estimates (e.g., “difficulty”) that are invariant across patient cohorts and ability estimates that are invariant across item sets/PRO forms measuring the same domain. These item and ability parameter invariance features of IRT make them particularly appealing for use in item analysis, form assembly, computerized adaptive testing (CAT) and differential item functioning, to name only a few.  Fundamentally, IRT provides us with an indication of whether the items selected for our PRO form are well targeted, in terms of overall difficulty or stringency level, to the patient’s estimated standing on the construct of interest (e.g. physical functioning). The latter information is central to the use of CAT as well as to assess whether floor or ceiling effects are present with our PRO scale.

In addition to going over common dichotomous and polytomous IRT models, this workshop allowed attendees to complete several practical exercises to gain a higher level of understanding of IRT and its potential to uniquely address a host of measurement issues.

Learning Objectives

1. Provide an introduction to common IRT models with an emphasis on polytomous models that are particularly well-suited to numerical rating scales (NRS) frequently used in PROs.
2. Outline practical applications of IRT with PRO measures, including item calibrations and scaling, equating, differential item functioning as well as computerized adaptive testing.
3. Allow participants to gain practical experience by completing several focused exercises to address common PRO IRT problems.

Organizers:

André De Champlain, PhD, AstraZeneca, Gaithersburg, Maryland, United States

Workshop level: Advanced

Workshop Goals:

Our goal was to introduce data-driven methods to detect differential item functioning (DIF) in patient-reported outcomes (PROs). The workshop focused on item-response theory (IRT) models for DIF detection. Extensions of conventional IRT models, which are based on unsupervised machine-learning methods, were used to explore heterogeneity in PRO responses. The workshop also briefly explored the use of unsupervised methods to examine response shift in longitudinal data.

Workshop participants benefited from new insights about the potential role of machine-learning methods as a component of the toolkit for PRO data analyses. Unsupervised machine-learning methods are advantageous for assessing variable importance in PROMs data that contain large numbers of variables (i.e., features) to characterize patients, or for assessing the sensitivity of individual items in large items banks to DIF.

Intended Audience

This advanced workshop was intended for researchers and clinicians seeking practical guidance on machine-learning methods to assess measurement validity of PROs. Participants were familiar with the theory of item response theory models and their applications. They were also familiar with the development and interpretation of multiple regression models, and had a basic level of proficiency with the R software package.

Overview/Outline

Part 1: Overview of supervised and unsupervised machine-learning methods (see attachment): This included least absolute shrinkage and selection operator (LASSO) and elastic net regression models, random forest classification models, recursive partitioning techniques, and latent class models. Examples of applications for cross-sectional and longitudinal PRO data. Advantages, limitations, and challenges to implementation.

Part 2: Item response theory models for DIF detection: theory and application; introduction of a practical example.

Part 3: Latent variable machine-learning methods (see attachment). Combining item response theory models and unsupervised machine-learning methods; illustration using the practical example from part 2.

Part 4: Extensions of latent variable machine-learning methods to detect response shift.

Learning Objectives

1. To examine methods to investigate differential item functioning (DIF) in high-dimensional data (i.e., many PRO scale items or many patient-level variables) and demonstrate the application of machine-learning models for DIF exploration and detection.
2. To describe the types of data and research problems that will benefit from the application of machine-learning models for DIF detection.
3. To practice the implementation of machine-learning methods using existing software packages, with a particular emphasis on R software.

Organizers:

Lisa Lix, University of Manitoba, Winnipeg, Manitoba, Canada

Tolulope Sajobi, PhD, Department of Community Health Sciences, University of Calgary, Calgary, Alberta, Canada

Yuelin Li, PhD, Department of Psychiatry & Behavioral Sciences & Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, New York, United States

Workshop level: Basic

Workshop Goals:

The aim of the workshop was to provide participants with knowledge, tools, and resources to optimize the development of successful learning activities for PROMs adoption. We supported participants in the planning, development, and delivery of effective PROMs training and education activities for health care professionals. To facilitate healthcare provider (HCP) incorporation of PROMs into routine clinical care, several questions need to be answered: what do clinicians need to use PROMs and interpret scores? what factors influence clinicians to incorporate and respond to PROM score data/feedback into patient care? what strategies could be used to assist clinicians? how do the needs of clinicians vary? In addition to exploring these questions, participants were given the opportunity to proactively learn about tools and resources available and design a training program.

Intended Audience

An intermediate level of knowledge about PROM in clinical practice and the implementation process.

Overview/Outline

Healthcare providers (HCPs) are often the lynchpin to the successful implementation of PROMs to engage patients in their care. Although the use of PROMs in practice in practice becoming more prevalent the training for providers is not standardized. Facilitating HCP adoption of PROMs is a complex behavior change, often necessitating flexible and tailored strategies to support training, education, and ongoing implementation support. This workshop drew on implementation science theory and practice to explore the multifaceted aspects of HCP behavior change associated with PROM adoption. The workshop presented theory-informed approaches to HCP training programs, continuing education and implementation support. The theory was complemented with real world case studies of HCP learning activities in diverse clinical settings. We concluded by identifying future opportunities for inclusion of PROMs in curricula to prepare the next generation of HCPs. This interactive workshop provided participants with the opportunity to apply the theory to their own context by “designing training with the end in mind.” Using this approach to design education addresses the needs of HPCS and specific barriers to PROMs implementation.

Learning Objectives

1. Apply implementation science theory to characterize the requisite knowledge, skills, and clinical reasoning capabilities of HCPs for both initial and ongoing learning to support PROM use in clinical care.
2. Utilize real-world examples of training / learning across a variety of disease types, patients, and clinician groups.
3. Design training programs and continuing education that focus on the learning needs, capabilities, and motivations of HCPs.

Organizers:

Angela Wolff, PhD RN, Associate Professor, School of Nursing, Trinity Western University, Langley, British Columbia, Canada

Kate Absolom, PhD, Leeds Institute of Medical Research/Leeds Institute of Health Services Research, University of Leeds, Leeds, United Kingdom

Elizabeth Austin, MPH, Research Scientist, Department of Health Systems and Population Health, School of Public Health, University of Washington, Seattle, Washington, United States

Maria Jose Santana, PhD MPharm, Associate Professor, Patient and Family-centred care Scientist, Departments of Paediatrics and Community Health Sciences, Cumming School of Medicine, Calgary, Alberta, Canada

Angela M. Stover, PhD, Department of Health Policy and Management and Lineberger Comprehensive Cancer Center, UNC-Chapel Hill, Chapel Hill, North Carolina, United States

Sara Ahmed, PhD PT, McGill University, Faculty of Medicine and Health Sciences, McGill University Health Center, Clinical Epidemiology, Montreal, Quebec, Canada

Susan Bartlett, PhD, McGill Medicine, Johns Hopkins Medicine, Montreal, Québec, Canada

Workshop level: Basic

Workshop Goals:

With the rise of digital studies, there is a need to support best practices for paper-to-eCOA migration with consideration to digital user experience (UX) and interface (UI).

This workshop discussed challenges associated with paper COA migration for use in digital real-world studies. The presenters provided an overview of common practice, the different levels of migration, and the evidence needed to support measurement equivalence.

The goal was to provide an overview of current gold-standard eCOA migration guidelines, demonstrating where challenges outside of current guidelines may be encountered and exploring potential solutions.

Intended Audience

Individuals interested in developing and/or using PROMs in digital healthcare research; applicable to all therapeutic areas/learners of all levels with prior experience of PROMs.

Overview/Outline

1. Introduction and objectives
2. Background on eCOA migration
3. Demonstration of challenge, together with activity polls

Different real-life challenges in paper-to-eCOA migration will be presented together with potential solutions.

Challenge 1: Maintaining validity
An example was presented using a COA where permission from the instrument developer was sought to shorten introductory content before each item, to improve readability on electronic devices. We explored the next steps and possible solutions to maximize UX without compromising on the measurement science.

Challenge 2: User experience
This challenge focused on unique issues for eCOA migration in specific responder groups, namely children and proxies. We explored solutions to approaching eCOA migration for child/proxy completion.

Challenge 3: User interface
We focused on the UI of varying types of electronic devices with examples of PROMs which may require content reformatting. The discussion explored solutions to formatting and structural changes, and ways forward when these are not agreed by the instrument developer.

Challenge 4: Non-standard format
Discussing some non-standard format PROMs (e.g., those with grid-type response options), we demonstrated the challenges in digital migration and discussed the best methods of approach.

4. Guided group discussion

This covered additional challenges identified by workshop participants, including discussion of potential solutions.

5. Q&A

The workshop concluded with a Q&A session and resources for further learning on UX/UI best practices.

Learning Objectives

1. To describe the processes for paper-to-electronic COA (eCOA) migration and identify when this must diverge for PRO measures (PROMs) with a non-standard structure.
2. To understand the balance between maintaining validity and user experience of the measure.
3. Given a PROM, to identify the most appropriate methods in approaching the eCOA migration and finding solutions which can be applied when challenges are raised that fall outside of existing guidelines.

Organizers:

Anna Richards, MA, Vitaccess, Oxford, United Kingdom

Catherine Bottomley, MPharm PhD, Vitaccess, Oxford, United Kingdom

Mark Larkin, PhD, Vitaccess, Oxford, United Kingdom

Workshop level: Basic

Workshop Goals:

This hands-on workshop provided a tutorial of how to build a clinical dashboard for tracking PROs and response rates. Dashboards provide study teams with a centralized, on-demand hub for visualizing key study metrics, replacing the need for ad-hoc reporting. We introduced methods for quickly developing a dashboard using free-to-use, open-source programming environments (R/RStudio). R is a statistical and graphing-centered programming language which provides the dashboard structure, allowing integration from multiple data sources, data manipulation and analysis, and construction of visual pieces needed for communicating data to stakeholders. Participants learned to weave together these elements, allowing for the creation of simple, elegant dashboards to track PROs and other clinical data. Participants then learned how to deploy and share the dashboard with team members.

Intended Audience

Beginner to intermediate. This workshop did not require any prior knowledge of statistical programming, but familiarity with R or general programming principles was helpful. Intended audience members include clinicians, research professionals, or statisticians who needed/wanted to learn how to visualize data including clinical trials, retrospective databases, and PROs. Participants were required to bring their own personal laptop.

Overview/Outline

1. Explore examples and motivations for creating PRO dashboards.
2. Introduce the tools for creating dashboards.
3. Walk members through downloading R/Rstudio.
4. Teach participants how to navigate the Rstudio interface.
5. Help participants master basic R commands, including reading in data and building basic tables/visual aids.
6. Provide attendees with an overview of Rmarkdown, Shiny and Flexdashboard and their applications to clinical dashboards.
7. Present the steps to incrementally build a dashboard for tracking PROs.
8. Review methods for deploying dashboards and resources for finding assistance.
9. Demonstrate complex examples of dashboards.

Discuss applications and future direction for dashboards.

Learning Objectives

1. Participants became familiar with the underlying programming (R/RStudio) techniques necessary to create a clinical dashboard. They were given the tools to be able to load data from various sources, build visual elements, and deploy a dashboard.
2. Participants became familiar with the necessary free library packages that are utilized to build dashboard structure and interactivity. With these elements and our support, participants combined these tools to create a simple dashboard to track quality-of-life protocol adherence from a blank canvas to a finished product.
3. Participants were immersed into current real-world dashboard applications involving clinical trials, prospective registries, and retrospective dashboards. This objective provided the participant with possible future directions and applications involving artificial intelligence, protocol lifetime management, and clinical patient tracking.

Organizers:

Michael Golafshar, MS, Mayo Clinic, Scottsdale, Arizona, United States

Molly Voss, Mayo Clinic, Scottsdale, Arizona, United States

Todd DeWees, PhD, Mayo Clinic, Scottsdale, Arizona, United States

Workshop level: Basic

Workshop Goals:

To provide participants with a comprehensive understanding of qualitative interview methods that can be used in clinical trials as an additional means of generating patient experience data. Using case studies from industry, key operational and methodological challenges and considerations when designing and implementing qualitative in-trial or exit interview research were outlined and options for overcoming those challenges were provided. The workshop highlighted methods to explore disease and treatment experience, trial participation, and support validation and aid validation and interpretation of clinical outcome assessment (COA) endpoints.

Intended Audience

Outcomes researchers, instrument developers or industry attendees involved in the design and implementation of health outcomes research. No prior knowledge of qualitative interview methods was assumed.

Overview/Outline

The value of collecting patient experience data from clinical trial participants to generate in-depth insights beyond traditional COA endpoints is increasingly recognized. Qualitative in-trial or exit interviews is still a relatively new and emerging method and the potential value is still being realized, however it is progressively utilized to capture patient and other stakeholder perspectives. The US Food and Drug Administration (FDA) acknowledge the method for the first time in their Patient-Focused Drug Development (PFDD) guidance as an approach by which to collect evidence on a range of topics, particularly in rare diseases where evidence is sparse and it may not be feasible to perform standalone non-interventional studies. The presenters provided an overview of the topics which in-trial/exit interviews can be used to explore, noting key considerations/challenges for each. The operational challenges of incorporating exit interviews into trials are considerable. Guidance was given for how these can be best anticipated, avoided and/or overcome, with real life examples. The workshop provided examples of best practice methods for collecting and analyzing in-trial/exit interview data, including approaches that can be taken to link qualitative findings to data collected from COA endpoints during the trial.

Learning Objectives

1. To understand qualitative in-trial or exit interview methods and how they can be used to elicit patient experience data. Participants gained an understanding of what in-trial or exit interviews are in the context of clinical trials and why it is a valuable approach for generating supplementary patient experience data. Participants were introduced to different qualitative interview methods that can be used depending on the research objectives and clinical trial phase. Participants learned about the various methods available to qualitatively explore the symptoms and health-related quality of life (HRQoL) impacts experienced by patients and changes following treatment, how to generate evidence of content validity of COAs in this setting and to qualitatively explore meaningful change to aid interpretation of thresholds. Participants applied their knowledge to develop interview guide questions to address different topics/research needs and role-played the conduct of an interview with another workshop participant.
2. To obtain an understanding of common operational challenges and considerations for implementing interviews in trials and ways of overcoming them. Participants developed an awareness of key considerations and challenges associated with implementing in-trial or exit interview studies. Topics covered included considerations for implementation in/outside a clinical trial program and the impact of this on study set-up, ethics, compliance, and timelines. With increasing complexity and scale of multi-national trials, common challenges in the design of in-trial interview studies include selecting an appropriate sample size across countries to be representative of the wider trial population and minimizing burden on site staff and patients involved in trials. Participants were provided with real life scenarios/challenges that have arisen in studies and asked to suggest potential solutions through discussion. Considerations for data handling were also discussed including safety reporting, linking data, and analyzing blinded or unblinded data.
3. To illustrate practical examples of conducting in-trial/exit interviews and the value of data generated. Real-world examples illustrated how qualitative interview methods can be applied in clinical trials and how data generated can be utilized. Participants gained hands-on experience of coding an example interview transcript and discussing the themes. Participants learned about the value of patient experience data in early phase trials where limited data are available regarding the benefit-risk of treatments, which presents an opportunity to improve understanding of the disease experience and inform future study designs and the design, refinement and/or interpretation of COAs for use in future trials. The utility of data generated in later phase trials to provide supplementary evidence to support treatment evaluation for regulatory submissions and health technology assessment decision-making was also outlined. Additionally, exploring the perspectives of trial staff/investigators to improve patient-clinician discussions and promote individualized treatment decisions.

Organizers:

Nicola Williamson, Adelphi Values Ltd, Bollington, United Kingdom

Jane Wells, Sanofi, London, United Kingdom

Nicola Bonner, Adelphi Values Ltd, Bollington, United Kingdom

Workshop level: Basic

Workshop Goals:

The overall goal of the workshop was to equip participants with information about progress in validity testing theory and with skills to put the theory into practice. Specific goals were to:

1. Describe the theoretical framework of the five sources of validity evidence from the 2014 Standards for Educational and Psychological Testing in relation to PROs
2. Contrast the notion of validation as a property of an instrument with the validation practice of accumulating evidence for interpretation and use of data in context for an intended purpose
3. Demonstrate when and why evidence based on each of the five sources would be included in a validation plan, including framing research questions and identifying qualitative and quantitative methods to investigate each source
4. Outline how to report validity evidence based on each of the five sources

Intended Audience

Researchers, clinicians, policy makers, and regulatory agents who develop self-report questionnaires, test the validity of self-report data in health contexts, make decisions about inclusion of PRO instruments in surveys, and decide on the appropriateness of promotional and labelling claims based on PROs.

Overview/Outline

Short presentations provided background to validity testing theory and addressed the imperative to advance validation practice in PRO measurement. Descriptions of the five sources of validity evidence were provided with examples of validity research questions and evidence generation methods, and with an emphasis on minimising measurement biases and ensuring accurate, useful (actionable), and equitable PRO measurements in diverse populations. Workshop presenters and participants planned the development of a hypothetical PRO and prepared a validation plan, including research questions, using the five sources of evidence framework. Small groups investigated qualitative and quantitative methods relevant to the research question/s for each source of evidence. Case studies of existing instruments provided examples. Finally, a plenary group discussion focused on how to report evidence based on each of the five sources, and considered how evidence synthesis might inform arguments about valid interpretation and use of PRO data in context for an intended measurement purpose.

Learning Objectives

1. To describe each of the Standard’s five sources of validity evidence and formulate a validation plan to answer research questions related to the kinds of evidence needed to interpret PRO data in context and for a measurement purpose.
2. To choose appropriate qualitative and quantitative methods to generate evidence based on each source of evidence to ensure accurate, useful (actionable), and equitable PRO measurements.
3. To craft a reporting format for presenting validity evidence based on each of the five sources.

Organizers:

Melanie Hawkins, PhD, Centre for Global Health and Equity, Swinburne University of Technology, Melbourne, Australia

Richard H Osborne, PhD, Centre for Global Health and Equity, Swinburne University of Technology, Melbourne, Australia

Sandra Nolte, Charité – Universitätsmedizin Berlin, Berlin, Germany

Nancy Mayo, PhD, McGill University, Montréal, Quebec, Canada

Kara Schick-Makaroff, PhD RN, University of Alberta, Edmonton, Alberta, Canada

Contributors:

• Christina Cheng, PhD, Centre for Global Health and Equity, Swinburne University of Technology, Melbourne, Australia;
• Gerald Elsworth, PhD, Centre for Global Health and Equity, Swinburne University of Technology, Melbourne, Australia
• Kevin Weinfurt, PhD, Duke University School of Medicine, Durham, North Carolina, United States
• Mirjam Sprangers, PhD, Amsterdam University Medical Centers, Amsterdam, Netherlands
• Richard Sawatzky, PhD, Trinity Western University, Langley, British Columbia, Canada

Organized by the New Investigator SIG

Speakers
Devin Peipert, PhD, Caroline Potter, DPhil, and Jill Carlton, PhD

Contributors
Ron D. Hays PhD, Brittany Lapin, PhD, and Jessica K. Roydhouse, PhD

Improving the Quality of Your Peer Reviews of Patient-Reported Outcomes Papers

Peer review is an essential aspect of the scientific process that improves the quality of the published work. Participating as a peer reviewer is an expected part of professional service for those active in research. Despite its importance, education and training about peer review are limited. This session drew on the expertise of editors of ISOQOL journals to provide tips and suggestions to help researchers be effective peer reviewers. It included a discussion about how skills needed for peer review are useful in writing better papers.

Prague Boats (River Cruise)

The Prague Boats cruise along the Vltava River offered the unique opportunity to visualize all of the city’s best architecture from the water. Attendees expected a unique and unforgettable experience while soaking up undisturbed views of Prague’s monuments from the boat’s deck or through the windows of the covered area below. The cruise began at the Čech Bridge and completed a three-hour loop along the river. Iconic landmarks such as Prague Castle, Charles Bridge, the Dancing House and National Theater were illuminated for a picturesque view at night.

The Experience Prague Social Event at the 29th Annual Conference was a unique and fun way for attendees to socialize with colleagues and familiarize themselves with landmarks of the city. The ticket price included heavy appetizers, an open bar, musical entertainment, and dancing.

Introduction to Quality of Life and Patient Reported Outcomes (IPRO)

Theory, Measurement, and Applications

For those looking to gain a better understanding of patient-reported outcomes (PROs), ISOQOL offered an introductory education course as part of the ISOQOL Annual Conference.

Learn more about the course here. 

Come meet with us to discuss the challenges and solutions related to strategic and operational management of PROMs and other COAs, as well as the keys to increase the visibility of your COAs and improve the COA user experience (and satisfaction!) – all to ensure the increased use of COAs worldwide.

*This was an Ancillary Event hosted by Mapi Research Trust and was not an ISOQOL event. Pre-registration was required.