Description:

The Cutting Edge Research plenary session featured some of the highest-ranked, innovative research from ISOQOL abstract submissions. In particular, these abstracts reflected research that truly “pushes the ISOQOL envelope” in providing new and different ways to look at quality of life.

Speakers:

Galina Velikova, MD PhD FRCP, Leeds Institute of Medical Research at St James’s, University of Leeds, Leeds Cancer Centre, Leeds, United Kingdom

Phase III randomised controlled trial of eRAPID (electronic patient self-Reporting of Adverse-events: Patient Information and advice)—An eHealth intervention during chemotherapy

Professor Galina Velikova is an academic Medical Oncologist at the University of Leeds and Leeds Teaching Hospitals, UK with over 20 years’ track record of successful patient-centred research using electronic Patient-Reported Outcome Measures in daily practice and clinical trials. She led a National Institute for Health Research (NIHR) 5-year Programme Grant for applied research on patient self-reported symptoms and toxicity, using an online reporting system (QTool), uniquely integrated in electronic records, along re-designed care pathways for remote monitoring during cancer treatment to improve patient safety. Further research projects evaluate the use of the electronic system for remote monitoring and follow-up of cancer patients after treatment.

Professor Velikova has experience in leading collaborative research, both nationally and internationally such as Quality of Life sub-studies of international breast cancer trials (TACT2, SUPREMO). She is the past Chair of the British Psychosocial Oncology Society; past Chair of the European Organisation for Research and Treatment of Cancer (EORTC) Quality of Life Group; and past President of the International Society for Quality of Life Research (ISOQOL). Professor Velikova is currently the Co-Chair of the National Cancer Research Institute (NCRI) Living with and Beyond Cancer Group and is also a member of the board of the EORTC.

Professor Velikova is a practising oncologist (Consultant in Medical Oncology) with clinical work focused on systemic treatment of breast cancer patients with early and advanced disease.

Antoine Regnault, PhD, Partner & Global Lead – Statistics Modus Outcomes, Lyon, France

Providing Meaningful Interpretation of a Performance Outcome Measure by Co-Calibration with a Patient-Reported Outcome: Illustration with Widely Used Multiple Sclerosis Measures

Antoine is a biostatistician with a strong interest in measurement science applied to patient-centered outcomes. As a consultant, he uses his combined skillset in psychometrics and statistics to support pharmaceutical companies with their analyses of patient-centered outcomes data from clinical trials and observational studies in a wide variety of disease areas (oncology, hematology, neurology and rare diseases, among others). He currently is a member of the Board of Directors of the International Society of Quality of Life (ISOQOL) and has been a member of international research initiatives such as SPIRIT-PRO and NeuroMET-2.

Sally Lewis, MRCGP, Aneurin Bevan University Health Board and Welsh Government, Cardiff, United Kingdom

Value Based Healthcare – a Welsh (UK) approach to delivering what matters to patients and clinicians

Sally has front-line experience of primary care at its most challenging having been senior partner and GP trainer in a practice in the Welsh valleys. She entered a career in medical management in 2011 and was appointed to Assistant Medical Director for value-based care in the Aneurin Bevan University Health Board in 2014.

Still practising as a GP, Sally is now National Clinical Lead for Value-based and Prudent Healthcare in Wales, and Honorary Professor at Swansea School of Medicine. Her current interests include the utilisation of value-based principles to allocate resource in publicly funded systems, patient outcome data and digital transformation

Tolulope “Tolu” Sajobi, PhD, University of Calgary, Calgary, Alberta, Canada

A Meta-analysis of Response Shift Effects in Patient-Reported Outcomes Studies

Dr. Sajobi is Associate Professor of Biostatistics in the Department of Community Health Sciences at the University of Calgary, Canada. He has a long standing interest in statistical methods for measurement and analysis patient-reported outcomes in clinical trials and observational studies. He is currently exploring novel latent variable models and machine learning algorithms for response shift detection in longitudinal patient-reported outcomes studies.

Moderator:

Todd DeWees, PhD

Description:

A plenary session focused on patient engagement and led by patient advocates. The goal of this plenary was to stimulate discussion regarding theory, methods and application of optimal patient and advocate involvement.

Speakers:

Allyson Berent, DVM DACVIM, Foundation for Angelman Syndrome Therapeutics and GeneTx Biotherapeutics, New York, New York, United States

Dr. Allyson Berent graduated from Cornell University College of Veterinary Medicine in 2002 and completed a residency in veterinary internal medicine at the University of Pennsylvania in 2005, followed by a fellowship in interventional radiology and endourology at the University of Pennsylvania and Thomas Jefferson University. Since 2009, she serves as the director of the Interventional Endoscopy Service at the Animal Medical Center in NYC. Dr. Berent is the CSO of the Foundation for Angelman Syndrome Therapeutics and the Chief Operating Officer for GeneTx Biotherapeutics, developing a novel antisense olignonucleotide for the treatment of Angelman syndrome, as rare neurogenetic disorder.

Kemi Doll, MD MSCR, University of Washington School of Medicine; Co-founder, Endometrial Cancer Action Network for African-Americans (ECANA), Seattle, Washington, United States

Dr. Kemi Doll is a gynecologic oncologist and Assistant Professor in the Department of OBGYN and an Adjunct Assistant Professor in the Department of Health Services in the School of Public Health. Her research centers on examining Black-White racial inequity in the care of benign and malignant gynecologic disease in the US. Her research has been funded by the NIH, the Robert Wood Johnson Foundation, and the Patient Centered Outcomes Research Institute (PCORI). She is the co-founder of ECANA, the Endometrial Cancer Action Network for African-Americans (ecanawomen.org) a community-research partnership formed to improve survival among Black women with endometrial cancer.

Moderator:

Elizabeth Unni, PhD MBA Bpharm

Description:

Innovations in healthcare technology are putting increasing pressure on constrained healthcare budgets, and this has led to an emergence of outcomes-based pricing models. The goal of this plenary was to discuss how clinical outcome assessment data are being incorporated into these access systems as well as the benefits and limitations of this model.

Speakers:

Lotte Steuten, PhD, Office of Health Economics, London, United Kingdom

Lotte Steuten, PhD, is Vice-President and Head of Consulting at the Office of Health Economics and Honorary Visiting Professor at City, University of London, UK. Lotte specialises in providing research-based and client-focused analysis and expertise to informing decisions regarding healthcare policy, pricing and regulation of pharmaceuticals, and health technology assessment (HTA). She effectively collaborates with technology assessors, payers and policy makers, (academic) researchers, clinical and patient representatives, pharmaceutical industry and capital investors. Lotte has contributed to the health economics and outcomes research for >15 years in various academic roles and executive functions in the UK, US and the Netherlands.

Paula Williamson, PhD FRSS FMedSci, COMET Initiative Management Group, University of Liverpool, Liverpool, United Kingdom

Paula Williamson is Professor of Biostatistics at the University of Liverpool, and was Head of the Department of Biostatistics from 2002 until 2018. Her research has focussed on clinical trials, meta-analysis, and health outcome selection. Paula leads the MRC/NIHR Trials Methodology Research Partnership. She is an NIHR Senior Investigator, a Fellow of the Academy of Medical Sciences, and gave the Bradford Hill Lecture in 2017. Paula co-founded the (Core Outcome Measures in Effectiveness Trials) Initiative in 2010, to improve the quality and relevance of health research to decision makers including patients, health professionals, regulators and policymakers.

Stefan Larsson, MD PhD, The Boston Consulting Group, Stockholm, Sweden

Stefan Larsson joined the Boston Consulting Group in 1996 and is a Senior Partner and a BHI Fellow. Stefan has lead strategy assignments and major transformational programs across all sectors of the healthcare industry: Academic medical centers; leading international providers; Biopharma; Biotech and MedTech – from R&D through commercial. He is a cofounder and board member of ICHOM. Stefan has been BCG‘s global sector leader for Healthcare Payers and Providers and later the Healthcare Systems sector. Stefan is an MD, PhD and Associate professor at the Karolinska Institute (KI). Stefan trained at Harvard MS, MRC-HGU, Edinburgh and EMBL Heidelberg.

Moderator:

Patricia Corey-Lisle, PhD MSN

Description:

The goal of this plenary session was to hear how technology is impacting our work. We heard from innovators working both inside and outside our field as they painted a picture for the future of outcomes measurement and quality of life research.

Speakers:

Marie Deserno, PhD, Max Planck Institute for Human Development, Berlin, Germany

Dr. Marie Deserno is a postdoctoral research fellow at the Formal Methods group of the Max Planck Institute for Human Development in Berlin, Germany. Supported by a NWO Rubicon fellowship her research deals with models of change in developmental psychology and quality of life research in the autistic community. Her work is strongly rooted in the network approach to psychology developed at the Psychosystems group (University of Amsterdam) and sits at the intersection of developmental psychology and psychological methods.

Katarzyna Wac, PhD, University of Geneva, Quality of Life Technologies Lab, Geneva, Switzerland

Katarzyna Wac is a Professor of Health Informatics at the Department of Computer Science, University of Copenhagen, Denmark, and at the University of Geneva, Switzerland, affiliated with Stanford University since 2013. Dr. Wac leads Quality of Life (QoL) Technologies lab researching how mobile and emerging sensor-based technologies can be leveraged for an accurate, longitudinal personalized assessment of the individual’s behavior and QoL, as they unfold naturally over time and in context, and the improvement of the latter. More info at katarzynawac.org.

Kim Oostrom, PhD, University of Amsterdam Medical Center, Amsterdam, Netherlands

Moderator:

Chris Sidey-Gibbons, PhD

Workshop level: Basic

Workshop Goals:

Although major advances allow individuals with formerly life-limiting or life-threatening illnesses to lead full lives, providing timely, responsive care for complex health conditions remains challenging and very costly to patients and society. A compelling example is chronic progressive organ injury leading to failure of vital organs (kidney, lung, heart, and liver) in children and adults. While solid organ transplants can extend life for decades, up to 50% experience ongoing physical symptoms, emotional distress, and social disruptions that impair HRQoL and result in ED visits/hospitalizations. We provided real world examples of how PROMs greatly impact communication, care, and HRQoL in complex chronic illness programs across the lifespan.

Intended Audience

This workshop offered a roadmap for patients, clinicians, administrators, and policy makers seeking to implement PROs for high stakes chronic illnesses. The unique psychosocial challenges of children, teens, and adults living with complex medical illnesses was highlighted.

Overview/Outline

Introduction (10 minutes)
Expert Presentations (80 min)
• PROMs are Changing Conversations and Transplant Care
• Multiple Needs, Multicultural Contexts: Ensuring PROMs are Accessible, Responsive, Interpretable, and Actionable to Broad Groups of Users
• Roadmap for Implementing PROs in Adults with Complex Health Conditions
• Measuring What Matters: an eHealth Intervention within a Pediatric Transplant Clinical Practice
Break (10 minutes)
Breakout Session (Activity/Debriefing 60 minutes)
Using a case-study approach, attendees worked in small groups to:
• Identify patient/family/clinician/stakeholder needs
• Review how stakeholder preferences, setting, IT, and psychometric factors impact PROM selection
• Identify potential barriers and strategies to facilitate uptake and sustainability
• Develop learning solutions for patients/families/clinicians/others to enhance value
Q & A (15 minutes)
Wrap-up (5 minutes)

Learning Objectives

1. Illustrate the potential impact of PROMs on communication and shared decision making in complex, high-stakes health conditions in children and adults using real world examples
2. Provide a roadmap for assessing the needs of patients, families, clinicians and other stakeholders when selecting a core set of PROMs across programs
3. Compare strategies to identify potential barriers, enablers, local champions and increase buy-in for implementation and sustain use of PROMs

Organizers:

Susan J. Bartlett, PhD, McGill University, Montreal, Quebec, Canada

Samantha Anthony, PhD, The Hospital for Sick Children, Toronto, Ontario, Canada

Istvan Mucsi, MD PhD, University of Toronto Health Network, Toronto, Ontario, Canada

John Devin Peipert, PhD, Northwestern University, Chicago, Illinois, United States

Workshop level: Basic

Workshop Goals:

The goals of the workshop were to discuss Rasch measurement theory and classical test theory and how they can complement one another when evaluating clinical outcome assessments. Specific objectives were to: (1) describe basic concepts of classical test theory and how it can be used to measure reliability and validity; (2) detail introductory concepts and how/when to use Rasch measurement theory to measure reliability and validity; (3) discuss similarities, differences, and limitations of Rasch measurement theory, classical test theory, and item response theory, along with how they can complement one another; (4) detail assumptions, model properties, fit parameters, and recommendations for Rasch analysis using Winsteps software; and (5) provide participants a hands-on experience using the Rasch rating scale model to analyze the validity and reliability of a sample clinical outcomes measure.

Intended Audience

Participants interested in learning modern psychometric techniques for measuring validity and reliability of clinical outcome assessments benefitted from this workshop. The level of the workshop was set as introductory.

Overview/Outline

The workshop outline was as follows: 1) Overview of clinical outcome assessments including patient reported/centered outcomes; 2) Classical test theory; 3) Rasch measurement theory; 4) How can Rasch, Classical test theory, and Item response theory complement one another?; 5) Rasch model specifications and recommendations; 6) Hands-on Rasch rating scale model analysis in Winsteps software.

Learning Objectives

1. Participants were able to discuss clinical outcomes assessments (COAs) including patient-centered outcomes, their components, item generation, biases, reliability and validity, scaling and responses as well as classical test theory. Specifically, discussion focused on qualitative and quantitative assessments of each these concepts and provided participants an overview of issues related to measurement. Emphasis was placed on basic concepts of classical test theory and how it can be used to measure reliability and validity. The presentation format for this objective was PowerPoint slides. Interaction was encouraged with Q&A.
2. Participants were introduced to theory and applications of Rasch measurement with specific emphasis on the assumptions of this model in COA development and validation. Introductory concepts and how/when to use Rasch measurement theory to measure reliability and validity were detailed. Similarities, differences, and limitations of Rasch measurement theory, classical test theory, and item response theory were discussed, along with how they can complement one another. Assessment was described in terms of person ability, scale reliability and validity, and item difficulty. Properties and requirements of Rasch models were explained in detail. For dichotomous and polytomous data, model fit parameters and recommendations were provided for analysis in WINSTEPS. Further, summaries of Rasch requirements, relevant WINSTEPS tables, and criteria-specific recommendations were provided. The presentation format was PowerPoint slides and handouts. Interaction was encouraged with Q&A, and input on interpretation of example figures and tables.
3. Participants gained hands-on experience with using a Rasch rating scale model and practice procedures to run the analysis in WINSTEPS, using the guidelines and recommendations learned during the session. Participants were asked to download the free version of WINSTEPS ahead of time. A sample dataset (for example, sample responses to a questionnaire with a 4-point Likert scale) of less than 75 observations and analysis code was provided. Instructors walked through the analysis and highlighted key procedures and result output table numbers, where the relevant coefficients and parameters can be found to effectively analyze the validity and reliability of the sample clinical outcomes measure. The presentation format of this learning objective was analysis demonstration in WINSTEPS software and PowerPoint slides to facilitate discussion of sample results and figures. Interaction was encouraged with instructors providing assistance with running analyses and group interpretation of findings.

Organizers:

Harman Dhatt, MPH PhD Candidate, Apex Health Solutions & University of Arizona College of Pharmacy, Scottsdale, Arizona, United States

Stacie Hudgens, MA (AbD), Clinical Outcomes Solutions, Tucson, Arizona, United States

Terri Warholak, PhD RPh CPHQ FAPhA, University of Arizona College of Pharmacy, Tucson, Arizona, United States

Workshop level: Basic

Workshop Goals:

There is a growing emphasis on understanding the patient perspective in health research, care, and policy. Patient-centered outcomes and health state utility evidence are one way to incorporate the patient perspective into healthcare decision making. Patient preference studies are starting to gain traction as an alternative or complimentary source of evidence on the patient perspective.

Stated-preference methods are increasingly used to engage patients and to inform decision making in variety of healthcare settings (e.g. clinical trial design, regulatory assessment, value assessment, shared decision making). Identification of the appropriate stated-preference method requires a thorough understanding of the research question and study objective. To ensure valid and reliable results, good research practices in the development, design, conduct, analysis, and interpretation of the methods need to be applied.

This workshop provided participants with a practical understanding of a variety of stated-preference methods including discrete-choice experiments (DCE), best-worst scaling (BWS), and rating/ranking techniques. We demonstrated how these methods can be used to engage populations of patients and other stakeholders. Challenges with particular methods were highlighted with case studies and potential solutions were identified. Attendees developed an understanding of the possible applications of stated-preference methods and strategies to overcome practical challenges of conducting stated-preference studies.

Intended Audience

Participants interested in learning about patient preference methods to incorporate the patient perspective into health research, care, and policy. No prior knowledge on stated-preference methods was required.

Overview/Outline

We identified research questions that can be answered through stated-preference studies. We discussed good research practices for stated-preference survey development, experimental design, statistical analysis, and interpretation of results.

• Overview of stated-preference methods
  • Definitions of stated preferences methods
  • How do they differ (PROs, QALYs, MCDA)
  • What are common stated-preference methods
    • Pros and cons of each method
  • Applications of stated-preference studies in healthcare decision making
• Good research practices fro stated-preference methods
  • Developing a stated-preference study
    • Survey walk-through
  • Principles of experimental design
  • Different analytical approaches
    • Data setup and analysis exercise
  • Critical appraisal of stated-preference studies
• Case studies in stated-preference methods

Challenges and solutions

Learning Objectives

1. Understand the difference between patient-centered outcomes, health state utility, and patient preference studies
2. Identify research questions and study objectives that can be answered through stated-preference methods and discuss how these methods can be used in a variety of therapeutic areas and stakeholder groups.
3. Recognize good research practice for survey development, experimental design, and analysis when designing stated-preference study

Organizers:

Ellen Janssen, PhD, Janssen Pharmaceuticals, Baltimore, Maryland, United States

Hannah Lewis, PhD, ICON plc, London, United Kingdom

Workshop level: Basic

Workshop Goals:

To provide participants with comprehensive guidelines outlining the steps involved in establishing a minimal important difference (MID) or clinically important response (CIR) on Patient-Reported Outcome (PRO) measures. Using examples from experience in academia and industry, key challenges and considerations when establishing MID or CIR estimates were outlined. The workshop was adapted from successful workshops run at ISOQOL in 2017/2018 to include recent developments in FDA guidance, triangulation, interpretation within clinical practice and item response theory-based approaches.

Intended Audience

Statisticians, Outcomes researchers, Instrument developers, Industry attendees. No prior knowledge of interpretation guidelines was assumed.

Overview/Outline

Establishing clinical relevance is important to numerous key stakeholders. Interpretation based on statistical significance alone isn’t recommended, as any observed difference can be significant depending on sample size. From a clinician perspective, benefits of a new treatment and its implications for clinical management of the patient are key considerations in judging the value of a treatment. The clinical relevance of differences in scores on a PRO instrument may need to be interpreted at the group-level, e.g. when assessing the average benefit of two treatments in a clinical trial, or at the individual-level, e.g. when determining whether a patient has improved or deteriorated significantly over a specified period of treatment.

Various techniques were introduced, using examples to highlight their practical application in deriving thresholds, and the contexts that each is best suited for.

Key challenges were discussed:

• Thresholds for group change (e.g. average of a sample of individuals) compared to thresholds for change within an individual (responder definition): why different methods may be required to establish these and why thresholds obtained may not be the same
• The role of distribution-based estimates, anchor-based estimates, and qualitative approaches (e.g. patient interviews)
• The process of triangulation to combine multiple estimates and a standardized approach.
• Anchor selection and the difficulties identifying good anchors plus the impact on the resulting threshold estimates

The workshop aimed to clarify the methodology in the field leading to a more standardized approach for derivation of MID/CIR.

Learning Objectives

1. Introduction to clinical relevance and methodological overview. Participants gained an understanding of what clinical relevance means in the context of PRO COAs and why it is important for interpretation of PRO data. Participants were introduced to the different terminology used in the field and a framework for interpreting PRO measures at group and individual levels, distinguishing between applications that require interpretation of PRO scores between groups/individuals versus within groups/individuals. Participants learned about the various methods available to judge whether the degree of change (within a group or individual) or difference (between groups or individuals) is clinically meaningful, including novel emerging methodology in the field. They learned that different techniques are required for each different type of interpretation within the above framework. There was a discussion around the use of historical thresholds and how these may need updating in the light of new methodology.
2. Practical examples of deriving and using interpretation guidelines. A real world example was used to show how estimates from anchor-based and distribution-based methods can be combined, and how qualitative data can be utilized. Participants learned about triangulation and how to plan this stage of establishing interpretation guidelines; a systematic approach to triangulation will be presented. In addition, the AURELIA trial was used to illustrate how some of the methodology discussed can be used in practice. This example included the use of cumulative distribution functions to highlight interpretation across a range of possible thresholds for clinical relevance.
3. Challenges, limitations and future direction. Participants developed an awareness of key considerations and challenges associated with deriving interpretation guidelines. Topics covered included finding suitable anchors that correlate well with PRO scores and the impact of this correlation on estimation of clinically relevant differences using ROC methods as an example and the limitations of underlying categorical scales in determining the appropriate level of individual response. Participants heard about the latest developments and solutions for interpreting PRO data.

Organizers:

Kim Cocks, PhD, Adelphi Values, Cheshire, United Kingdom

Andrew Trigg, MSc, Adelphi Values, Cheshire, United Kingdom

Madeleine King, PhD, University of Sydney, Sydney, Australia

Workshop level: Advanced

Workshop Goals:

This tutorial: 1) showed you what the BNP look like using a lot of visual explanations; 2) explained the essential mathematical derivations often omitted in existing expositions you find online; and 3) demonstrated how to fit a simple BNP model. The R code was explained line-by-line to make the equations concrete and explicit, so that you knew precisely how the computation algorithm works. I hoped you would take home with a fundamental intuition to continue to learn more on your own.

Intended Audience

The intended audience was researchers who want to know BNP but find existing tutorials too technical or too simple.
The prerequisites were:

1. 1. Familiarity with R programming (e.g., how to simulate data from standard distributions). However, the programming skills required were no more complicated than writing simple functions or sampling data from standard distributions.
   2. Familiarity with basic Bayesian statistics at an introductory level (e.g., conditional probability and familiarity with why conjugate priors make Bayesian computation easier)
   3. Consider bringing a laptop with R already installed so that you can run the R program right away

Overview/Outline

Machine Learning analytic techniques are gaining popularity at ISOQOL in recent years, e.g., in the Cutting-Edge Research sessions in the 2019 meeting in San Diego. However, exiting applications are primarily based on well-established off-the-shelf methods. There is a need to help ISOQOL members to go beyond these basic methods. Bayesian nonparametric (BNP) models are part of Machine Learning and AI in the internet era. However, tutorials found online tend to be abstract and complex. Key ideas are assumed understood or explained at a level not easily accessible to non-technicians. This tutorial’s primary aim was to make BNP accessible to a broad audience of non-experts, using the Dirichlet Process (DP) mixture model as an illustrative example. I hoped that this workshop would also be useful to more experienced analysts and psychometricians, perhaps as a refresher course.

Learning Objectives

1. Treatment heterogeneity. It is well known that different people respond to interventions differently. Examples of behavioral interventions are programs to encourage smokers to quit, hypertensives to take medications, diabetics to exercise, or individuals with a high risk of a disease to engage in more frequent screening. They may also involve adhering to a daily medication. Scientists always look for newer and more precise person profiling methods to find out the how, when, under what circumstances and for whom the intervention would work best. Bayesian nonparametric methods may be used to discover subgroups of patients with unique PRO profiles. These unique PRO profiles may be linked to differential treatment responses. Our first aim was to understand why we need to discover these PRO profiles.
2. BNP is flexible and adaptive. The main strength of a Bayesian nonparametric (BNP) analytic technique is that BNP models can grow in complexity as it encounters more data. This is accomplished by a unique analytic approach called the “Stick Breaking” algorithm. For example, researchers at Microsoft use BNP methods to predict online fraud based on user characteristics in digital experiments. We went over the “stick breaking” algorithm and its flexibility and discussed how we can capitalize on this feature in the discovery of PRO profiles. We went over the stick-breaking algorithm step by step and explained why it works and how to use it. Hypothetical data was used to help attendees grasp the fundamental ideas of BNP.
3. Real-world data analysis. We used real-world data collected in a grant (NIH R01 CA128134, PI: Breitbart), a randomized controlled trial comparing Individual Meaning Centered Psychotherapy (IMCP, n=109), Supportive Psychotherapy (SP, n=108) and Enhanced Usual Care (EUC, n=104), in reducing psychological distress and improving meaning making in patients with advanced and terminal cancer. I showed participants, step by step, how to use BNP to discover treatment heterogeneity in baseline patient-reported anxiety and depression. Briefly, a BNP cluster analysis identified 5 subgroups of patients with unique profiles of anxiety and depression scores before psychotherapy. BNP shows that cancer patients who report mild symptoms in both anxiety and depression are most likely to respond to IMCP as compared to EUC. BNP cluster analysis in IMPC psychotherapy is just an illustrative example. The techniques can be applied in any other mediation/moderation analysis.

Organizer:

Yuelin Li, PhD, Memorial Sloan Kettering Cancer Center, New York, New York, United States

Workshop level: Advanced

Workshop Goals:

The goals of this workshop were to:
1. Provide the context for patient-reported endpoints in clinical trials and real-world prospective studies,
2. Learn how to define appropriate endpoints from patient-reported outcome (PRO) measures,
3. Learn techniques for statistically evaluating PRO endpoints while considering missing data, and
4. Presenting PRO endpoints for optimum interpretability.

Intended Audience

The workshop was intended for academics, outcomes researchers, and clinical trialists who design clinical trials and real-world prospective studies with patient-reported endpoints and/or who analyze PRO data.

Overview/Outline

1. Background
   1. Importance of PRO endpoints:
      1. It is now mandated in the US that patients are active participants in drug development.
      2. Regulatory and reimbursement authorities in European countries have moved from considering PRO information supportive to required.
      3. As such, the ability to define and interpret these endpoints is increasingly critical.
   2. Considerations for PRO endpoints:
      1. PROs directly measure a patient’s symptoms, functional status, and health-related quality of life.
      2. These complex constructs require careful consideration to capture temporal relationships and severity of response.
      3. Variability in the scoring methodologies and recall periods makes interpretation of results is complex.
      4. Analytic approaches must address complexities to interpret conclusions for non-statistical stakeholders.
2. Defining interpretable patient-focused endpoints
   1. A patient-centered endpoint quantifies patient experience in a meaningful way.
3. Interpretation of patient-focused endpoints. Specifically:
   1. Evaluating meaningful change through responder analyses, including:
      1. Comparing proportions of responders at specific endpoints
      2. Evaluating clinical and symptomatic response jointly
   2. Cumulative likelihood of meaningful response
   3. Time to improvement or deterioration of symptoms or function, including:
      1. When to measure improvement versus deterioration
      2. Defining meaningful improvement/deterioration
      3. Inclusion of death in the event definition
   4. Single item analysis. Specifically:
      1. Why it is important to evaluate item-level response
      2. Heterogeneity of scales
      3. Analytic approaches to:
         1. describe distribution of responses
         2. estimate the likelihood of 1- or 2-grade change
         3. estimate time to 1- or 2-grade change.
   5. Missing PRO data
      1. How can we mitigate the impact of missing PRO response?
      2. Analytic methods for handling missing data
      3. Sensitivity analyses for PROs

Estimands for PRO endpoints

Learning Objectives

1. Given a PRO measure, design three interpretable patient-focused endpoints for a clinical trial or real-world study
2. Articulate the importance of evaluating meaningful change and describe four ways in which to analyze within-person change while accounting for missing data
3. Describe the value of analyzing single items within a PRO scale including the concepts of item construct heterogeneity, temporal relationships to the endpoint and severity variability

Organizers:

Lysbeth Floden, PhD MPH, Clinical Outcomes Solutions, Tucson, Arizona, United States

Melanie Bell, PhD, University of Arizona College of Public Health, Tucson, Arizona, United States

Stacie Hudgens, MA, Clinical Outcomes Solutions, Tucson, Arizona, United States

Workshop level: Advanced

Workshop Goals:

To review measurement theory, connections between theory and various scoring alternatives, and ramifications for how these different scoring alternatives impact the assessment of change in the measurement of health-related states.

After attending, participants understood:

• The fundamental role that a theory of a latent state plays in determining a measurement strategy of that latent state. By showing how to ground measurement approaches in theory, the content and construct validity concepts can be effectively bridged and coherently capture patient experiences.
• The different approaches and mathematics that underpin scoring for classical and modern psychometrics–specifically the differences between sum scores and model-based scores. By evaluating the different approaches, participants understood how the approaches to scoring items rated by individuals capture similar and different aspects of latent stats and how the differences can be expected to manifest in real-world situations.
• Different alternative approaches to analyzing changes in scores in terms of individual differences versus group comparisons. Participants understood the merits of different techniques such as raw changes, residual changes, and regression methods as well as modeling methods. Further the participants explored the important concept of within-individual versus between-group change assessments and how this dichotomy relates to the different approaches to analyzing change scores.

Intended Audience

Practitioners of classical and modern psychometric methods for measurement of health states within clinical research settings.

Overview/Outline

• Introduction and overview
• Measurement and measurement theory
  • Measurement across different fields such as physical and social sciences
  • Psychometrics and clinical/medical research
• Psychometric scoring
  • Classical methods—scores assuming tau-equivalency
  • Modern methods—model-based scoring
• Analysis of scores
  • Group methods
    • Raw change scores
    • Alternative methods such as residual scores and regression estimates
  • Individual differences—accounting for within-person change
  • Implications for practice: Considering the importance of linking within-person change to scores
• Conclusions
  

Learning Objectives

1. For participants to understand the fundamental role that a theory of a latent state plays in determining a measurement strategy of that latent state. Therefore, by grounding measurement approaches in theory, the content and construct validity concepts are bridged and coherently capture patient experiences. To illustrate the relationship between the theories and practices of measurement, examples from across different fields of study, e.g., physical, social, and medical sciences, will be compared and contrasted.
2. The different approaches and mathematics that underpin scoring for classical and modern psychometrics were compared including sum scores and model-based scores, e.g., homogeneous expected a priori scores such as those from a Rasch model, and expected a priori scores from heterogeneous graded response models. By presenting the different approaches, participants gained an understanding of how the approaches to scoring items rated by individuals are similar and different and how the differences can be expected to manifest in real-world situations.
3. Different alternative approaches to analyzing changes in scores in terms of individual differences versus group comparisons were presented, such as raw changes, residual changes, and regression methods as well as modeling methods will be presented. The merits of these different techniques were explored. The important concept of within-individual versus between-group change assessments was used to juxtapose the different approaches to analyzing change scores.

Organizers:

David Andrae, PhD, Endpoint Outcomes, Austin, Texas, United States

Brandon Foster, PhD, Endpoint Outcomes, Boston, Massachusetts, United States

Description:

With increasing evidence supporting the beneficial role PROs can play in clinical practice how should we be advancing the field? Can more progress be achieved through research activity or should implementation and quality improvement projects be given priority? At this roundtable, we discussed the pros and cons, barriers and facilitators around each approach. Attendees had the opportunity to share opinions and experiences.

Host:

Kate Absolom, PhD
University of Leeds

Description:

Metrology offers a natural framework for the development of psychometrics theory. In this roundtable, we discussed how quality-assured measurement can be achieved for PRO measures, building on modern psychometrics methods, typically the Rasch measurement theory. We addressed challenges and opportunity of using metrological principles for clinical research and clinical practice.

Host:

Antoine Regnault, PhD
Modus Outcomes

Description:

The term “minimal clinically important difference” or MCID describes the smallest magnitude of change in a patient-reported outcome score that represents an important patient benefit; yet, in the quest for a minimal threshold may lose sight of what is most meaningful to patients. We shared and discussed methods that provide insights for understanding meaningful change over time.

Host:

Kathy Wyrwich, PhD
Pfizer

Description:

Host:

Chris Sidey-Gibbons, PhD
MD Anderson Cancer Center

Description:

The phrase ‘Patient Engagement’ takes on different meanings when used in different contexts. Through interactive discussion this roundtable explored the different ways patient engagement is defined and how it has and continues to evolve.

Host:

Danielle Lavalle, PharmD PhD
BC Academic Health Sciences Network

Moderator:

John Brazier, PhD, School of Health and Related Research, The University of Sheffield, Sheffield, United Kingdom

Discussant:

Madeleine King, Cancer Australia Chair in Cancer Quality of Life, School of Psychology, University of Sydney, Sydney, New South Wales, Australia

Individual Presentations:

EXTENDING THE QALY (E-QALY): Developing a new generic preference-based measure for use with patients, carers and those needing care – an overview

John Brazier, PhD, School of Health and Related Research, The University of Sheffield, Sheffield, United Kingdom

E-QALY: Identifying dimensions and items for a generic preference-based measure

Clara Mukuria, University of Sheffield, Sheffield, United Kingdom

E-QALY: Psychometric testing and selecting the items for a new generic preference-based measure

Ole Marten, MSc, School of Public Health, Department of Health Economics and Health Care Management, Bielefeld University, Bielefeld, Germany

E-QALY: Face validity testing the items for a new generic preference-based measure

Zhihao Yang, PhD, Jinan University, Guangzhou, China

E-QALY: Valuing the items for a new generic preference-based measure using time trade-off and discrete choice experiments

Donna Rowen, PhD, University of Sheffield, Sheffield, United Kingdom

Moderator:

Corneel Coens, EORTC HQ, Brussels, Belgium

Discussants:

Galina Velikova, PhD MD, Leeds Institute of Medical Research at St James’s, University of Leeds, Leeds Cancer Centre, Leeds, United Kingdom

Madeleine King, PhD, University of Sydney, Sydney, Australia; Submitted on behalf of EORTC Quality of Life Group

Individual Presentations:

Interpretation of patient reported outcome data at group-level versus individual-level; can we use the same clinical meaningful thresholds for both scenarios?

Jammbe Musoro, PhD, EORTC HQ, Brussels, Belgium

Development and use of thresholds for clinical importance to facilitate interpretation of scores from patient-reported outcome measures

Johannes Giesinger, PhD, Medical University of Innsbruck, Innsbruck, Austria

Norm data for the EORTC QLQ-C30 health-related quality of life questionnaire based on 15,386 persons from the European general population, Canada, and the USA.

Sandra Nolte, PhD, ICON plc, Berlin, Germany

Moderator:

Angela Stover, PhD, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, United States 

Discussant:

Maria Santana, PhD, University of Calgary, Alberta, Canada

Individual Presentations:

One vehicle, too many wheels: towards consistent use of theory for guiding the implementation of PROMs/PREMs in routine clinical practice

Caroline Potter, PhD, University of Oxford, Oxford, United Kingdom

Implementation of PROMs in routine clinical care within an integrated pain network

Sara Ahmed, PhD, McGill University, Montreal, Quebec, Canada

Using the Integrated framework Promoting Action Research in Health Services (iPARIHS) Framework to study implementation of PROMs into oncology care

Natasha Roberts, PhD, Queensland University of Technology, Kelvin Grove, Australia

Using Implementation Science to Inform Integration of Electronic Patient-Reported Experience Measures (ePREMs) into Healthcare Quality Improvement

Kimberly Manalili, PhD Candidate, University of Calgary, Calgary, Alberta, Canada

Implementation of the KLIK PROM portal using the Consolidated Framework for Implementation Research (CFIR) retrospectively

Hedy van Oers, PhD, Emma Children’s Hospital Amsterdam UMC, Amsterdam, Netherlands

Implementation science metrics to evaluate patient-reported outcome measure (PROM) implementation initiatives in routine care settings

Angela Stover, PhD, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, United States

Sponsored by Pharmerit

Moderator:

Pronabesh Dasmahapatra, MD MPH, Head, Heath Economics and Value Assessment Strategy, Rare Disease and Rare Blood Disorder, Cambridge, Massachusetts, United States

Individual Presentations:

Most Bothersome Symptom in Rare Disease: A Look at Endpoint Precision and Validity

R. J. Wirth, PhD, Vector Psychometric Group, LLC, Chapel Hill, North Carolina, United States

Hypothesis Tests to Evaluate Treatment Efficacy that Optimize Power and Precision in Rare Disease Settings

Charles Iaconangelo, PhD, Pharmerit – an OPEN Health Company, Brooklyn, New York, United States

A New Method for Testing Significance of eCDF Separation for Meaningful Change in Rare Diseases

Daniel Serrano, PhD, Pharmerit – an OPEN Health Company, Bethesda, Maryland, United States

Moderator:

Gita Thanarajasingam, MD, Division of Hematology, Mayo Clinic, Rochester, Minnesota, United States

Individual Presentations:

Analytic and Interpretation Approaches for PRO-CTCAE, CTCAE, and Other Clinical Data to Characterize Drug Tolerability

Amylou Dueck, PhD, Mayo Clinic, Scottsdale, Arizona, United States

Measuring the Tolerability of Cancer Therapy: Incorporating Host Factors as Moderators of Toxicity and Outcomes

Patricia Ganz, MD, Department of Health Policy and Management, UCLA Fielding School of Public Health, Los Angeles, California, United States

Relationships between aging-related conditions and treatment tolerability in older adults with advanced cancer

Eva Culakova, PhD, University of Rochester, Rochester, New York, United States

Analysis of ECOG-ACRIN Clinical Trials to Advance Longitudinal Assessment of Cancer Treatment Tolerability

John Devin Peipert, PhD, Northwestern University Feinberg School of Medicine, Chicago, Illinois, United States

Moderator:

Danielle C. Lavallee, PharmD PhD, University of Washington, Seattle, Washington, United States.

Individual Presentations:

Recommendations for governing PRO use across health systems

Danielle C. Lavallee, PharmD PhD, University of Washington, Seattle, Washington, United States

Governing PROs across health systems: Case study at University of Utah

Rachel Hess, MD MS, University of Utah, Salt Lake City, Utah, United States

Recommendations for integrating PROs into clinical care delivery

Elizabeth Austin, University of Washington, Seattle, Washington, United States

Integrating PROs into care delivery: Case study at University of Pittsburgh Medical Center

Janel Hanmer, MD PhD, University of Pittsburgh, Pittsburgh, Pennsylvania, United States

Moderator:

Sophi Tatlock, Adelphi Values Ltd, Bollington, United Kingdom

Individual Presentations:

Using qualitative exit interviews to explore the disease experience and evaluate Performance-Reported Outcomes (PerfOs) with patients who experience muscle weakness

Maggie Tabberer, MSc, GlaxoSmithKline, London, United Kingdom

Qualitative exit interviews with study clinical investigators to explore trial and treatment experience and feasibility of use in general clinical practice

Nicola Williamson, Adelphi Values Ltd, Bollington, United Kingdom

Patient perspectives on the benefit of a novel therapy for a rare disease: Using qualitative exit interviews to inform post-launch value messaging and to support shared treatment decision-making

Parth Vashi, PharmD, Bayer Pharmaceuticals, Whippany, New Jersey, United States

Use of qualitative exit interviews to explore individual experiences of treatment and meaningful change in two clinical trials for autism spectrum disorder.

Elizabeth Gibbons, MSc, Clinical Outcomes Solutions, Folkestone, United Kingdom

Moderator:

Philip Griffiths, PhD, Adelphi Values, Bollington, United Kingdom

Discussant:

Michelle Campbell, U.S. Food and Drug Administration, Silver Spring, Maryland, United States

Individual Presentations:

How can transdisciplinary collaboration help us ensure that the digital medicine tools we are being asked to place our trust in are indeed trustworthy?

Jennifer Goldsack, MChem MA MBA, Digital Medicine Society (DiMe), Boston, Massachusetts, United States

Industry Considerations for the use of Technologies for Continuous Data Capture

Jiat Ling Poon, PhD, Eli Lilly and Company, Indianapolis, Indiana, United States

Technologies for continuous data capture in clinical trials – the logistics of makings it happen

Paul O’Donohoe, Medidata, London, United Kingdom

Technologies for continuous data capture: Discussing dilemmas with data and considering potential analysis approaches

Carrie Houts, Vector Psychometric Group, Tempe, Arizona, United States

Moderators:

Bellinda King-Kallimanis, PhD, U.S. Food and Drug Administration, Silver Spring, Maryland, United States
Madeline Pe, PhD, EORTC Quality of Life Department, Brussels, Belgium

Individual Presentations:

An HTA perspective on the treatment policy estimand if the endpoint is repeatedly measured over time

Christoph Schürmann, Dr., IQWiG – Institute for Quality and Efficiency in Health Care, Cologne, Germany

Why & how to use time-to-event endpoints for COAs & how can the estimand framework help?

Rachael Lawrance, Adelphi Values Ltd, Bollington, United Kingdom

Applying the Estimand Framework to Describe Patient Experience While on Treatment: A Case Study

Mallorie Fiero, PhD, U.S. Food and Drug Administration, Silver Spring, Maryland, United States

Moderator:

John Chaplin, PhD, Sahlgrenska Academy at University of Gothenburg, Gothenburg, Sweden

Individual Presentations:

The Kids-CAT: a computer adaptive tool to measure quality of life in children and adolescents. Development, validation and implementation in clinical settings and population health reporting

Christiane Otto, PhD, University Medical Center Hamburg-Eppendorf, Hamburg, Germany

From translation to implementation: the challenge of PROMIS® CATs in daily pediatric practice.

Michiel Luijten, Emma Children’s Hospital, Amsterdam UMC, Amsterdam, Netherlands

Comparison of PROMIS Computerized Adaptive Testing-Administered Item Banks versus Fixed Short Forms in Juvenile Myositis

Kaveh Ardalan, MD MS, Duke University School of Medicine, Durham, North Carolina, United States

Using CATs to measure symptom burden reported by children with brain tumors

Jin-Shei Lai, Northwestern University, Chicago, Illinois, United States

Application of the Kids-CAT in the clinical setting: Predictors of health-related quality of life in children and adolescents with chronic diseases

Kathrin Fischer, Charité Universitatsmedizin Berlin, Berlin, Germany

Moderator:

Linda Nelsen, MHS, GlaxoSmithKline, United States

Speakers:

David Cella, PhD, Northwestern University, United States
Heather Gelhorn, PhD, Evidera, Bethesda, United States
Antoine Regnault, PhD, Modus Outcomes, France
Jill Bell, PhD, Takeda Oncology, United States

Discussant:

Elektra Papadopoulos, MD MPH, U.S. Food and Drug Administration, United States

Items banks and libraries such as PROMIS, PRO-CTCAE and EORTC Item Library support measurement of PRO concepts across therapeutic areas. Creating custom PRO measures, or short-forms, from item banks / libraries may enhance measurement precision and validity, and reduce resources required for PRO development. However, development of short forms optimized for use in specific target populations from these measurement systems requires a robust research plan, with clarity of concepts, rigorous qualitative research, and sophisticated psychometric methods.

This symposium provided an overview of item banks and libraries, with examples of the application of these measurement systems for specific populations, highlighting benefits and challenges to their use.

Presentations considered item selection approaches supporting appropriate content coverage and ensuring an understanding of measurement properties of these custom measures.  An interactive panel discussed the value of item banks to patients, clinicians, and industry.

Moderators:

Motolani Ogunsanya, BPharm MS PhD, University of Oklahoma Health Sciences Center, United States
Carla Dias Barbosa, MSc, Evidera, United Kingdom

Speakers:

Sophie Cleanthous, PhD, Modus Outcomes, United Kingdom
Mona Martin, PhD, Evidera, United States
Motolani Ogunsanya, BPharm MS PhD, University of Oklahoma Health Sciences Center, United States
Carla Dias Barbosa, MSc, Evidera, United Kingdom
Skye Barbic, PhD, University of British Columbia, Canada

Recent advances in mixed methods research (MMR) provide an opportunity to better understand patient experience with disease and/or treatment and impact on their quality of life (QOL) by integrating qualitative and quantitative methods in research design and using patient-reported outcome measures (PROMs) developed through robust patient engagement and advances in metrology.

This symposium provided an overview of MMR in the context of health outcome research and highlight multiple international examples about the application of these methods in clinical trial research for evaluation of treatment benefit, PROM development, and testing and implementation of PROMs in clinical practice and policy. This symposium used a graded learning approach by first introducing MMR in QoL research to the audience (presentation #1), use of MMR in PROM development (presentation #2), PROM testing and validation (presentation #3), treatment risk-benefit assessment in clinical trials (presentation #4) and PROM implementation in policy and system transformation (presentation #5).

Moderator:

Henrik Rode Eshoj, PhD, Odense University Hospital, Denmark

Speakers:

Frances M. Yang, PhD, University of Kansas Medical Center, United States
Andrew Trigg, MSc, Adelphi Values, United Kingdom
Philip Griffiths, PhD, Adelphi Values, United Kingdom
Jakob Bjorner, MD PhD, Optum Patient Insights, United States
Berend Terluin, MD PhD, Amsterdam University Medical Center, the Netherlands
Myriam Blanchin, PhD, University of Nantes, France

Discussant:

Carrie R. Houts, Vector Psychometric Group, United States

Item Response Theory (IRT) is normally used at a single timepoint. However, when longitudinal data is available other advantageous applications of IRT exist. In this context, IRT can be used to evaluate change, derive meaningful change thresholds and understand response shift. Additionally, longitudinal data may help support IRT models for instrument validation in small samples.

This symposium presented the “what, when, and how” of Longitudinal IRT (LIRT). Attendees were introduced to the concepts of IRT and how to apply this in a longitudinal dataset to explore a tool’s measurement properties. We built on this by showing examples of using LIRT to assess and interpret change and investigate and adjust for response shift. Content was discussed by an experienced psychometrician, ensuring this complex topic was delivered in a practical and accessible way. Example code was provided to help attendees apply LIRT to their own research.