Opening and Session 1


The meeting started with opening remarks by Janet Woodcock, Director of the Center for Drug Evaluation and Research. (CDER), and a reminder of the context, i.e., the PDUFA V Commitments, by Theresa Mullins, PhD, Director of the Office of Strategic Programs, CDER.

The Prescription Drug User Fee Act (PDUFA) was first authorized in 1992 and was conceived to overcome the “drug lag” of the 1980s by providing the FDA with increased staffing and resources, and by encouraging medical innovation. The PDUFA was reauthorized (as PDUFA V) as part of the Food and Drug Administration Safety and Innovation Act, enacted in July 2012, and went into effect on October 1, 2012. The PDUFA V can be seen as a set of targeted program enhancements: (1) Enhancing regulatory science/expediting drug development (i.e., by promoting innovation through better communication between the FDA and sponsors during drug development, methods for meta‐analysis, biomarkers and pharmacogenomics, use of Patient‐Reported Outcomes (PROs), and encouraging development of drugs for rare diseases); (2) Enhancing benefit‐risk assessment; and (3) Enhancement and modernization of the FDA’s drug safety system.

Regarding the use of PROs, Theresa Mullins acknowledged the fact that there is a high study‐failure rate for PRO endpoints not qualified in advance of Phase III trials. This situation led to two recommendations: (1) Enhance clinical and statistical capacity to address submissions involving PROS and other endpoint assessment tools, including providing IND consultation, and (2) Hold a public meeting to discuss the FDA’s qualification standards for drug development tools, new measurement theory, and implications for multi‐national trials (which was basically the program of the workshop).

Finally, Theresa Mullins introduced the FDA's Patient-Focused Drug Development initiative as part of the commitments under PDUFA V (i.e., benefit-risk assessment) that aims to more systematically gather patients’ perspectives on their condition and available therapies to treat their condition. Twenty public meetings over the course of PDUFA V are planned; each will be focused on a specific disease area.


Session 1: Experiences with FDA Guidance on Patient-Reported Outcome Measures and the Clinical Outcome Assessment Tool Qualification Process
Session 1 was chaired by Elektra Papadopoulos, the acting Associate Director for Study Endpoints and Labeling Development at CDER’s Office of New Drugs.

Session 1 gave the floor to four speakers: Elektra Papadopoulos, Katarina Halling, Bryce Reeve, and Paul Kluetz.

In her introduction of Session 1, Dr. Papadopoulos provided some context with the take-away points from the Brookings Institution meeting. These points were organized around “What the FDA Can Do.” She presented four points: (1) Work with internal and external stakeholders to show how PRO development and implementation can be operationalized in real world; (2) Be more proactive in encouraging use of PROs; (3) Enhance communications, processes, and transparency (e.g., address the need for earlier FDA/industry communication (e.g., compound startup phase before pre-IND/IND submission), and publish a list of “potentially acceptable” clinical outcome assessments); and (4) Enhance consistency of advice. She also introduced the Clinical Outcomes Initiative to increase transparency in regulatory decisions, which is organized into two areas: (1) The development of a compendium of COAs (see Dr. Trentacosti’s presentation in Session 2); and (2) The advance of scientific standards and policy development.


Clinical Outcome Assessment to Demonstrate Treatment Benefit: An FDA Perspective - Elektra Papadopoulos, MD, the acting Associate Director for Study Endpoints and Labeling Development at CDER’s Office of New Drugs

This presentation was a reminder of definitions (i.e., treatment benefit, COAs, and content validity) and basics such as the roadmap to patient-focused outcome measurement in clinical trials, the FDA’s 2009 PRO guidance, and the FDA’s qualification process.

Dr. Papadopoulos reminded attendees that treatment benefit is demonstrated by evidence that the treatment has a positive impact on how a person with the condition or disease survives, feels, or functions in daily life. Treatment benefit is measured by various types of outcome assessments, i.e., survival, COAs, and surrogates. Surrogates are often biomarkers intended as a substitute for how a patient feels, functions, or survives. She recalled the official definition of COAs, i.e., any assessment that may be influenced by human choices, judgments, or motivations and may support either direct or indirect evidence of treatment benefit. On the regulatory standpoint, treatment benefit must be documented by “substantial evidence” [21 CFR 201.56(a)(3)]. Evidence must be obtained from “adequate and well-controlled clinical trials.” Therefore, the methods of assessments need to be “well-defined and reliable.” [21 CFR 314.126] For COAs, this means measuring the right thing (concept of interest), in the right way in a defined population (targeted context of use), and the score that quantifies that “thing” does so accurately and reliably, so that the effects seen in the outcome assessment can be interpreted as a clear treatment benefit.

Then, Dr. Papadopoulos presented the roadmap to patient-focused outcome measurement in clinical trials (see Figure 1) to illustrate how one might embark upon a sound and orderly instrument selection or development pathway that is in alignment with the objectives of the drug development program and the clinical trial context of use. It showed that proper assessment of an outcome in a clinical trial should be underpinned by a sound understanding of the disease (column 1) and a precise and specific conceptualization of the expected treatment benefit (column 2).

Figure 1. Roadmap to Patient-Focused Outcome Measurement in Clinical trials


She recalled the FDA’s 2009 PRO guidance for industry [1] which defines a PRO as a measurement based on a report that comes from the patient about the status of a patient’s health condition without amendment or interpretation of the patient’s report by a clinician or anyone else, and describes good measurement principles, many of which are applicable to other types of COAs. And while it provides an optimal or ideal approach to PRO development, she recognized that flexibility and judgment are needed to meet practical demands.

She reminded the audience that establishing content validity is key in determining how a measure is adequate. Content validity is defined as the evidence that the instrument measures the targeted concept in the context of use, and that the score represents the concept. It is supported by a literature review as well as by expert and patients inputs. It begins after confirmation that the concept of interest and the context of used are appropriate. Testing other measurement properties (e.g., test-retest reliability, construct validity, and ability to detect change) will not replace or rectify problems with content validity.

Dr. Papadopoulos recalled the spokes and wheels of the FDA’s PRO guidance (see Figure 2), emphasizing the important of identifying the context of use and the concept of interest from the start during PRO instrument development.

Figure 2. Spokes and Wheels


She then finished her presentation with an overview of the Drug Development Tool (DDT) [2] qualification process, which was intended to expedite development of publicly available DDTs that can be widely employed.[3] Drug developers can use a DDT that has been qualified within a specific context of use for the qualified purpose during drug development as long as: (1) The study is conducted properly (e.g., all procedures and protocols specified in the context of use are followed); (2) The DDT is used for the qualified purpose; and (3) At the time of qualification, there is no new information that conflicts with the basis for qualification. In the 2014 guidance,[3] the qualification process is described for biomarkers, animal models, and COAs. Qualification is a conclusion that within the stated context of use, the DDT can be relied on to have a specific interpretation and application in drug development and regulatory review. Formal qualification through a COA drug development qualification program is completely voluntary and it in no way means that an instrument needs to be qualified to be acceptable for use in a clinical trial to support labeling claims.


  1. FDA’s PRO guidance for industry, available at
  2. DDT COA qualification program, available at
  3. FDA’s DDT qualification program guidance for industry, available at


A Pharmaceutical Perspective on PRO evaluation in Drug Development - Katarina Halling, MSc, Group Director, PRO Center of Excellence, AstraZeneca

Katarina Halling started her presentation by recalling that 20 years ago the patient’s perspective in drug development was not a priority. It wasn’t easy to convince medical departments to include health-related quality of life evaluation (as it was called at that time) in clinical trials. In her perspective, the FDA PRO guidance was an important event. With this guidance, the FDA had built the foundation for PRO evaluation with the objective of obtaining a PRO claim in the label. However, she recalled that the release of the guidance led to a slight decrease in PRO claims which was a concern for Pharma. In addition, she had noticed variations in the interpretation of the guidance across FDA divisions and with the SEALD, which were confusing for the industry.

Over the years, she has seen a better collaborative effort between pharmaceutical companies in sharing the development of PRO measures. She considered this as a huge step forward. However, she reminded the audience that this collaboration could be challenging and time consuming. There have been a lot of critics because only one instrument has been qualified so far, i.e., the EXACT-PRO. She said that researchers are learning from this experience and that, although things are not perfect, she believed we are moving in the right direction. She also noted that recently FDA has been more responsive and giving more details in the reviewers’ feedback.

She recalled the key points discussed during the Brooking Institution meetings, i.e., the need to improve communication with the FDA, the suggestion of having early meetings to review the COA strategy in drug development, the development of a list of potentially acceptable instruments (which will go into a compendium – see Session 2), the discussions around establishing content validity and minimally important change, and the development of a research agenda to solve pending issues collaboratively, especially in oncology (see Dr. Kluetz’s presentation).

Ms. Halling summarized her speech by stating that she believed that these aforementioned points are good foundations to move forward in solving all pending issues, in collaborating in the identification of the best endpoints to understand treatment benefit, and in establishing the best ways to communicate with the FDA, with the goal of including the patient’s perspective more and more in drug development.


Revisiting Guidance on PRO Measure Design and Evaluation - Bryce Reeve, PhD, Associate Professor, Health Policy & Management Member, Lineberger Comprehensive Cancer Center

The presentation of Dr. Reeve was given from an academic perspective.

He recalled that the first draft FDA PRO guidance was released in 2006 and, since then, the guidance has done a lot to advance the field of outcome research. It has set standards for the development of PRO measures and the type of evidence needed to use PRO measures for drug labeling claims, as well as standards for use of PROs in clinical outcomes research.

There have been a lot of meetings, conferences and hallways discussions around this guidance, of which the Brookings meetings are a good example. Dr. Reeve evoked a quote taken from the last Brookings meeting in October 2014: “Only one PRO instrument has been fully qualified by the FDA, and the PRO label claim approvals have declined slightly since the publication of the guidance.” He also shared commentaries from Kate Rawson [1]: “The requirements…are so onerous as to dissuade companies from pursuing PRO claims.” “…a sense that sponsors have to develop ‘perfect’ tools.” He also shared with attendees a quote from Heisenberg that he uses with his students, i.e., “What we observe is not nature itself but nature exposed to our method of questioning.” He said that it was a good reminder of what the field of PRO research is doing, by recognizing that researchers are not capturing the true outcome, the true concept of interest, but rather an outcome that is filtered through a measure.

He added that no matter how well the measures are designed, they will never get a true indicator of what the concept is. Therefore, in his view it is right to say that there is no perfect COA. Recognizing that none of them are perfect is acknowledging that there is some type of measurement error.

Therefore, what the researchers are trying to do is to reduce this measurement error, so the observed score more closely reflects what the true score actually is. This fact was acknowledged by the writers of the FDA guidance who put forward a set of standards and attributes that would reflect a high-quality PRO. To illustrate this, Dr. Bryce presented the eight attributes that were put forward by the Medical Outcomes Trust [2]. See Figure 1.

Figure 1.


The challenge has been in the fact that some in the industry—and some FDA reviewers in their design and evaluation—have thought that each instrument had to meet the highest criteria of rigor and quality and that the design had to be done within the context of a very specific population, for a very specific population and a very specific context of use. This strategy increased efforts, timelines, and costs for the sponsors.

Dr. Reeve reminded the audience that methodological research was conducted to look at the assumptions of the field about principles and guidelines around PRO evaluation. He gave the example of studies that looked at what is the optimal recall period or administration modes. He also cited guidance from organizations such as ISPOR, ISOQOL, PCORI, and COSMIN.

He welcomed the opportunity given by the workshop to revisit the FDA guidance, look at the new recommendations and research, and improve the understanding of what has changed with the goal of enhancing the recommendations about the use of PROs for drug labeling claims.

Dr. Reeve ended his presentation by presenting a list of topics that could fuel future discussions and help in revisiting the FDA guidance. See Figure 2. Under area 3, he suggested the development of function-specific measures, such as fatigue that could be used across several diseases and conditions.

Figure 2.

Bryce areas


  1. Kate Rawson, “Getting with the PRO-gram: Making PROs work.” Pharma & MedTech Business Intelligence, Jan 2015.
  2. Scientific Advisory Committee of the Medical Outcomes Trust. Assessing health status and quality‐of‐life instruments: Attributes and review criteria. QOL Research 2002; 11:193‐205.


Addressing the challenges of clinical outcomes assessments in oncology - Paul G. Kluetz, MD, Office of Hematology and Oncology Products, CDER

Dr. Kluetz’s presentation of was organized around two key points, i.e., (1) the challenges of drug development and PRO evaluation in oncology, and (2) the possible solutions, either scientific or organizational.

After telling an anecdote that happened during his shift at Georgetown Hospital, Dr. Kluetz recalled all the meetings he attended in 2014 about PROs and the patient’s perspective in drug development (see Table 1).

Table 1. Recent Meetings Highlighting PROs

May 2014 FDA co-sponsored AAADV PRO Plenary Session
May 2014 PRO Consortium
July 2015 Brookings “Enhancing the Development and Use of Patient-Reported Outcomes in Drug Development”
October 2014 AACR Turning the Tide Against Cancer – Patient Perspective Panel
October 2014 Brookings “Advancing Development and Use of Patient-Reported Outcomes in Drug Development: Near-term Opportunities”
October 2014 FDA co-sponsored NBTS “Brain tumor Clinical Trial Endpoints Workshop 2 (Clinical Outcomes Assessments)

He presented the take-home messages from these meetings:

  • Capturing the patient’s perspective in drug development continues to gather momentum as a priority; all stakeholders agree on this now.
  • What to measure, how to measure it, how to interpret what we have measured, and how to present it are very complicated issues. They are even more so in oncology.
  • All parties appear dedicated to finding a way to improve PRO instruments, collection, and interpretation across different disease areas.

Dr. Kluetz also presented the feedback he had heard from industry, advocates and PRO developers. He noted it has been focused on three key areas:

  • The 2009 FDA PRO guidance was needed at the time. However, it is considered as too stringent, too unrealistic, and unfeasible.
  • There is an inconsistency in advice about PROs between SEALD and OHOP, and within different OHOP Divisions.
  • FDA OHOP does not put PROs in labels as often as other therapeutic areas.

What are the challenges in oncology?

Dr. Kluetz listed the main challenges in oncology:

  • There is a lack of agreed-upon instruments;
  • The oncology trial design is not optimized for PROs. The trial designs are very challenging for PROs. If there is a great drug in oncology, it will probably be approved on a single-arm trial to make it available to patients as soon as possible. How do you interpret PROs in a single-arm trial? It is a challenge and something that needs to be clarified. In addition, even if oncology trials are randomized against a comparator, they are frequently just open-label to start because blinding is often unfeasible, since the toxicity of the drug is so obvious, everyone already knows what they are on.
  • A significant proportion of PRO data is frequently missing.
  • Data analysis and data presentation are not standardized.

And finally, there is a lack of familiarity with PRO data analysis for oncology clinical trials reviewers (both statistical and clinical) as oncologists have relied on survival and radiographic evidence of treatment benefit. In other areas, such as psychiatry or irritable bowel syndrome, there is considerable familiarity with PROs. They have extensive experience because this is the only endpoint they can measure. It is a heavy lift for oncologists to change their attitude towards PROs and make it an important thing for them.

Towards more consistent PRO advice

Dr. Kluetz presented the changes of organization within his division and the increased interactions with the SEALD over the last several years:

  • Increased collaboration between OHOP and SEALD: OHOP-SEALD working groups with monthly meetings.
  • Improved OHOP PRO and labeling expertise
    • Office level dedication to advance PRO in oncology
    • PRO leads and associate directors of labeling in each OHOP clinical division provide PRO and labeling expertise and interact with SEALD: consistency between OHOP divisions
    • Overall OHOP educational opportunities for reviewers: monthly hematology oncology case series.

The main change was moving away from a parallel process toward a more collaborative approach. See Figure 1 and 2.

The goal is to provide more detailed, consistent, and proactive PRO advice to sponsors.

Figure 1.

Parallel Process

Figure 2.

Collaborative effort

Identifying PRO Instruments

Dr. Kluetz emphasized the fact that, in the long term, new instrument development should be encouraged. However, in the short term, he approved the idea of identifying existing instruments that can be sued or modified as “reasonable” for use in trials. Dr. Kluetz recalled that the OHOP acknowledges that the PRO guidance is a road map for a “gold standard” for PRO instruments in drug development. However, he recalled that the perfect is the enemy of good, and flexibility may need to be exerted.

Moving Forward

Dr. Kluetz said that he would like to see standardization of PROs in cancer clinical trials. Core concepts to be measured in all oncology trials should be identified. Those may include disease symptoms, treatment-related symptoms, and physical function. Researchers should focus on how we measure these concepts:

  • Instrument identification
  • Optimal trial design and assessment frequency
  • Optimal methods to minimize missing data
  • Optimal statistical analysis methods

Finally, he emphasized the fact that ways to most accurately present the results in a label should be found, i.e., most informative vs. least misleading ways, and that there is a need to strike a balance. See Figure 3.

Figure 3.



Dr. Kluetz ended his speech by stating that OHOP is working closely with SEALD and all stakeholders to improve the quality of patient-reported data obtained in cancer clinical trials, and that standardization of what is measured and how it is measured will be critical as we move forward.


Priorities and challenges of Session 1 were synthetized by Elektra Papadopoulos:

- Development of the compendium;

- Advance scientific standards;

- Need for greater regulatory flexibility;

- Need for good principles for the development of other COAs than PROs;

- Need for standards for the analysis of pro data and display of pro results.

She also briefly mentioned the Critical Path Innovation Meetings Guidance.