NCHR comments on FDA’s Framework for Real-World Evidence Program

National Center for Health Research; February 5, 2019

National Center for Health Research’s Comments on
FDA’s Framework for a Real-World Evidence Program; Availability


Thank you for the opportunity to provide our views on the proposed framework for the Real-World Evidence (RWE) Program.

The National Center for Health Research is a nonprofit think tank that conducts, analyzes, and scrutinizes research, policies, and programs on a range of issues related to health and safety. We do not accept funding from companies that make products that are the subject of our work.

Studies designed around RWE have the potential to provide useful information about the effectiveness of drugs and biologics, particularly if they can increase the demographic diversity of patients in these studies and more closely reflect typical medical practice. But real-world data (RWD) and sources of RWD have many limitations and complicating factors that raise concerns about the validity and credibility of the data for either safety or effectiveness. Because RWD and RWD sources are not necessarily designed to answer questions about the safety or effectiveness of specific treatments, there are frequently problems with measuring outcomes or controlling for confounding variables.

As with traditional clinical trials, RWE endpoints or outcomes should demonstrate safety and effectiveness in ways that are meaningful to patients. Many of the outcomes recorded in RWD sources do not fulfill these criteria. For example, using outcomes such as the numbers of days in the hospital or need for surgery would be useful information but not sufficient if numerous patients were too fragile for additional treatment or decided they wanted to die at home. Although hospitalization and surgery are outcomes that are important to patients, there are many other outcomes that affect quality of life that patients may value even more. For example, the ability to walk up a flight of stairs without getting winded or to eat without getting nauseous are very valuable to many patients, but would not be coded in a medical record or included in billing data. Similarly, the fact that a patient changed medications may not be as important as the reason why he or she changed medications.

Outcomes that measure benefits or complications may not be recorded consistently by different clinicians or healthcare facilities. To provide accurate outcome data, researchers may need to develop methods to make sure that information is accurately extracted from RWD. Unfortunately, it is not always possible to get an accurate assessment of the outcome because it is not always completely recorded or coded.

The population that is studied should be similar to the population of patients that is likely to consider using the product. While clinical trials can be biased if they tend to include healthier patients, the manner in which the RWD is collected (for example, from an app) could bias the sample toward younger, more educated, or tech savvy patients. In addition, patient characteristics, such as obesity or tobacco use, may not be consistently reported or available in a dataset, making it difficult to determine how generalizable the data are or to evaluate confounding variables.

We commend the FDA for the proposed Framework and the planned guidances. We support the FDA’s efforts to urge the use of RWD sources that are the least biased and most valid and credible, and to use methods that also minimize bias and are scientifically valid. However, even the current and planned guidances discussed in the Framework may not be sufficient to completely mitigate the limitations of these studies.

We applaud the FDA’s efforts to encourage the best possible study design for RWE. As with traditional prospective clinical trials, studies that generate RWE are more convincing when they are randomized, have control or comparison group(s), and include blinding. Studies lacking any or all of these design components increase risk for bias to skew results. Retrospective observational studies have additional concerns. For example, it can be challenging to use medical records or billing records to define the appropriate patient population and to clearly identify the reason why the patient is receiving the specific treatment. Those and other design or statistical aspects of the study can be manipulated in order to maximize the likelihood of statistically significant results. Guidances on relevant clinical trial and study designs as discussed in the Framework can encourage better studies, but cannot mitigate all concerns inherent in an RWE study design.

In summary, studies generating RWE have additional complications beyond the design concerns typical for traditional clinical trials. The guidances and policy changes proposed in the Framework can improve RWE studies but cannot completely resolve their limitations and biases. Relevant and reliable RWD and controlled clinical trial designs increase the validity and credibility of the results, but other aspects of the study are important for determining how useful the results are to help patients and physicians make informed treatment choices. For these reasons, observational studies and other non-controlled RWE studies should not be used as the primary evidence for FDA’s regulatory decisions about a product’s effectiveness. They can provide very useful supplemental evidence and help us interpret other data. However, RWE should not replace traditional, randomized, blinded, controlled clinical trials.

In conclusion, the FDA needs to maintain high standards for evidence of effectiveness and safety whether the source of the data is from traditional clinical trials or RWE. Double-blind, randomized clinical trials remain the gold standard, and along with other types of well-controlled trials (including those that generate RWE), provide information that observational studies cannot. RWE should supplement rather than replace traditional, randomized and blinded controlled clinical trials except in situations where such traditional clinical trials are not possible.


The National Center for Health Research can be reached through Stephanie Fox-Rawlings, PhD at