May 2, 2023
The National Center for Health Research (NCHR) appreciates the opportunity to submit public comments on the Food and Drug Administration (FDA) Considerations for the Design and Conduct of Externally Controlled Trials for Drug and Biological Products.
NCHR is a non-profit think tank that conducts, analyzes, and scrutinizes research on a range of health issues, with particular focus on which prevention strategies and treatments are most effective for which patients and consumers. We do not accept funding from companies that make products that are the subject of our work, so we have no conflicts of interest.
This draft guidance addresses considerations for the design and analysis of externally controlled clinical trials to study the effectiveness and safety of drugs, including potential bias that can undermine the validity of trial results. The guidance primarily focuses on the use of patient-level data from real-world data (RWD) sources, such as registries, electronic health records, and billing records which are used in the control arm of these trials.
FDA’s recognition of the many threats to the validity of a trial that comes with an externally controlled design is important, however, this guidance should be more explicit about the criteria that must be met to ensure that externally controlled trials are only used when necessary. A randomized, placebo-controlled trial is the gold standard, and any deviation from this standard must be closely scrutinized and restricted. We agree that determining the suitability of an externally controlled trial design warrants a case-by-case assessment, but the FDA should be more explicit than stating that “sponsors and investigators should consider the likelihood that such a trial design would be able to distinguish the effect of a drug from other factors that impact the outcome of interest.”
The draft guidance includes numerous important considerations to determine the appropriate use of externally controlled trials. We provide comments and additional recommendations, including the following:
- Characteristics of study populations
The effective use of RWD requires a substantial number of patients that are demographically and medically similar to the treatment population. We appreciate that the FDA is addressing potential population biases based on differing demographic attributes (e.g., age, sex, race, socioeconomic status, geographic region) in externally controlled trials, as well as other attributes that are often more challenging to address, such as disease characteristics (e.g., severity, duration, specific signs and symptoms, performance status); prognostic or predictive biomarkers; comorbidities; and prior and current medical treatments. We agree that the protocol for an externally controlled trial should include specific plans for evaluating eligibility criteria to determine if the criteria can be applied in a manner that maximizes similarities between patients in the treatment and external control groups, recognizing the limitations of information available in many RWD sources. Equally important, the RWD on safety and efficacy must be similar in order to make meaningful comparisons. Electronic health records (EHR), billing data, and registry data lack many patient-reported outcomes that may be important,1,2 whereas registry data often fail to include confounding patient and medical center variables that were not evaluated but influenced outcomes.3
- Attributes of Treatment
The draft guidance recognizes potential imbalances between the treatment arm and the external control arm that are either not documented or cannot be evaluated (e.g., adherence, dose, timing of initiation, and duration of treatment). The guidance states, “specific concerns regarding missing data from RWD sources obtained as part of routine clinical practice can threaten the validity of the results of an externally controlled trial.” We agree with this statement, and it is also important to note that “Big Data” from billing records and EHRs frequently do not indicate whether patients took medication that was prescribed or complied with the doctors’ instructions for treatment.1,2 Additionally, when patients report adverse effects, especially those that are subjective and relatively common, that information is not always included in billing records or EHRs, making it difficult to accurately evaluate the frequency of those events.
- Assessment of Outcomes
The draft guidance recognizes the challenges created through the lack of blinding in externally controlled treatments, which potentially can lead to a biased estimate of the effect of treatment. Patients who are enrolled in an open label trial, and the physicians treating them, often report positive outcomes that can only be accurately evaluated when compared to placebo, not to historical controls. We fully agree that the “well-defined, reliable, and clinically meaningful outcomes that are typically used in randomized trials may be particularly difficult to ascertain and evaluate in an RWD source that is being considered for an externally controlled trial.”
There are additional limitations that should be considered when evaluating the validity of using RWD for externally controlled clinical trials. Access to information from registries is a major problem in the U.S., as most if not all registries are controlled by medical societies that limit access to the data. These societies decide what data will be collected, how the data will be analyzed, and which results will be made available to government agencies or other researchers. This is why section B under part IV, “Considerations to support regulatory review”, is an essential measure to ensure the submission of relevant patient-level data (i.e., all outcome and safety data for each participant or patient in the externally controlled trial) to the FDA, as required under FDA regulations,4 for both the treatment and external control arms. FDA suggests that sponsors who do not own the data used for the external control arm should structure their agreements with the data owners to ensure that all patient-level data can be provided to FDA in support of the marketing application. We strongly recommend that this measure be strictly enforced, because any limits on publicly available data (other than identifying data) could bias the results. Although results must be made publicly available on clinicaltrials.gov, research indicates that that information is often delayed for years after the study was scheduled to be completed.
Lastly, RWD from registries, EHR, and billing records typically include little, if any, patient-reported outcomes, such as pain or quality of life, or data on many of the adverse events that truly matter to patients. We strongly urge that the final version of the draft guidance be revised to be more explicit and specific in the ways we have recommended.
[1] Mourik, M., Duijn, P., Moons, K., et al. (2015). Accuracy of administrative data for surveillance of healthcare-associated infections: A systematic review. BMJ. https://bmjopen.bmj.com/content/5/8/e008424
[2] Lawson, E., Louie, R., Zingmond, D., et al. (2012). A Comparison of Clinical Registry Versus Administrative Claims Data for Reporting of 30-Day Surgical Complications. Annals of Surgery. 256 (6): 973-981. https://pubmed.ncbi.nlm.nih.gov/23095667/
[3] Nørgaard, M., Ehrenstein, V., & Vandenbroucke, J. P. (2017). Confounding in observational studies based on large health care databases: problems and potential solutions – a primer for the clinician. Clinical epidemiology, 9, 185–193. https://doi.org/10.2147/CLEP.S129879
[4] See 21 CFR 314.50(f) and 601.2.