NCHR’s comments on FDA’s Draft Guidance for Industry on Demonstrating Substantial Evidence of Effectiveness for Human Drug and Biological Products

National Center for Health Research: March, 19, 2020

National Center for Health Research’s Public Comments on FDA’s Draft Guidance for Industry on Demonstrating Substantial Evidence of Effectiveness for Human Drug and Biological Products
[Docket No. FDA-2019-D-4964]

Thank you for the opportunity to comment on the FDA Draft Guidance on Demonstrating Substantial Evidence of Effectiveness for Human Drug and Biological Products. The National Center for Health Research (NCHR) is a nonprofit think tank that conducts, analyzes, and scrutinizes research, policies, and programs on a range of issues related to health and safety. We do not accept funding from companies that make products that are the subject of our work.

The FDA’s draft guidance describes circumstances under which exceptions can be made to the requirement of two adequately controlled clinical trials. Although the law allows these exceptions under certain circumstances, we have concerns over the frequency with which these exceptions are made. In addition, the guidance names no requirements for conducting trials with adequate diversity, despite the known need for diverse sample populations.

The FDA guidance provides reasonable guidelines for reliance on one study instead of two. It states, “Reliance on a single large multicenter trial to establish effectiveness should generally be limited to situations in which the trial has demonstrated a clinically meaningful and statistically very persuasive effect on mortality, severe or irreversible morbidity, or prevention of a disease with potentially serious outcome, and with other characteristics described below, and confirmation of the result in a second trial would be impracticable or unethical. For example, conducting a second trial after a strongly positive trial had demonstrated a decrease in post-infarction mortality, or prevention of pertussis would generally present significant ethical concerns. Repetition of positive trials showing only symptomatic benefit would generally not present the same ethical concerns. In addition to the expectation that the single trial is large and multicenter, there should be no single trial site that is the main contributor to the observed effect, either by virtue of having a much bigger effect or many more patients than other sites; these characteristics help address concerns about bias and chance findings associated with a single trial. As noted above it would also be expected that the effect size on the primary endpoint and the statistical analysis results are both persuasive.

We agree with the above analysis, but unfortunately FDA has approved numerous drugs that do not meet those criteria:

  1. The most egregious was the approval of Exondys 51 for a type of Duchenne’s muscular dystrophy, which was approved based on surrogate endpoints studied on 8 patients who were compared to a “control group” of 4 patients. The trial became open-label after 6 months so that all control group patients were on Exondys. Since Sarepta had enrolled more patients after the initial study, the FDA should have waited until those study results were available, which would have enabled patients to receive this experimental treatment for free in a clinical trial. Instead, the FDA approved the drug, which was sold for $300,000-$450,000 per year for each patient, depending on their weight.
  2. FDA accelerated approval of Makena for prevention of preterm birth based on a single trial. After it had been sold for several years, a new, much larger clinical trial did not find a benefit, so in October 2019 an FDA advisory panel recommended pulling Makena from the market. FDA has not announced a decision in the 5 months since that recommendation, leaving physicians and patients uncertain about whether this expensive treatment provides any benefit to patients.
  3.  A 2.5mg dose of Empagliflozin was approved for improving glycemic control after only a single trial of very short duration, based on a small improvement on a surrogate endpoint. It is not clear that the small improvement in this one trial is clinically meaningful for patients, especially since the long-term risks and benefits were not studied. Moreover, 97% of the patients in the study were white, and the drug was approved for all races, despite concerns that the safety and effectiveness of the treatment could differ between racial/ethnic groups.
  4. In January 2020, an FDA advisory committee met to consider the approval of the opiate Oxycodegol. Although two trials were conducted, one trial failed to demonstrate efficacy, and the success in the other trial was questionable. Most worrisome, the proposed indication was for chronic lower back pain, which experts now agree should not be treated with opioids because of the questionable benefits and the clear risks of addiction. The advisory committee voted 27-0 against approval, and the company withdrew its application the next day. However, the important question is why was the FDA considering approval of this drug seriously enough to hold a public Advisory Committee meeting?

While non-inferiority studies can be a valuable tool in some circumstances, they create a major problem for patients and the physicians that treat them. When clinical trials use larger non-inferiority margins or margins equal to the estimated treatment effect, there is a much higher likelihood that a new product is less effective than the one that is already on the market. Inferiority can increase during each subsequent comparison of a new drug to a slightly older drug, with each new drug somewhat worse than the drug it was compared to. This could result in new drugs on the market that are substantially inferior to older drugs. Even a non-inferiority margin of 10% means that 1 in 10 patients may receive a new, less effective drug than the older therapy. Non-inferiority trials are common for new antibiotics, which is not only harmful to patients but also raises additional concerns because less effective antibiotics may contribute to antibiotic resistance, potentially leaving patients without effective treatments. The information about comparative effectiveness of the new drug to the most effective older drugs are not available to patients. To add to that lack of transparency, new treatments tend to be widely advertised and more expensive, promoted in ways that imply superiority even when they are actually inferior.

Surrogate endpoints that are “reasonably likely” to predict a clinical benefit enable clinical trials to be completed more quickly, but cannot predict benefit if they are not well-validated. Too often, positive outcomes for these surrogate endpoints are not resulting in clinically meaningful benefits. For example, Benlysta helped some patients with lupus get rid of their rash, which was thought to indicate overall improvement, but it also left them susceptible to severe infection or even death. Numerous drugs tested for Alzheimer’s disease seemed effective when measuring beta amyloid plaques in the brain, but were found to be ineffective or even harmful when memory was the outcome measure.1 A surrogate endpoint is appropriate only if it accurately predicts the outcomes patients care about, such as their symptoms, quality of life, and how long they live.

Even if a treatment is tested using two well-controlled clinical trials, a lack of diversity in the trials may lead to insufficient evidence regarding the safety and effectiveness of the treatment for minority populations. Racial and ethnic minorities are frequently underrepresented in studies submitted to FDA.2, 3 Those from minority populations are therefore being prescribed treatments that have not adequately been tested and shown to have a favorable benefit/risk profile for them. It is necessary to include sufficient numbers of people representing minority populations in clinical trials and to also conduct subgroup analyses that determine the safety and effectiveness for those specific populations.

1. Zuckerman Diana M, Jury Nicholas J, Silcox Christina E. 21st Century Cures Act and similar policy efforts: at what cost? BMJ. 2015; 351 :h6122
2. Fox‐Rawlings, S. R., Gottschalk, L. B., Doamekpor, L. A., & Zuckerman, D. M. (2018). Diversity in Medical Device Clinical Trials: Do We Know What Works for Which Patients?. The Milbank Quarterly, 96(3), 499-529.
3. U. S. Food and Drug Administration. FDASIA Section 907: Inclusion of Demographic Subgroups in Clinical Trials. 2018.