FDA has been mulling over reliance on real-world evidence for about a decade, keeping it on the edges of the existing approval system that runs on data from conventional clinical trials.
Three years ago, RWE was used for an expansion of one indication, from breast cancer in women to breast cancer in men. But the agency has not publicly acted on an application based entirely on RWE.
This changed on Oct. 28, when the agency asked its Oncologic Drugs Advisory Committee for a recommendation on an application based on RWE. The meeting set a twofer of precedents—(1) the first known RWE-based application and (2) the first such application to go to ODAC.
ODAC has been known to have difficulty with concepts that come before it. Example: even 30 years into the accelerated approval program, the committee needs an occasional reminder about the standard of evidence, i.e. the meaning of the words “reasonably likely to predict” benefit to patients.
By contrast, on Oct. 28, the clinical experts demonstrated remarkable ease in scrutinizing an RWE-based study design—and unanimously voted down the application that was placed before them.
The sponsor—Y-mAbs Therapeutics Inc.—was claiming that its agent, 131I-omburtamab, extended overall survival in the treatment of central nervous system/leptomeningeal metastases in pediatric patients with neuroblastoma following standard multimodality treatment for CNS disease.
Here, speed-readers would be advised to scroll back to the paragraph above.
This is not a typo: Yes, Y-mAbs collected the data in a single-arm study and was claiming overall survival benefit! Again, the sponsor wasn’t claiming an increased response rate, but, yes, OS. And Y-mAbs was seeking traditional—as opposed to accelerated—approval.
But, one might point out, in the past, a person claiming to have divined OS from a single-arm study might as well have been reporting regular visitations from Archangel Michael. FDA sometimes accepts response rate data from single-arm trials, because tumors don’t shrink by themselves.
OS, by contrast, requires a comparator. It’s one of the fundamentals of the pre-RWE world.
So, how could Y-mAbs claim to have measured OS in a single-arm study and seek traditional approval for doing so?
Y-mAbs placed a gutsy bet on the new methodology. The company did have a control arm. It chose a control arm, compiling it from the Central German Childhood Cancer Registry. In principle, “synthetic” or “external” control arms can be used in RWE trials, although this should be done very, very carefully.
Since the agency’s stance on RWE has never been tested, the company’s aggressive strategy wasn’t entirely foolhardy.
There were a lot of unknowns:
Would FDA be more inclined to accept uncertainty in an ultra-rare pediatric indication? Would the agency feel compelled to throw the application to ODAC, if only to avoid political fallout that could follow the tossing out a pediatric indication? Would ODAC be sufficiently savvy to interpret the RWE data, or would it roll the dice to give young, desperately ill patients a treatment option?
Losing is part of the nature of a bet, and the Y-mAbs wager resulted in unambiguous defeat.
Though ODAC that day consisted largely of pediatricians—some of whom had little or no experience on the committee—they had no problems delving into the novel aspects of the application and reaching a unanimous conclusion.
It’s not publicly known whether FDA has offed other RWE-based applications on the staff level, perhaps by discouraging sponsors. But we do know that three years ago, the agency acted on RWE when it expanded the indications of Pfizer’s aromatase inhibitor Ibrance (palbociclib) to include men (The Cancer Letter, April 19, 2019).
FDA relied on data from electronic health records and post-marketing reports of the real-world use of Ibrance in male patients sourced from three databases: IQVIA Insurance database, Flatiron Health Breast Cancer database and the Pfizer global safety database.
The agency didn’t consult ODAC before granting this indication.
The 131I-omburtamab application came to the committee at a time when FDA, NCI, and the oncology profession, are devising standards for RWE and pragmatic clinical trials in oncology (The Cancer Letter, Nov. 22, 2019; Sept. 25, 2020).
At NCI, Director Monica Bertagnolli, a former group clinical trials group chair and an expert in the evolving role of bioinformatics and the electronic medical record, is certain to focus on “modernizing” the institute’s clinical trials infrastructure.
And at this important moment in history, Y-mAbs has produced a map of roads best not taken. Any drug developer who contemplates submitting an RWE-based application should be required to print out every publicly available page in the 131I-omburtamab docket, sharpen a No. 2 pencil, and delve into the subtleties of mistakes that were made.
Comparator arm imported from Germany
The indication sought by Y-mAbs is much smaller than male breast cancer, which strikes a bit more than 2,500 men in the U.S. a year.
Around 650 cases of neuroblastoma are diagnosed every year in the U.S. About half of them are high-risk neuroblastoma, and approximately 6% of patients with high-risk neuroblastoma who experience metastatic relapse have metastases to the CNS or LM.
131I-omburtamab is an iodine-131 radiolabeled murine monoclonal antibody that binds to the B7-H3 antigen expressed on the surface of neuroblastoma tumor cells. The agent is administered as an intraventricular infusion using an intracerebroventricular access device.
The primary efficacy data supporting the Y-mAbs application were derived from a single-center investigator-initiated single-arm study titled “Phase I Study of Intrathecal Radioimmunotherapy using 131I-omburtamab for Central Nervous System/Leptomeningeal Neoplasms,”or Study 03-133. The study was conducted entirely at Memorial Sloan Kettering Cancer Center.
Since response rate data weren’t collected, Y-mAbs had no Plan B—only Plan A.
The efficacy population consisted of a subset of 94 pediatric patients ages 0.9 to 13 years with CNS/LM relapsed neuroblastoma over 14 years between 2004 and 2018. The primary endpoint was OS at 3 years.
Prior to 131I-omburtamab treatment, all patients received at least one type of CNS-directed therapy, and the majority of patients (76%) received all three treatment modalities. The 3-year OS rate after CNS/LM relapse in the efficacy population of 94 patients was 54% (95% CI: 0.43, 0.64).
The OS results were compared with an external control arm that was constructed from a registry that includes data from patients with stage IV neuroblastoma included in the German national neuroblastoma clinical trials NB90, NB97 and NB2004 from 1990 to 2015. Y-mAbs identified 79 patients in the source population who received at least one type of post-CNS relapse treatment.
The application states that 131I-omburtamab prolongs survival vs. the external control arm, with an estimated hazard ratio of 0.58 (95% CI: 0.31, 1.09), log-rank test p-value=0.0544.
“Albeit not statistically significant, the 42% reduction in the risk of death with 131I-omburtamab observed in the primary analysis is considered clinically meaningful,” the sponsor said in the application. “In addition, an improved 3-year OS rate by 23% and a clinically relevant OS benefit exceeding 2 years was observed with 131I-omburtamab treatment.”
The application also included interim data from an ongoing international multi-center single-arm trial, Study 101, focused on 131I-omburtamab in pediatric patients with neuroblastoma with relapse in the CNS including parenchymal or LM metastases.
The primary endpoint of the trial is 3-year OS rate, with a key secondary endpoint of overall response rate. Study 101 is fully enrolled, but survival data remain immature.
Era-of-treatment bias
The FDA briefing document acknowledges that randomized trials, while preferable, aren’t always feasible.
“Randomized studies minimize the effect of both known and unknown differences between populations by providing a direct outcome comparison,” the briefing document states. “Despite the significant limitations associated with interpretation of OS in non-randomized settings, there may be contexts where a randomized trial is difficult to conduct. For serious rare diseases with unmet medical need, there is interest regarding use of an external control in which all enrolled patients receive the investigational drug without randomization to a concurrent comparator group.”
What concerns me the most, and what I just can’t get past, is the comparator group not using craniospinal radiation versus the group that has craniospinal radiation.
Tobey J. MacDonald
However, the document points to several fundamental areas of concern.
The patients in the German database were not treated contemporaneously and differed substantially from those in the MSKCC study, the agency pointed out.
This difference in timing of care, known as “the era-of-treatment bias,” can occur as staging or treatment approaches evolve, making it problematic to compare more recent datasets with those that contain data that reflect old staging and treatment standards.
Prior treatments received by the MSKCC patients were more intensive than those in the external control, and data indicate that OS in patients with CNS/LM neuroblastoma has improved over time, the agency said.
Usually, a mismatch between the experimental arm and controls is enough to sink an application. However, the agency performed additional statistical analyses, concluding that differences in survival cannot be reliably attributed to 131I-omburtamab.
Since ORR data were not collected in the MSKCC study, the application doesn’t provide supportive evidence of the treatment effect of 131I-omburtamab, the agency said. Also, according to FDA, no patient in Study 101 demonstrated a response that can be unequivocally attributed to 131I-omburtamab.
Confounders—measured and unmeasured
FDA asked ODAC to discuss the following questions:
- Discuss if real-world data (RWD) from the Central German Childhood Cancer Registry are appropriate for comparison of overall survival with Study 03-133, taking into account the known differences in the study populations and other potential sources of bias.
- Discuss whether additional information is needed to assess the benefit of 131I-omburtamab for neuroblastoma with CNS/LM relapse.
The highlights of the discussion follow:
Professor of pediatrics, Division of Pediatric Hematology Oncology, University of Washington School of Medicine, Seattle Children’s Hospital
I think that there was a very thorough discussion about the confounding differences between the external control group and the experimental group.
And I think one area that we did not really delve into is the likely significant difference in the upfront treatment that these patients received as well, highlighting the importance of the era of treatment and how it impacts the aggressiveness with which the neuroblastoma community approached relapsed neuroblastoma.
That has changed greatly over time with advent of newer therapies. I think all of those are extremely important, confounding effects [that] really highlight the importance or limitations of statistical analysis.
Aflac Endowed Chair for Pediatric Neuro-Oncology, Professor of pediatrics, Director, Pediatric Neuro-Oncology Program, Aflac Cancer & Blood Disorders Center, Children’s Healthcare of Atlanta, Emory University School of Medicine
As a pediatric neuro-oncologist, what concerns me the most, and what I just can’t get past, is the comparator group not using craniospinal radiation versus the group that has craniospinal radiation.
We know from other malignant diseases, primary to brain, medulloblastoma, ependymoma, that in metastatic disease, craniospinal radiation is much more effective at controlling the disease and improving survival, than focal radiation.
We don’t even know the dose of the focal radiation given in the other group. So, to really make any comparison, to me, is impossible from that standpoint alone.
Second, tumor responses. We know that radiation can have a long-term effect, and that responses may be delayed, and seen on imaging months later—three months, six months later—you can actually see responses occurring with radiation alone.
So, to me, that is the absolute obstacle point in trying to detect response data as well as survival data.
Professor of Biostatistics (emeritus), Dana-Farber Cancer Institute;
Harvard T.H. Chan School of Public Health
What I would like to acknowledge upfront is that it’s very rare when you are using uncontrolled studies against observational controls that you can isolate a treatment effect. For me, when I look at the comparison of a study with historical controls, what I look for is a certain robustness in the analysis, and by that I mean, if you go at it several different ways, do the results hold up?
And I think that what we’re seeing here is that approaches taken by the sponsor and approaches taken by the FDA can lead to very different conclusions here.
And those are how you adjust for initial treatment, and coming down to very small sample sizes. Of course, I have to balance this against using this in a rare disease, which would preclude having a large, observational database for control.
But for me, there are just too many differences in the way one looks at the study to be comfortable that there’s a plausible establishment of an important association with the administration of the drug.
Professor and associate chair, Department of Biostatistics, Gillings School of Global Public Health, University of North Carolina at Chapel Hill
I agree with comments that were made by others on the panel that there’s a lot of uncertainty in these data and the way it’s analyzed. I see two major concerns. Whenever we do these observational data analyses, we worry about adjustment for confounding.
We worry about unmeasured confounders, but we even are worried about measured confounders. We know calendar time is a confounder here. We know that the treatment that the controls and the treated have received are different. And there’s a lack of overlap or, or a lack of positivity related to some of these measured confounders.
And then there’s the immortal time bias that folks have talked about, and I don’t want to harp on that, but I do want to address the second part of the discussion, which is this question about what additional data is needed.
And one thing that I think might be helpful is an analysis that tries to emulate a target trial, to say what’s the randomized trial we would like to do, but we couldn’t do, that would articulate very carefully what the eligibility criteria would be for that trial, what the different regimens would be that would be compared, what would be the control arm, what would be the treatment arm, what would be time zero—to spell all those things out, and then use the German registry data, whatever data we have available, as best we can to analyze those data in a way that’s consistent with this trial emulation idea.
(Acting chairperson)
Associate professor of medicine, Associate director for clinical research, Co-director, Gastrointestinal Medical Oncology, University of Colorado Cancer Center Aurora
Thank you for that discussion. I’d like to summarize what we just discussed over the last 10 minutes, and that is a fairly consistent theme across the discussion regarding measured and unmeasured confounders.
And so, regarding overall survival data, there’s concern from the panel regarding the era of treatment and how treatment has changed [to] a more modern approach. Also, there is significant concern regarding cerebrospinal irradiation and how that may impact data.
There is also expressed concern regarding response rate data and confounders to potential responses as well as a desire from the panel to see more robust data in an analysis that would be slightly more consistent, but would certainly include additional patients.
And there was comment in regard to possible pathways forward to what additional data could be helpful; comments made from Dr. Hutchins and others in regard to a more clinical trial-like dataset to be able to compare a control arm, even utilizing real-world data, compared to the treatment arm.
