Ten health care research organizations, with help from FDA and NCI, have developed a set of common definitions for real-world endpoints, including overall survival, progression-free survival, and other non-traditional endpoints.
The new common definitions are published as part of a pilot study led by Friends of Cancer Research, which announced the conclusions of this phase of the project at a recent gathering in Washington, D.C.
At the Sept. 18 event, the 8th Annual Blueprint for Breakthrough Forum, a speaker nicknamed the collaboration “Frenemies of Cancer Research.” The suggestion set off a wave of loud, albeit nervous laughter, because the joke was on the nose—to collaborate, many of these companies had to set aside their competitive agendas, which made for an uneasy peace.
The companies that participated in the Friends Pilot Project 2.0 are: Aetion, CancerLinQ, Concerto HealthAI, COTA, Flatiron Health, IQVIA, Kaiser Permanente, OptumLabs, McKesson Life Sciences, Syapse, and Tempus.
The Friends effort is central to realizing one of the primary mandates within the 21st Century Cures Act of 2016, which requires FDA to consider using real-world evidence to complement and supplement data generated through traditional clinical trials in drug regulation.
The project has one especially important patron. “FDA was instrumental in providing expertise throughout the entirety of the project, including its development,” Jeff Allen, president and CEO of Friends, said to The Cancer Letter.
Most of the companies involved in the Friends effort regularly compete against each other for grants, access to health systems, and funding from pharmaceutical companies that are plotting strategy in the new RWE world.
The stakes are high:
As FDA continues to approve immunotherapies and targeted therapies that may have broad application across disease types, academic institutions, professional associations, as well as Big Data and Big Pharma, are vying to acquire patient data and put it to commercial use.
And at the heart of the business is the core lexicon—standard, agreed-upon definitions for real-world endpoints.
With FDA guiding the creation of the pilot methodologies and definitions in the Friends endeavor, data companies and research organizations that participate gain a real advantage—by putting their thumbprints on the process, they shape the development of these endpoints and definitions, perhaps ensuring that these elements correspond with the strengths of their respective data sets.
The Cancer Letter wanted to know how the collaboration was structured in a systematic way, which problems were being solved, and what are the questions that have yet to be answered.
As they jostle for prominence, are these companies creating an RWE equivalent of the Tower of Babel, even as they claim to be finding a common tongue? |
To gain a deeper understanding of how these disparate groups are working together while continuing to compete with each other, The Cancer Letter presented the leadership of 10 companies with the same set of questions. These include:
What is your organization’s business model, and how is your work unique?
What are your takeaways from the Friends project?
How did you address issues of data quality and transparency in this project?
Will you participate in phases three and four of the project? What are the next steps?
How would you describe your organization’s RWE portfolio?
What will you do with these endpoints should FDA accept the final definitions?
Is it possible for you to collaborate with competitors after the Friends project is complete?
As the companies discuss the future of data sharing in cancer research and their individual projects, The Cancer Letter found broad consensus on the applications of the Friends framework, the necessity of collaboration in data research, and what it would take to make FDA comfortable with using real-world endpoints in regulatory decision-making.
Their responses appear here.
“Ultimately, the results of [the Friends project] will be informative to oncologists and patients by filling evidence gaps about the performance of medical products used in a real-world setting, including populations that may not have been represented in clinical trials,” Allen said.
“It has also helped to characterize how several metrics that are readily obtained from electronic health data (such as time-to-treatment discontinuation) correlate to more traditional clinical measures like tumor progression or survival.”
A conversation with Allen appears here.
Which patient populations and disease subtypes are being studied in the Friends collaboration? |
The project focuses on patients with advanced non-small cell lung cancer who received immune checkpoint inhibitors.
This second phase builds on earlier work in 2018, which previously concluded that it was possible to identify a high level of shared characteristics across varying data sets—demonstrating that it is feasible to extract data about specific patient populations from disparate sources of data.
“It was amazing to see [the Friends effort] moving from Pilot 1.0 to Pilot 2.0, moving from six data partners to 10 data partners,” said Amy Abernethy, principal deputy commissioner and acting chief information officer at FDA. “The idea of collaboration, on a scale like this, with the speed to which these projects got done is pretty remarkable.
“What happened is, in doing the project about endpoints, they exposed a whole bunch of other issues such as definitional issues, and differences in the source data systems, etc.,” Abernethy said Sept. 18 at the Friends Breakthrough Forum. “And so, one of the things that’s really important about doing this work is starting in one place exposes a whole bunch of other things that are important to work our way through.”
How does the Friends project fit into FDA’s priorities for building a regulatory infrastructure for RWE? How do studies based on RWE differ from traditional clinical trials? |
Last December, FDA published a framework for evaluating the use of RWE, which lays out the fundamentals of the agency’s approach to developing guidances for using real-world data in drug regulation.
Future guidances will focus on trial designs using real-world data as well as assessment of the reliability and relevance of real-world evidence in describing drug effectiveness (The Cancer Letter, Jan. 4).
“One of the first things to say about real-world evidence is that it’s already here. FDA is already using RWE in certain areas,” said Ned Sharpless, then FDA acting commissioner, at the Friends Breakthrough Forum. Sharpless returned to NCI as director earlier this month (The Cancer Letter, Nov. 8).
“The FDA, for its part, has to, in some ways, upgrade our infrastructure to be able to be a better partner for industry,” Sharpless said at the Sept. 18 event, addressing the agency’s Technology Modernization Action Plan (The Cancer Letter, Sept. 20). “FDA is going to try and upgrade its information-handling infrastructure to be a better partner for using things like real-world evidence and other kinds of data.”
Unlike traditional clinical trials, in which patients are enrolled based on eligibility criteria, and information is collected according to parameters set in prospective trials designed to evaluate conventional endpoints, real-world studies use existing historical and real-time data—captured from electronic health records, claims data, and retrospective population-level data—to extract evidence that could then be evaluated in synthetic arms.
What are the objectives of the Friends study? And how are the real-world endpoints defined? |
In Pilot 2.0, the Friends collaboration developed common definitions for endpoints, based on the description of advanced NSCLC patients in real-world data sets. The objectives for phase two of the study were:
Describe demographic and clinical characteristics of patients with advanced NSCLC receiving frontline chemotherapy doublet, PD-(L)1 monotherapy, or PD-(L)1 + doublet chemotherapy—to provide baseline understanding of the similarities and differences among the datasets to better understand the confounding factors that may need to be considered when interpreting the data.
Evaluate treatment effect size in frontline therapy regimens using real-world endpoints—so that researchers could agree on data source-specific definitions and measurement of endpoints assessed through real-world data in order to ensure reliability, consistency, and conservation of clinical meaning.
The Friends project concluded that the myriad data organizations were able to reach “high-level alignment” on important data elements and definitions for real-world endpoints in the context of a focused research question, despite variation in the underlying sources of data.
“This effort showed that it’s possible for real-world oncology data organizations to align on considerations for identifying patients across diverse types of sources, from claims-based datasets to EHRs,” Nicole Mahoney, senior director of regulatory policy at Flatiron Health, said to The Cancer Letter. “We were also able to align on high-level definitions for real-world endpoints, and identify important data elements that need to be collected in order to help answer a specific clinical question.”
The four common definitions used—and published—by Friends are:
Real-world Overall Survival (rwOS): Length of time from the index date to the date of death, or disenrollment (need to define gap in enrollment). For claims data, health plan disenrollment date is incorporated if deaths are not captured among those who leave health plan coverage.
Real-world Time to Next Treatment (rwTTNT): Length of time from the index date to the date the patient received an administration of their next systemic treatment regimen or to their date of death if there is a death prior to having another systemic treatment regimen.
Real-world Time to Treatment Discontinuation (rwTTD): Data: Length of time from the index date to the date the patient discontinues frontline treatment (i.e., the last administration or non- cancelled order of a drug contained within the same frontline regimen). Discontinuation is defined as:
Having a subsequent systemic therapy regimen after the frontline treatment;
Having a gap of more than 120 days with no systemic therapy following the last administration; or
Having a date of death while on the frontline regimen.
Real-world Progression Free Survival (rwPFS): Length of time from the index date to the date of a real-world progression (rwP) event (i.e., distinct episode in which the treating clinician concludes that there has been growth or worsening in the aNSCLC based on review of the patient chart) at least 14 days after frontline treatment initiation, or death.
The full definitions of these real-world endpoints are available here.
Did the Friends study conclusively determine whether the survival outcomes and treatment patterns for advanced NSCLC patients—generated based on these endpoint definitions—are similar throughout real-world data sets? |
At first glance, some of the Kaplan-Meier curves for these endpoints appear to be visually “tight,” as if to suggest that patient survival outcomes, for instance, may be similar across data sets.
However, it’s too early to derive formal conclusions regarding the performance of these treatments in real-world settings or demonstrate that real-world endpoints are accurate proxies for conventional clinical trial endpoints.
“The Friends pilot may provide an opportunity to discuss how the underlying quality of specific data elements may impact the outcomes we observe,” Mahoney said. “For example, date of death is not always captured in real-world clinical settings. Given that incomplete information on death can skew overall survival analyses, data organizations have to link or supplement information with external sources.
“The impact of incomplete death data highlights the importance of benchmarking it to the gold standard, which is the National Death Index, to generate quality metrics, such as sensitivity.”
Validating these endpoints will require stratification of patient populations by demographic characteristics as well as by PD-(L)1 status to compare outcomes, and additionally, subsequent benchmarking of these outcomes against clinical trials.
“We will be pursuing this and looking into other questions, like using the framework in other disease settings, and applying clinical trial inclusion/exclusion criteria to the real-world populations to validate endpoints over the coming months,” Allen said. “While our goal isn’t to make real-world studies mirror clinical trials, this may be an important internal validation step to increase confidence in the data quality and conclusions being drawn from a broader real-world dataset.”
What role did NCI play in the study? |
The Friends collaboration also includes patient information from SEER, a rigorously curated population-based data set.
“We felt like being a part of this pilot was important to understand real-world evidence and really understand how we can produce analytics on emergent therapies, obviously, the project focuses on immunotherapies,” said Donna Rivera, a scientific project officer in the Surveillance Informatics Branch within the Surveillance Research Program of the Division of Cancer Control and Population Sciences at NCI.
“I think our dataset is the only population-based cancer-specific data set. SEER-Medicare is comprised of data from both SEER, which is 16 population-based cancer registries covering 34.6% of the U.S. population linked with Medicare claims data,” Rivera said to The Cancer Letter. “Data quality studies are conducted, and, importantly, within SEER is the categorization of tumor data. So, the site, histology, laterality, grade, these categorizations of the tumor—the way we code our definition for our data is different than some of the ways the other groups are using it, because they don’t have the same level of detail in their data.
“The vast amounts of data in this analysis require a lot of collaborative discussion on data elements, because the elements are coming from different places—we have claims data, but there’s certainly many other companies within this pilot that also have EHR data. Some people have structured fields, unstructured fields,” Rivera said. “Alignment on definitions, understanding the statistical analysis, even cohort variation, all of these things, I think, are fundamental.
“So, differences in age, differences in stage, PD-(L)1 testing, smoking as a smoking status. Those really have to be contextually understood by each dataset when evaluating outcomes.”
How is RWE used in innovative trial designs as our understanding of the genetic underpinnings for cancer continues to evolve? |
Increasingly, cancer researchers are exploring the uses of RWE in hybrid and pragmatic trial designs—and within master protocols or adaptive trial designs—to evaluate the treatment effect of immunotherapies and targeted therapies.
“Precision medicine presents a substantial challenge to the current clinical development model: as patients are categorized into smaller and smaller cohorts based on molecular and clinical criteria, it will become difficult to perform RCTs for every drug-molecular-clinical indication due to lack of patient availability and high costs,” Jonathan Hirsch, founder and president of Syapse, said to The Cancer Letter.
A scenario that is growing in importance involves the use of RWE to support granting of an expanded indication for a drug already approved in another indication on the basis of randomized clinical trial data, Andrew Norden, chief medical officer of COTA, said to The Cancer Letter.
“A recent example was the approval of Ibrance (palbociclib) for male breast cancer, which relied on multiple types of RWE against the backdrop of RCT data previously generated for breast cancer in women,” Norden said (The Cancer Letter, April 19). “A related application involves the creation of an external control group from RWD. Imagine that a new drug is being developed to target a novel mutation in patients with highly refractory solid tumors.
“In this case, patients are unlikely to accept randomization to an existing standard of care—which is associated with poor outcomes—and oncologists have ethical concerns about randomization because some evidence of unusual activity has been observed during a phase I study.
“Therefore, the sponsor initiates a single arm phase II study with the blessing of the FDA. In this circumstance, the control group may be selected from a robust RWD set. Robustness is important because of the requirement to match prognostic factors between the experimental group and the RWD-derived control group as closely as possible.”
In structured format, real-world data is a powerful tool that can be used to rapidly understand whether subpopulations of patients are responding to drugs that aren’t indicated for their disease, whether patients with rare and potentially actionable mutations exist, and how well a drug performs in real-world patients that may be less healthy and older than those accrued to a clinical trial.
“I think that for organizations and regulatory bodies to trust real-world data more, it’s not necessarily certified using ‘the entirety of the dataset,’ but ensuring the dataset you’re using for a particular analysis is fit-for-purpose,” Sarah Alwardt, vice president of data, evidence and insight operations at McKesson Life Sciences, said to The Cancer Letter. “So, fit-for-purpose was definitely the phrase that we heard a lot, and making sure that everything that was outlined in the FDA framework for real-world evidence around general reliability, quality, and transparency are achieved, but starting to get into, ‘What does that actually mean?’ and ‘How is that to be defined?’”
What does it take to curate RWE that is not only fit-for-purpose, but also fit for submission to FDA and other regulatory agencies? |
To mine real-world data that isn’t readily structured, many data companies invest heavily in large teams of quantitative experts, data scientists, software engineers, programmers, and support staff.
The objective is to abstract information from patient records in a reliable manner without introducing errors and data artifacts as well as design programs and application interfaces that can both receive structured data and enable analyses, and ensure that the curated data is high quality and sufficiently complete for use in studies.
“In order to fully realize the potential of research and analytics in [precision and cancer], we also need to work toward better data availability to address questions in these areas—and this points to another area of data limitation currently,” Lawrence Kushi, director of scientific policy in the Division of Research at Kaiser Permanente Northern California, said to The Cancer Letter. “One key part of this is being able to identify readily, in a structured format, people who’ve been tested for specific clinical genetic tests, and the results of those tests. For example, PD-(L)1 testing or EGFR testing—relevant to this specific project—are being done to guide clinical decisions.
“When these gaps in data availability are solved, then any future guidance on generating real-world evidence from the FDA would be easier to follow.”
To use RWE in a consistent and meaningful way to inform regulation of cancer drugs—whether for new indications, or to confirm clinical benefit in the post-market setting—the data needs to be organized to answer specific questions rooted in standard definitions for real-world endpoints.
“Through these projects, we can develop and implement common endpoints across different real-world data sources,” Robert Miller, medical director of the American Society of Clinical Oncology’s CancerLinQ, said to The Cancer Letter. “We have demonstrated that the different sources of data can yield fairly similar results regarding patient outcomes.
“Another unique aspect is that this project is exploring non-traditional endpoints, such as time to next treatment and time to treatment discontinuation, that do show promise as potential alternate clinical endpoints to progression-free survival and others commonly used in clinical trials. TTNT and TTD may provide more clinically relevant endpoints because they are related to reasons that patients and clinicians alter clinical care, taking into account toxicity, efficacy, and other factors.”
Regulatory agencies in other countries are also developing frameworks for using RWE, said Nancy Dreyer, chief scientific officer and senior vice president of IQVIA Real-World Solutions Center for Advanced Evidence Generation.
“Once there is clarity about the evidentiary requirements for the FDA and other regulatory bodies, drug companies and medical product developers will feel more confident about using real-world evidence to supplement their regulatory applications for new indications and label expansions,” Dreyer said to The Cancer Letter. “IQVIA is also helping regulators in Europe, Japan and China develop guidance documents. All these regulators are calling for health stakeholders to share their experiences using real-world evidence, including pilot projects that will inform the development of formal guidelines.”
Researchers and companies are already able to use real-world evidence in many settings, even without specifically established guidance, said Jeremy Rassen, president and chief science officer at Aetion.
“Global regulators and value assessment bodies are increasingly incorporating RWE into their decision-making, and letting us know—in ongoing discussions and through public documents—what works and what doesn’t,” Rassen said to The Cancer Letter. “You can glean a few meta-themes from Pilot 2.0 and the related white paper: first, the importance of collaborative work among stakeholders including sponsors, data holders, analytic experts, regulatory agencies, and groups like Friends.”
Is transparency in data needed? What happens once FDA finalizes the definitions for real-world endpoints? |
As RWE is increasingly used to support regulatory approvals, some experts and patient advocates have called for greater transparency and a demonstration of replicability of results based on real-world data.
“Tempus agrees that transparency is critical to ensure confidence in RWE results, and the importance of investigating fit-for-purpose, quality of underlying data along with any variability in the population characteristics and/or methodological assumptions made during the analysis,” Gary Palmer, chief medical officer of Tempus, said to The Cancer Letter.
“When the FDA issues the final guidance for RWEndpoints, collaborations between data organizations or pooling data (rather than analyses) from them become more achievable. We will still need to investigate the fit-for-purpose of each contributing data organization or dataset, as well as address the unknown level of overlap between them.”
With formalization, the industry will have even more confidence and clarity as to where and how real-world data can aid pre- and post-approval decisions, said Mark Walker, chief scientific officer of outcome science and services at Concerto HealthAI.
“It further is aiding the generalizability of regulatory intent studies to the treatment decisions of community practitioners—a further goal of this move towards real-world evidence being integral to different study phases and decisions,” Walker said to The Cancer Letter. “Different data sources can yield similar patterns of findings across endpoints—and that is what we saw with the Friends of Cancer Research project.
“The value here is that we can achieve insights into specific populations or diseases, across different data sources that are comparable—thus increasing the confidence and the utility of those different data sources alone or in combination.”
There will be a need for more, not less collaboration, as the field matures, said CancerLinQ’s Miller.
“The FDA framework may define regulatory endpoints for RWD, but there are still a lot of unanswered questions,” Miller said. “There will continue to be conflicting interests, but based on this experience, I believe there is an opportunity to work together and collaboratively explore unanswered questions about real-world data quality, new endpoints, comparison with trials, and a host of other methodologic issues. ASCO is highly interested in continuing to be involved in this type of exploration.”
Companies should be able to continue to find areas in real-world research where their interests may align, McKesson’s Alwardt said.
“Of course, the easiest thing is if all data are free and freely available, and we all move forward. That is the most unlikely to happen,” Alwardt said. “So, I don’t think we’re there yet with this, but I do believe that there is an opportunity for us to find a way forward and common ground with the data that both protects the individual value and the perception of value for the individual companies, but still be able to provide really good workable datasets for regulators and to be able to continue to make good decisions for us.
“This was definitely a first step in thinking about how different our data sets are across a number of organizations, and where we can start to find commonality to be able to use these data for the benefit of patients.”
