FDA has created a framework for evaluating the use of real-world evidence to support additional indications for already approved drugs as well as to satisfy drug post-marketing study requirements.
The framework lays out the fundamentals of the agency’s approach to developing guidances for using real-world data in drug regulation.
The report, published in December, as required by the 21st Century Cures Act, applies to drugs and biological products. It doesn’t apply to medical devices.
In addition to staking out a swath of the burgeoning field of real-world evidence, the agency has recruited Amy Abernethy, an expert in generating and applying real-world evidence, to the job of principal deputy commissioner, making her the second-highest ranking official at FDA.
“First of all, if you haven’t read [the framework], it’s really worth reading,” said Abernethy, who is now the chief medical officer, chief scientific officer and senior vice president of oncology at Flatiron Health. She is expected to start at the agency in February 2019, after leaving Flatiron.
A conversation with Abernethy appears here.
“I think that the major takeaways [from the FDA framework] are consistent with what we were expecting before, which is that there’s a move towards trying to understand how to use real-world evidence,” Abernethy said to The Cancer Letter. “And that move is asking, ‘How do we make sure that the data quality and the analytic quality is adequate to address the kinds of questions that are going to be brought before the agency?’ Really focusing on quality is going to be critical.”
Over the past three to five years, pharmaceutical and health technology companies have invested heavily in real-world evidence—hiring hundreds, if not thousands, of data scientists and funding the development of regulatory-grade data processing systems capable of aggregating and annotating electronic health records and billing data.
In a move viewed as industry-changing in health technology, Roche Pharmaceuticals in February 2018 acquired Flatiron for $2.1 billion—signaling broad interest in the application of machine learning and natural language processing technologies in generating real-world evidence (The Cancer Letter, March 2, 2018).
The applications for real-world evidence, particularly in oncology, are wide-ranging. Researchers can track adverse outcomes as well as the uptake and off-label use of drugs in real time, capture patient-reported outcomes, and study drug effectiveness in multiple indications in diverse patient populations.
“The American Society of Clinical Oncology commends the FDA for taking this important first step in developing a comprehensive framework for advancing medical care with real-word evidence,” ASCO Chief Medical Officer Richard Schilsky said to The Cancer Letter. “Our research collaboration between CancerLinQ and FDA continues to inform this framework on the relevance and quality of real world data. We appreciate the opportunity to provide comments on the framework and will do so after we have reviewed it in detail.”
Per the Cures Act, real-world evidence is defined as “data regarding the usage, or the potential benefits or risks, of a drug derived from sources other than traditional clinical trials” (The Cancer Letter, Dec. 16, 2016).
Evaluating real-world evidence in the context of regulatory decision-making depends not only on the evaluation of the methodologies used to generate the evidence, but also on the reliability and relevance of the underlying real-world data.
For the purposes of the framework, FDA defines RWD and RWE as follows:
Real-World Data (RWD) are data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources.
Real-World Evidence (RWE) is the clinical evidence about the usage and potential benefits or risks of a medical product derived from analysis of RWD.
“This framework is the culmination of a longstanding effort that’s been underway inside the FDA; and it’s another milestone in our effort to advance the use of RWD and RWE to better inform patients and providers,” FDA Commissioner Scott Gottlieb said in a statement. “This framework is intended to advance the collection of data that are appropriate, consistent and provide information and knowledge that can better inform regulatory decision-making.
“For example, currently used EHRs and medical claims data may not capture all of the data elements needed to answer significant questions of interest.
“That’s why part of our new framework is to explore strategies for filling the gaps other sources of RWD, which may include the use of mobile technologies, electronic patient reported outcome tools, wearables, and biosensors,” Gottlieb said. “Recording individual patient data accurately and consistently while ensuring the utmost privacy will be an important challenge as we aim to shape the future of evidence generation.”
Under the Cures Act, FDA’s RWE Program must evaluate the potential use of RWD to generate RWE of product effectiveness to help support approval of new indications for approved drugs. RWD can also be used to improve the efficiency of clinical trials, even if not used to generate RWE regarding product effectiveness.
Examples of RWD include data derived from:
electronic health records (EHRs);
medical claims and billing data; data from product and disease registries;
patient-generated data, including from in-home-use settings; and data gathered from other sources that can inform on health status, such as mobile devices.
RWD sources can be used for data collection and, in some cases, to develop analysis infrastructure to support many types of study designs to develop RWE, including, for example, randomized trials—for instance, large simple trials, pragmatic clinical trials—and observational studies, prospective or retrospective.
FDA does not consider evidence from traditional clinical trials to be RWE. The agency’s RWE program will cover clinical trials that generate RWE in some capacity (i.e. sources other than traditional clinical trials) and observational studies.
“Another thing that was in there that I thought was important is ultimately, the focus on pragmatic trials and asking the question, ‘When can the generation of real-world evidence be retrospective vs. when should it be prospective?’ And I think that’s going to be something that, as a whole community, we’re going to need to think through,” Abernethy said.
Some clinical trials may use hybrid design. For example, certain elements of a clinical trial could rely on the collection and analysis of RWD extracted from medical claims, EHRs, or laboratory and pharmacy databases. Researchers could collect other data specifically for the trial, such as results of exercise stress tests or radiographic analyses, using methods typical of a traditional clinical trial.
“A hybrid trial could use RWD for one clinical outcome (e.g., hospitalization, death), while other elements were more traditional (e.g., specified entry criteria, monitoring and collection of additional study endpoints by dedicated study personnel),” the framework states. “FDA will consider these hybrid trial designs to have the potential to generate RWE. Clinical trial designs can also include some elements that more closely resemble routine clinical practice, which are sometimes described as ‘pragmatic’ elements. These pragmatic clinical trials often rely on RWD and have the potential to generate RWE.”
According to the framework, real-world data can help with:
Generating hypotheses for testing in randomized controlled trials
Identifying drug development tools (including biomarker identification)
Assessing trial feasibility by examining the impact of planned inclusion/exclusion criteria in the relevant population, both within a geographical area or at a particular trial site
Informing prior probability distributions in Bayesian statistical models
Identifying prognostic indicators or patient baseline characteristics for enrichment or stratification
Assembling geographically distributed research cohorts (e.g., in drug development for rare diseases or targeted therapeutics)
“The FDA framework document provides much needed clarity about the uses of RWE that the FDA may consider, the evaluation standards that may be applied, as well as the outstanding issues that need to be solved going forward,” Jonathan Hirsch, founder and president of Syapse, said to The Cancer Letter. “One of the most significant outstanding issues that has not yet been addressed, which is of particular importance in oncology, is the real-world outcomes measures that will be acceptable as study endpoints.
The FDA framework document provides much needed clarity about the uses of RWE that the FDA may consider, the evaluation standards that may be applied, as well as the outstanding issues that need to be solved going forward.
Jonathan Hirsch
“This has been a challenge in the development and use of RWE that has not yet been solved, and it will require collaborative engagement with the FDA, oncology leaders, biopharma companies, and technology platform companies.
“It is encouraging that the FDA recognizes that real-world evidence can and should be derived from data sources beyond the EHR—it is our position at Syapse that, by looking at multiple source systems, including registries, molecular labs, pharmacy, you can see a full and accurate picture of the patient’s care journey.”
Because the use of RWD to improve efficiencies of drug development programs that rely primarily on traditional clinical trials is already well established and generally encouraged by FDA, that approach is not the focus of the framework.
FDA has used RWD primarily in its evaluation of safety, and only in limited circumstances to inform decisions about effectiveness.
FDA’s RWE Program will therefore focus on exploring the potential of RWD/RWE to support regulatory decisions about product effectiveness. Specifically, FDA’s RWE Program will evaluate the potential use of RWE to support changes to labeling about drug product effectiveness, including adding or modifying an indication, such as a change in dose, dose regimen, or route of administration; adding a new population; or adding comparative effectiveness or safety information.
“My personal expectation is that we do the science really well—that real-world evidence is not about creating a landscape where we can be sloppy in our scientific work,” Abernethy said. “And I feel like this framework document reminds us that that’s the case.”
The strength of RWE submitted in support of a regulatory decision depends on the clinical study methodology and the reliability (data accrual and data quality control) and relevance of the underlying data.
In general, FDA does not endorse one type of RWD over another.
“Data should be selected based on their suitability to address specific regulatory questions,” the framework states. “For the purposes of evaluating drug safety, for example, FDA has already outlined its perspective on using RWD available in electronic health care data systems for safety studies in its guidance for industry and staff. The Pharmacoepidemiologic Guidance includes recommendations for evaluating the data sources used in pharmacoepidemiologic safety studies.
The framework includes consideration of the following:
Whether the RWD are fit for use
Whether the trial or study design used to generate RWE can provide adequate scientific evidence to answer or help answer the regulatory question
Whether the study conduct meets FDA regulatory requirements
(e.g., for study monitoring and data collection)
FDA intends to use this three-part approach to evaluate individual supplemental applications, as appropriate, and more generally to guide FDA’s RWE Program.
In limited instances, FDA has accepted RWE to support drug product approvals, primarily in the setting of oncology and rare diseases.
This framework is intended to advance the collection of data that are appropriate, consistent and provide information and knowledge that can better inform regulatory decisionmaking. Recording individual patient data accurately and consistently while ensuring the utmost privacy will be an important challenge as we aim to shape the future of evidence generation.
Scott Gottlieb
When approval is based on a single-arm interventional trial—often when using a parallel assignment control arm is unethical or not feasible and usually when the effect size is expected to be large, based on preliminary data—the supportive RWE has consisted of data on historical response rates drawn from chart reviews, expanded access, and other practice settings.
For instance, Blincyto (blinatumomab) was initially approved under accelerated approval for the treatment of Philadelphia chromosome-negative relapsed or refractory B-cell precursor acute lymphoblastic leukemia.
This was based on evidence of complete remission and duration of CR from a single-arm trial, the response rate of which was compared to historical data from 694 comparable patients extracted from over 2,000 patient records from European Union and U.S. clinical study and treatment sites (Przepiorka et al. 2015). Further study in a randomized controlled trial was required by FDA to verify the clinical benefit (Tower study NCT02013167).
The agency has stated that it sees promise in the opportunities created by pragmatic clinical trials, including broader inclusion and exclusion criteria and streamlined data collection.
FDA, therefore, will evaluate the strengths and limitations of pragmatic approaches using RWD, considering the following factors:
What types of interventions and therapeutic areas might be well-suited to routine clinical care settings?
What is the quality of data that can be captured in these settings?
How many patients can be accessed (particularly when outcomes are rare)?
What are the variations inherent in clinical practice?
Along with other activities under FDA’s RWE Program, the agency will assess the data standards and implementation strategies required to use RWD/RWE, identify any gaps between those requirements and existing FDA systems, and recommend a path forward to ensure that RWD/RWE solutions are an integral part of the full drug development and regulatory life cycle at FDA.
Activities for this work include:
Identify data standards and implementation considerations that apply to proposed uses of RWD/RWE at FDA
Review existing RWD/RWE-driven work, both internally and with external stakeholders, to identify gaps that need to be addressed. This evaluation could include projects such as development and use of relevant data standards, implementation strategies for applications and databases to connect with RWD sources, and strategies for coping with variations in data quality
Collaborate with internal and external stakeholders to adapt or develop standards and implementation strategies for RWD/RWE-driven solutions at FDA
Integrate RWD/RWE-driven solutions with existing FDA systems. This activity could include assessment of topics such as Health IT strategies for RWD receipt and processing for potential use at FDA, impact on reviewer workload, and tools and training needed for FDA reviewers.
“It’s hard to write a group of recommendations or guidances for real-world evidence without having enough examples sitting in front of you to be able to say, ‘Here is what it should look like,’” Abernethy said. “So, a critical element of this entire book of work is to have a series of meaningful real-world evidence examples that the FDA can critically review, and they can then form how they think about building the guidance and the recommendations.”
According to the framework, the agency plans to develop and issue the following guidances:
Building on the Pharmacoepidemiologic Guidance, FDA plans to issue guidance on how to assess the reliability and relevance of RWD from medical claims and EHRs used to generate RWE regarding drug product effectiveness. FDA will also examine how to assess the reliability and relevance of registry data and international electronic health care data.
FDA will review and, where applicable, publish guidance on potential gaps in RWD sources and strategies to address them.
FDA’s RWE Program will develop guidance on considerations for designing clinical trials that include pragmatic design elements and that generate evidence of effectiveness for regulatory decisions. FDA will explore pragmatic approaches to each stage of a clinical trial, including recruitment and enrollment of patients, strategies for facilitating interventions, and approaches to assessing outcomes.
Guidance on the use of RWD to generate external control arms is also being considered.
Adapting and building on the Pharmacoepidemiologic Guidance, FDA plans to issue guidance about observational study designs using RWD, including whether and how these studies might provide RWE to support product effectiveness in regulatory decisionmaking. FDA will also consider reporting requirements for such studies used to support effectiveness determinations.
FDA plans to finalize the guidance Use of Electronic Records and Electronic Signatures in Clinical Investigations Under 21 CFR part 11 – Questions and Answers and consider its applicability to different study designs. FDA will also issue additional guidance as needed on regulatory considerations raised by different study designs using RWD to generate RWE that is submitted to support drug product effectiveness.
