Prasad: FDA has confused merely approving drugs with making the world a better place

I have been following Vinay Prasad’s work for several years, agreeing with some of what he said, but never quite finding time to look carefully at his argumentation.

This changed on June 8, when The New York Times published an editorial based in part on Prasad’s opinion pieces. The editorial argued that FDA is approving drugs too fast and on too little data, thereby benefiting drug companies, but not the cancer patients.

Prasad’s papers were cited incorrectly as studies. However, one of them was slugged “Comment” in Nature Reviews Clinical Oncology. The other was labelled “Perspective” in Nature.

The time had come to give Prasad a call.

I had general questions about his thoughts about Twitter, about his views on the FDA standards for drug approval, and—more urgently—I had profound questions about statistical and methodological underpinnings of the two opinion pieces cited in the Times editorial.

Prasad spoke with Paul Goldberg, editor and publisher of The Cancer Letter.

Paul Goldberg: I’ve been reading your work, but I’m really calling primarily because of The New York Times editorial, which refers to you a couple of times. Are you basically in agreement with it?

Vinay Prasad: I am in agreement with the spirit of it, and with most of it. I thought it was a very provocative and timely and important editorial, and I support the central message here, which is there is a price for less data. Less data for drugs means more uncertainty, and it can exacerbate some of these issues with dealing with price, affordability, access.

I’ve also been reading your Twitter feed for years. I guess what’s interesting there is, it makes me wonder whether Twitter is actually the place to deal with issues like drug approval and endpoints—issues that have been traditionally dealt with in peer reviewed literature.

VP: I guess I would say that, although I tweet about things often, I do not believe I have made any arguments on Twitter that I have not first made in the peer reviewed literature. I have some arguments that I purposely do not make on Twitter, because the paper is under review. I’m actually cognizant of that, although I think Twitter is … let’s be honest, why do I use Twitter?

Number one, I find it fun. I find it fun to use Twitter, it’s enjoyable, it’s interactive, you get to hear from interesting people. I do not use Twitter to debut ideas, I use Twitter to get ideas out that were published in peer reviewed journals.

I’m pretty sure that everything I’ve said on surrogate endpoints we’ve already published in a couple of papers.

VP: I must clarify one thing: I referred to their analysis as “boneheaded,” but I did not ever refer to the individuals as “boneheads.” And I refer to their writing is cocky and arrogant, which their writing is cocky and arrogant.

It’s hard to argue with that.

VP: That paper is fundamentally flawed. If it were flawed that would be one thing, I would let it go, there are lots of things I read that are flawed, I don’t have time to critique everyone, but this is not just flawed. It will hurt human beings and the combination and the fact that it was published in that journal and it makes a fundamental mistake that now not just I pointed out, Bob Califf and others have pointed out on Twitter, suggest that I really didn’t like that paper. It shouldn’t have been printed, frankly, it’s not the appropriate way to look at phase I response rates.

That’s sort of interesting how Twitter is becoming a more common place to air disagreements, probably thanks to our president.

VP: But, also because for years, journals have buried disagreements. They’ve actively suppressed letters that made provocative points, and that’s not just my opinion, but Frank Harrell and Rodney Hayward, who are two very senior esteemed people, have both said the same thing. They feel the same way. Journals have not wanted to publish letters that were very damning to their articles, and now there is a platform where lots of people can read it, and it skips the middle man of the journal.

I think that is a very democratized thing.

Just tweeting at them… But still, drug approval is a very complicated thing. Seeing that being on Twitter makes me kind of scared, but that’s just me, I guess.

VP: There are two separate issues. Phase I, I don’t think is about drug approval, but drug approval is complicated, I agree. But I think it’s nonsense to scientific dialogue, it’s also social and political dialogue. The New York Times editorial takes it to a broader audience, and so does Twitter.

It just feels like it drags out sometimes into the gutter, but I guess maybe we’ve all been too polite too long.

VP: On which topic, drug approval or about bad papers, journal articles?

The way we all discuss papers, the number of characters is just not enough sometimes, for me at least.

VP: I agree, but I think with Twitter you can also thread it, and give tutorials, and visual aids, you can’t do that in a letter. A letter is a 150-word limit, that is more restrictive than Twitter or a blog. I actually think the old way of publishing critiques of journals is more restrictive than the new way, where there are no limits to characters. You can just keep tweeting a long stream of tweets. I tend to think it’s a good thing that more people can engage.

I think facts do prevail. I think that my tweets are not popular because of the spicy word choice. I think they are popular because I actually patiently explain what I think is wrong. I know that other people sometimes use spicy language, but they do not patiently explain what is wrong with the paper, and they don’t get the same reception. This is my view.

Of course. Can we get to The New York Times editorial for a couple more minutes?

VP: Yep.

Your work is cited in a couple of places, do you think it’s cited correctly?

VP: The sourced paper is a paper we published in Nature Reviews Clinical Oncology. I would say that first instance is cited very correctly: drug approval has become so lax and relatively inexpensive, one recent study suggests, in a thought experiment, that a company could theoretically test compounds known to be ineffective with the hope to get a false-positive result that would enable them to market a worthless medicine at enormous profit.

I think that is a fair summary of what our paper suggests. In our paper, we are very clear to say we don’t think that is actually happening, but that the problem isn’t that drug companies are that bad. The problem is they might not be much better. I think that is very accurate.

The second instance, I think is slightly inaccurate, where it said, according to a recent study, targeted cancer studies will benefit fewer than two percent of cancer patients they are aimed at.

They’re citing a paper of mine, where I show that for patients who have exhausted other therapies, if they subject themselves to next-generation sequencing, I estimate that less than two percent of them would obtain a response from sequencing and being paired with a targeted therapy.

That was my estimate in 2015 in Nature, it’s slightly different than what they’ve written. In fact, my estimate was validated by a trial called MOSCATO 01, which appeared in, I believe, CCR. It was exactly 2.1 percent response rate for that population. I think my estimate is accurate.

The way I read the actual piece, you say that 30 percent of the people getting the therapy benefit. I guess I’m a little bit confused about denominators. Do you have to go from the entire population even though those that don’t have the mutation? Is it sort of like running a trial of a TB drug, using all infectious diseases as your denominator?

VP: No, that’s incorrect. Let’s take MOSCATO 01. First of all, that paper’s three years old. Now we have the actual study, MOSCATO 01.

Let’s look at MOSCATO 01, because in MOSCATO 01 they took about 1,000 people with cancers of diverse histology—these are 1,000 people who in the community would say, “Should I send my tumor to F1CDX? Should I send it to MSKCC Impact? should I send it to MD Anderson? What should I do, should I send my tumor?”

Of that 1,000 people, baseline like 20 percent would be able to be matched with a therapy, about 200, and of that 200, about 10 percent have a response in MOSCATO which is about 20—so about 2 percent of the overall 1,000.

The other 800 people who consented to the study it’s a small solace to them that there was no targetable mutation found.

They participated with the highest hopes, so the denominator should be of all the people who subject themselves to sequencing, who desire that sequencing, what percent will end up with drugs that give them a response?

I think that is the only, that’s an intention to treat denominator, that’s the right denominator.

Of course, any other denominator will inflate the benefits story, excluding many patients in whom there is no match, and that would be deeply unfair to those patients who put their hopes in the test.

If you could explain this to me, how it’s not like running a trial for TB drugs and using all infectious diseases as the denominator.

VP: No, it’s not that way at all.

In that situation, all infectious diseases—I don’t know a lot about TB—but what you’re talking about is if everybody with an infectious disease subjected to an intervention… In this case, 1,000 people are saying, “I’m going to get sequenced.”

In MOSCATO trial, only about 900 could be sequenced. Of that 200 could be matched with a drug, and out of that, 20 had a response. These people are actually participating. The denominator is correct.

In fact, that number has been vindicated.

The New York Times, what they’ve written and this thing you cited are slightly different.

I have another paper that actually is a perfect fit for what they said, but I don’t know what to say about this.

I believe that if you participate in sequencing, you are entering into the protocol, you want to have your tumor sequenced. The question is, of all those people who want to have their tumor sequenced, how many people get matched on a therapy and have a response? And that is 2 percent, based on most cocktails.

One could, you could ask the MOSCATO investigators why did they report the 1,000 in their paper in CCR, because that is the true denominator, I think.

Well, at the AACR this year I saw your slide in which you reach a fairly similar conclusion by pooling basket studies, which is sort of interesting. Jose Baselga said that it would irresponsible to even discuss it, because basket studies shouldn’t be pooled. Do you accept his criticism on this?

VP: What is the criticism? That in his opinion they shouldn’t be pooled? I think that it is reasonable to pool studies that have been published. What we are doing, we systematically investigated every basket trial that has been pooled, and we asked a different question, which is, “If you enrolled in a basket study that was later published, what is the average response rate. That’s 20 percent?” He’s saying you can’t pool them, but he’s not actually giving any reason why one would not want to pool them.

He’s saying that methodologically that shouldn’t be done, because…

VP: Why?

Because they were not designed to be pooled.

VP: Similarly, Paul, one could say that to everyone who did a meta-analysis of aspirin, were those studies designed to be pooled? It’s in the view of the meta-analyst. When one looks at any type of meta research, the meta researcher decides, could they be pooled or not.

He’s not actually giving you are reason. He’s using words saying he doesn’t like this, but he’s not actually providing a reason why he doesn’t want it to be pooled. We can disagree about that, but I think this is asking a different question, and this is a fair way to ask that question.

I just wanted to ask about it.

VP: Yeah, I know, I really don’t … he’s saying he believes that they are not designed to be pooled. Well, one could say when one did the first randomized trial of statin, one never thought there would be forty randomized trials of statin, and that randomized trial was never designed to be pooled. And yet the cholesterol treatment trialists have pooled them many times.

I think it’s not a logical thing to say.

I’ve read a lot of your other papers, and you basically are advocating, in terms of FDA, you’re advocating bringing back overall survival or cure as well as standards for approval and also two controlled trials. That’s correct right?

VP: No, not quite. I’ve never said that. That’s The New York Times editorial.

OK, so you disagree with that? I just want to establish that, because that’s how I’ve read your work as well; maybe I have misread it.

VP: No, I think, have you read my paper, well, OK then, let me correct you. I guess I would say that I do not believe that two well established trials powered for survival need to be done prior to the approval of all therapies. I do not believe that that is the case.

I believe that surrogate endpoints can be used to approve new drugs. In fact, I have a paper called, it’s an open access paper, it’s published in BMC Medicine, and it’s with Robert Kemp is the first author.

It’s called “When Should Surrogate Endpoints be Used for Drug Approval and are They Currently Overused?” In my paper, we argue that we believe that they can be used, and they should be used for drug approval, under certain circumstances, as accelerated approval, with a post-marketing commitment for at least one trial that measures survival and quality of life—in most cases.

We also say the FDA is currently using them too much. I think the nuance here is that it would be easy to say, and incorrect, that I have asked for two trials in all cases prior to approval, that’s wrong.

What I’m saying is they are currently overusing surrogates. They’re using surrogate for regular approval, where there are no post-marketing commitments, which I believe is a dangerous precedent, especially when those surrogates are unvalidated, and a paper with Chul Kim and I in the Mayo Clinic Proceedings shows are about a third of the case. That they are unvalidated surrogates used for full regulatory approval with no post marketing commitment, and I disagree with that.

But I do believe surrogates can be used for accelerated approval in certain instances, with post-marketing commitment.

I think the only thing I would say strongly is that, in the current world, in a paper we showed, of 36 drugs were approved based on surrogate, in 4.5 years of followup, only five of them later showed survival benefits.

I guess I would say that 31 drugs on the market for five years with uncertain survival benefits, or quality of life benefits, I think that is a bit too high, out of 36. I think that we could do a better job of providing that information.

VP: I guess I would say that’s not really the purpose of these papers. The purpose of these papers is to outline a different regulatory framework for how we can move forward. Gleevec for CML obviously stays, because it’s a life-changing therapy.

But I think that one has to realize that there are ways drug regulations could be improved. These are broad ways, that doesn’t mean that, how could I put it: You have to have these rules laid out at the outset of drug approval to make really systemic changes. You can’t go retroactively and start pulling drugs off the market.

I’d actually don’t support that. I think we have to have clearer standards moving forward and move forward with a shared understanding. Nobody wants the rug pulled out from under them in the middle of a situation.

You can’t change the rules midstream, you have to change them slowly for future drugs, I think that is what I would say. Going forward, I think we need to think about this more carefully.

Gleevec has been around for decade, but within that decade they did show survival; right? It’s actually validated. What about something like crizotinib for ALK-positive lung cancer?

VP: I guess I would say that I would not want to go through specific drugs and say should they pulled or yanked. I’m kind of put out a better regulatory framework going forward.

OK, that’s fair enough. I’m wondering about randomizing, in which case, in some cases, I’m just looking at the ethics of it. If you’re running a randomized trial of say a BRAF inhibitor in melanoma, would you want to randomize to dacarbazine and a disease progression would you want to cross that patient over to a BRAF inhibitor? How would that impact OS?

VP: They did such a trial called BRIM 3.

But that’s not with the dacarbazine, was it?

VP: It was B vemurafenib vs dacarbazine.

Right now, would you do that kind of a trial?

VP: Right now? No, I wouldn’t do it right now.

I would say that moving forward, I believe that the majority of anti-cancer drugs could be subjected to randomized controlled trials at the outset. I think it’s a different question about what to do about drugs that have already been approved under a variety of circumstances.

The particular drug you are referring to vemurafenib was approved based on a randomized controlled trial called BRIM 3, which was talked about extensively.

Sure, I really should’ve made it more clear about going forward vs. what happened then.

VP: There are many drugs that we could do more randomized trials for at the outset, prior to approval.

I’ll give you an example: atezolizumab in bladder cancer. Second line was approved on a 13 percent response rate in an uncontrolled study. I think that is something that you could have done the randomize control at the outset. Atezolizumab vs standard of care at the outset. I don’t think there’s lacking equipoise, given a 13 percent response rate with an immunotherapy. But that was not the case, it was approved based on a response rate, and now we have a negative study.

I think we could do some more randomized trials at the outset. We could also measure survival and quality of life sometime in the life cycle of the drug trial; it doesn’t always have to be at the outset, but it has to be at some point, I think.

VP: Yes, it is a thought experiment—yes.

But a study?

VP: Is a thought experiment a study? I don’t know. I think a thought experiment is a type of study, it’s a thought study. In certain fields, some of the studies are purely thought experiments. I think it’s a very clever paper.

I guess at the end of the day, I think that’s a good paper. It’s a very good paper, it’s a very clever experiment, and I haven’t heard anyone articulate anything they think is fundamentally wrong with that thought experiment that would change the conclusion.

I can walk you through the thought experiment, if you would like.

VP: No, that was one-sided p. If you want two sided then it would be 2.5, and that would be $880 million, and answer will still be true.

I thought it would be one in 40 then?

VP: You’re saying two-sided, but I think, I’ll give you an example that’s actually in a second that will make this very clear, but I will say this: In our paper, we use a thought experiment of a one-sided p-value of a .05. If you would prefer, Paul, we can use a two-sided, for the sake of this phone call in which case I would say that would be one in forty, and that would change our estimate to $880 million, but we’ve already proven in our paper that it’s about $1.7 billion in profits.

Our conclusion would still be true, but I would say to you that I think our estimate is better for the following reason: olaratumab, which is a drug approved in sarcoma, was approved on the basis of a clinical trial where there was a two-sided p-value of .2, which is equivalent of a one-sided p-value of .1 which is double of what we actually put in our paper, which would lower our value to $220 million.

[Editor’s note: this study was actually a phase II approval study, Prasad’s analysis focuses on the p-value cutoff for phase III studies. The sponsor won an accelerated approval on PFS, which allowed them to look at overall survival, as stated in the package insert. A confirmatory trial is ongoing.]

I would say there is a track record of the FDA using even a more permissive p-value than the one we use in our paper, which is olaratumab, in that Lancet Oncology paper.

I would say that somebody might want us to have used a two-sided p-value, but we chose the one we did, because it’s a nice way to illustrate it. The truth, the real world, is that we’re even more permissive than what we supposed in our paper.

The other thing I say to you is, why stop with two-sided p-value, Paul? You could push it, you could say what if one were to look at the fact that all clinical trials partition a p-value over many looks, you could look at the data many times.

If you partition the p-value in that way, one might reach a slightly different number. I think our conclusion in this paper is fundamentally strong and sound, and that it is profitable with these assumptions. It would be profitable with these assumptions. In fact, we have an example of it, even more lax approval, which is olaratumab.

VP: Yes, the power calculation is even… it’s exactly right.

In that specific case?

VP: Yeah, in that specific case. All one needs is to say is this: a bar is only as low as the lowest thing that crawls over the bar. The person who wins the limbo is the person who limbos beneath it—that’s where the bar is.

I guess I would say that I like this idea of two-sided. I think it’s clever; I mean we thought about it when we did it.

We could use a two-sided p, and it would be $880 million, if you want. It’s a different thought experiment. What I think the true thought experiment is what is actually happening in America, which is, that we had approved a drug that had a two-sided p-value in the power calculation of .2, which is a one-sided p of .1, which would be twice, or—one half of our…—which is twice as permissive as the one we used in our hypothetical, which would give you half the number.

I would say that our number is good.

Just looking at those numbers in the paper again, I guess I’m still confused a little bit, because in your assumption on the cost of testing each drug, you note that the HHS report provides and estimated cost of conducting a single phase III trial in oncology at approximately $22.1 million. In the same reference, the key cost drivers chose that a stacked cost of cancer drug approval is at $37.8 million across all phases of study.

VP: But you can’t look across all phases.

Why wouldn’t phase I and phase II matter? These are costs.

VP: No, but it’s the sunk cost. What we’re talking about now, we’re assuming that drug companies have a portfolio of drugs that made it through early phase testing. We’re asking them, we’re pretty much trying to explain what you found, which is why do we see so many redundant duplicative PD1 trials? You’ve already sunk the cost on phase I and II, and now the question is, “Is it worth it to test a PD-1 drug over, and over, and over again?”

The drug companies, the ones that are testing over and over again are Avastin, or nivolumab or pembrolizumab. They’re not new drug, new drug, new drug. To that degree, for the purpose of this thought experiment, those kind of costs are sunk, but Paul, let me say just for the sake of argument, let’s consider the $37 million—let’s do it.

In our graphic in that paper, you can see there’s an X axis that goes—from the top of my head—from 20 to 40. So, $37 million would fall within that X axis, and you can just move your finger along the line and take the number at $37 million, if you’d like.

I think that would still be below the average profitability of a cancer drug, so our thought experiment would still be sound.

What about adjusting for inflation here, because you’re talking about a report that goes back from 2004 to 2012, and this is six years later.

VP: It was published in 2014 or 16; when was it published?

If you were to adjust for inflation…

VP: Yeah, you can adjust for inflation, but it’s still … adjusting for inflation … you can adjust for inflation but it won’t change the number dramatically.

It would change it from $37.8 to I believe $44.4 million per study drug.

VP: OK, but I don’t like $37.8 [million] I like the $22 because I told you the sunk cost, but then it will go up by a few million, but I don’t think it will change the overall conclusion.

OK. Well it doubles it, actually.

VP: That’s only if you don’t believe…

If I don’t believe you should be looking at phase I and II. Right?

VP: And I don’t think you should be, because those costs are sunk, as we explain in the paper. You could ask Chris McCabe about that, because he’s the economist that we have as a co-author.

OK, so you do have an economist on that? On the paper?

VP: Yeah, exactly.

The statistics of the thing kind of, the p-value thing I’m still a little bit stuck on that, because it just a fundamental thing.

VP: No, I don’t think so. I think that there are, in fact, cancer drugs that use one-sided p values. That’s one. For instance, ECHELON-1, I’ll give you a trial.

ECHELON-1 used one-sided p-value of .025 in their paper, and you’ll see the p-value that got them drug approval was like .04; it’s above the one-sided p-value.

The olaratumab is a very strong example. One could look at p-value in pertuzumab [in] APHINITY, the adjuvant study—that’s also a .047, off the top of my head, something like that.

I would say that one could pick whatever p-value cutoff one wants, but across, I think, a range of reasonable cutoff, and across what the FDA has proven they will approve drugs at, this kind of thought experiment is sound.

That’s my opinion, I know everyone doesn’t like this thought experiment. People don’t like this data, I get it, but, these are the facts.

Is there anything we’ve missed, anything you would like to focus on?

VP: If you really want to understand about how I feel about this issue, I would start with the article by Robert Kemp and I. Robert Kemp and I wrote an article in BMC Medicine about surrogate endpoints.

I’ve gone through it.

VP: We spend five pages trying to really put out what our view is on when they should be used and shouldn’t be. I think there are a lot of people who want to distort my position on this issue, and I think that they want to distort it, because they want to create a straw man that’s easier to defeat.

The truth is, I’ve put out, I think, a better proposal than what the current drug approval process is. I’ve always put out my proposals in the peer reviewed literature prior to tweeting about it.

I’ve never tweeted a proposal prior to that, except for things like this New England Journal letter thing, which is not really what this is about how we should communicate science and, frankly, I think that is a different sort of thing; it’s not FDA drug approval.

I think the letter to the editor is a dead field. I think it’s a dead thing and social media has changed that.

I would say about this thought experiment. I think this thought experiment is a very good thought experiment that across a range of different sensitivity considerations one would find this a true thing. What does it really mean? I think it really does explain why we find drug companies willing to spend large amounts of outlay on redundant duplicative trials with low pre-clinical rationale, often for drugs that lack single-agent activity. I think that is a key question.

We have a human welfare problem in the clinical trials system, which is that when you offer a trial of a very low value, you are squandering the scarcest resource that is existing in the system, which are patients.

We have currently incentivized that squandering through bad public policy. That’s all we’re trying to highlight.

The last thing I want to say, The New York Times should have just used our paper in JAMA Oncology, where we estimate genomic drugs to be 9 percent, 5 percent responders, I think that is a better estimate, for that particular quote that they’ve used.

I see, you’re talking about in the editorial; it should be about 5 percent response?

VP: It depends on the question you’re asking. If the question you’re asking is, of all the people with relapsed tumors who go on NGS, then the answer is 2 percent. If the question you’re asking is, of all the de novo cancer patients in America who may benefit from a genomically-targeted drug, the answer is about 5 percent, and that’s our estimation paper in JAMA Oncology that came out last month.

I think there are two different estimates. I think these numbers are much lower than what I think many would suspect them to have been. I think they are sobering.

I don’t think anybody really sees these numbers as especially high, it’s just that you have to go through a certain number of patients to find the patients who are likely to benefit. It’s a question of denominators. That’s really not a surprise.

VP: The reality is that those other patients are people too, and they’re people who are not benefiting. My heart aches for them.

What would you propose?

VP: They are being misled by the cancer centers, and the ads, and the rhetoric around it. They’re being misled, because they, people are paying out of pocket for F1CDx before this coverage guidance. People have gotten CMS to pay for this, this was not an ideal way to run this research agenda.

When you do find these people who otherwise would not benefit and they do benefit…

VP: That’s great! Fantastic!

Would you actually advise a patient to not get tested?

VP: I guess I would advise the patient, I would advise CMS to have conducted that as a randomized study which is what I advised CMS in that paper. Why CMS should run a randomized trial of F1CDx rather than pay for it. We really don’t know. You guys have covered PSA screening. Would you really advise healthy men to not get a PSA screening? Well it depends, if a randomized follow-up of PSA screenings showed the benefits then yeah, of course advise them.

If a randomized follow up of PSA screenings showed no benefits, then of course don’t advise them, and similarly, if a randomized trial of F1CDx shows a benefit, then of course, advise patients to do it. If a randomized of F1CDx does not show a benefit, then of course not.

The question is, will we ever see such a randomized trial? Who is going to a randomized trial like that? Who will force the randomized trial?

Medicare, when they covered F1CDx, have the legal authority to use a coverage with evidence development rule that said they could have mandated that as a coverage with evidence development. Being that you could get this paid for in the context of an RCT. They did that with a device called Wingspan a few years ago. It actually was a successful thing, it led to a trial that gave us information. They decided not to do that here. That is the only objection I have.

I’ve never said “Don’t study this,” I’ve never said, “Don’t do research on this,” I’ve never said every surrogate is bad. I have always said something slightly different…

There are a lot of surrogates out there and a lot of indications. Looking at the Times editorial—and I know you didn’t write it—they’re basically suggesting that we go back to the era when there was a survival rule, and when you needed two trials to prove it. I don’t think anybody really misses that era, do you? I don’t think you do.

VP: I don’t think there was ever that era, actually, Paul. If you go back, drugs have been approved based on response rate for a long, long time.

The question now is, when you approve drugs based on surrogates, are those surrogates unproven or validated? Do you have post-marketing commitments or not? Do you enforce them or not? Two Government Accountability Office reports say FDA is not enforcing them for surrogate approval.

Then the next question is, do you use accelerated pathway, where there is a post market commitment or do you use regular pathway, where there is not? They often are using the regular pathway and skipping that. Or the surrogate, they are using many, many surrogates.

They have confused merely approving drugs with making the world a better place, and it’s easy to just approve things. It’s harder to have knowledge that the things you’re approving make actual Americans who are older and frailer and clinical trial patients better off.

That’s the question that regulators have to deal with. Do these approvals make the population better off? Do they make average Americans better off? We don’t have answers to that, because there are so few, so little data being generated in this space.

The spirit of The New York Times editorial, I think, the spirit is, it’s not always about the least amount of data possible, sometimes you need more information. That’s a spirit that I think is very important.

Thank you very much.

VP: Thank you Paul, I appreciate it.

Thanks for walking me through this, thanks.

VP: Thank you for asking me tough questions. I would say I think you asked good questions. I really fundamentally disagree with some of them, but I think there is precedent, and I think the p-value thing is wrong, but I made my case and I wish you well.

Thank you so much.