Repeating the Third Year?
As March 2008 drags on, Perez is unable to resolve his problems.
In addition to pressure from Potti, he has to deal with an upcoming poster presentation to HHMI in May. If Perez is to walk away from the Potti lab, how would he assure HHMI that he has been doing actual work rather than, say, catching Frisbees at the campus’s Sarah P. Duke Gardens?
His goal is to get residency in radiation oncology the following year, but with his third year of med school producing no publications, his case for getting residency would be weak.
And, in view of the circumstances, Perez hoped that HHMI would give him another fellowship to repeat his third year, making it possible for him to work at another lab.
On the evening of March 27, Perez has a conversation with Caroline Haynes, director of student affairs and associate dean for medical education at Duke.
The following day, Haynes brings in Phil Goodman, associate dean for medical education, who proposes that Perez bring his concerns to Nevins, director of a center within the Duke Institute for Genome Sciences and Policy.
“I believe that your best course of action is to set an appointment with Dr. Nevins to discuss your concerns,” Goodman writes. “He would certainly want to know if there is some research misconduct going on in his labs. If you are unsure about seeing him on your own, then Dr. Haynes said she would be happy to accompany you. (I would, too, but Joe Nevins is a personal friend and I wouldn’t see me being there as constructive.)”
Goodman’s use of the word “misconduct” in an email to a medical student is worth noting. At that time, critics of the Duke group would not have even whispered such accusations. Rather, they claimed only that Nevins and Potti were making spectacularly crude errors. The word “misconduct” wouldn’t be uttered until after Potti’s enhancement of his credentials caught up with him.
Perez responds late that afternoon.
He proposes a course of action that one would expect for a person of his generation: write Nevins an email.
“If I go through Dr. Potti it will likely be very difficult for me to get a word in edgewise during a meeting between the 3 of us,” Perez writes to Goodman. “I suspect that it would develop into Dr. Potti asking me difficult questions that are unrelated to my concerns in order to avoid dealing with the issues at hand.
“Finally, I could just send Dr. Nevins an email detailing my concerns. This seems like a cowardly way to go about raising my concerns and frankly I am not sure what would happen after that but it would avoid the issue of sitting down and having a meeting where it is possible that I am not able to articulate all my concerns.”
No, don’t send an email, Goodman responds later that day.
“I think you should set up a meeting with Dr. Nevins,” he writes. “If you are unsure that a personal meeting with him may leave you speechless or unable to get your points across, then prepare a letter with those thoughts, bring it, apologize to him, and then read it to him. This at least would set up points for discussion. You should do this ASAP since you might be able to salvage the year with 5 months to go. Dr. Haynes could come with you for moral support if you like.
“I wouldn’t worry about Dr. Potti at this point. You’ve brought the issue to him already.”
Perez appears to have taken Goodman’s advice and writes out a single-spaced, three page summary of his concerns, providing a robust picture of what was allowed to go wrong at the Potti lab
Research Concerns
“As a student working in this laboratory, I have raised my serious issues with Dr. Potti and also with Dr. Nevins in order to clarify how I might be mistaken,” Perez writes in his Research Concerns memo. “So far, no sincere effort to address these concerns has been made and my concerns have been labeled a ‘difference of opinion.’
“I respectfully disagree. In raising these concerns, I have nothing to gain and much to lose. In fact, in raising these concerns, I have given up the opportunity to be included as an author on at least four manuscripts. I have also given up a Merit Award for a poster presentation at this year’s annual ASCO meeting. I have also sacrificed seven months of my own hard work and relationships that would likely have helped to further my career.
“Making this decision will make it more difficult for me to gain a residency position in radiation oncology. As a third-year medical student, these are all very important things that I have given up. As a result of these circumstances, I am spending another year of my life pursuing a more meaningful research project.
“The reason that I have made the decision to leave the lab and make these concerns known is because it is important that the work be done right for the sake of our patients and for the field of genomic medicine.”
After presenting a thorough critique of problems with the work of the Duke team, Perez proposes this course of action:
“At this point, I believe the situation is serious enough that all further analysis should be stopped to evaluate what is known about each predictor and it should be reconsidered which are appropriate to continue using and under what circumstances.
“By continuing to work in this manner, we are going a great disservice to ourselves, to the field of genomic medicine and to our patients. I would argue that at this point nothing should be taken for granted. All claims of predictor validations should be independently and blindly performed. Unfortunately, since validation databases on the supplementary website have been shown to be misrepresented in multiple situations, those datasets should be obtained from their respective sources through channels that bypass the researchers.”
“Brad Perez is a Hero”
Had this been a test, Perez would have nailed it.
According to top-tier biostatisticians who were asked to review these documents, his understanding of biostatistics was extraordinary for a med student—or even for someone with specialized training.
“Brad Perez is a hero,” said Donald Berry, a biostatistician at MD Anderson. “To recognize rot is one thing. To challenge and to try to correct one’s supervisors and recognized world authorities takes chutzpah. And conviction. And integrity.
“As he said in his refreshingly erudite ‘Research Concerns,’ he had much to lose, and he had already lost a lot.
“At a President’s Council of Advisors on Science and Technology panel in January 2014 dealing with reproducibility in science, someone suggested that young scientists should learn about the scientific method from their mentors.
“I responded: ‘Commenting on education, the problem is really who are the educators. The senior scientists are the problem. They are not the solution. And…young statisticians are just as clueless as the senior scientists.’
“A great example is this wonderful story about Perez teaching Potti and Nevins about science and his older, but not wiser students being too entrapped by their hypotheses—or too ignorant about science—to understand.
“There is more to this story than the heroic and principled actions of an erudite young man and the shame that has befallen a great university in blindly and selfishly defending its own. It is indicative of a lack of understanding of the scientific method among many scientists.
“The Duke scandal is extreme, to be sure. But irreproducibility in academic research is common. And the reward structure and complacency of universities is to blame. In the same PCAST panel I suggested that ‘the utility is so different for senior scientists. They get a paper in Nature. That’s wonderful. If they have to do a correction they get another paper. It’s all in the utility structure.’”
Perez’s memo had the look of a document that deserves to be taken seriously, statisticians say.
“Perez seems to bring up four criticisms,” said Gary Rosner, director of the Division of Biostatistics and Bioinformatics at the Johns Hopkins University Sidney Kimmel Comprehensive Cancer Center.
“1) The development of the genomic signature is flawed, because they chose to remove data elements that made things look bad.
“2) Potti and Nevins never really tried their signatures in a truly independent dataset.
“3) The software they used (BinReg, presumably) was not stable, leading to substantially different results across versions.
“4) Despite repeatedly raising questions and concerns, Perez was effectively stonewalled by his mentor (Potti), and the project was taken away from him. Essentially, Perez is saying that the response to his questions was that this is more complicated than he can understand.
“It sounds to me like he is saying that they stacked the deck in favor of reproducing the results from the ‘training’ dataset when they applied the model in the ‘validation’ data.
“I looked at the exchange between Coombes & Baggerly and Nevins & Potti in Nature Medicine. One of the concerns Coombes and Baggerly raised was the use of all data (training and validation datasets) when generating the metagenes. Nevins & Potti dismiss this concern, arguing incorrectly, that Baggerly & Coombes reproduce the results when developing the model using just the training data.
“Perhaps Perez’s concern relates to this same flaw.
“In terms of Nevins & Potti removing data that were not consistent with their model, there’s not much we can say. If the data were not made available, then no one would have been able to determine this. Don Berry talks about investigators preprocessing the data prior to giving the dataset to the statistician as a form of multiplicity.
“On the other hand, this practice may explain why no one could reproduce the Nevins & Potti results when they had all of the data.”
Perez Meets With Nevins
Perez takes Goodman’s advice, and later on March 28, he shoots an email to Nevins.
“I’ve been having a difficult time, and it was suggested that since you are one of my mentors for this research year that I try to set up a time to meet with you,” Perez writes.
Nevins responds two days later, on March 30.
“I am very sorry we haven’t kept in touch prior to now,” he writes. “I frankly have just let it slip and shouldn’t have.”
Sent just after midnight on March 31, the Perez response outlines the issues he wants to discuss.
“I am not sure if you are aware that I asked Dr. Potti to remove my name from all publications recently submitted or accepted for publication,” Perez writes. “I also decided it was best to avoid resubmitting my own first author publication to JCO. I also received an ASCO Merit Award for this year’s meeting, but I don’t feel comfortable giving this presentation because it involved applying predictors I was ultimately not comfortable with.”
Perez works in an artful backhanded compliment to Potti:
“I have learned a lot and I think that’s a testament to Anil and others willingness to spend time teaching me about how we do genomic analysis,” he writes. “Some of what I have learned however, is disappointing and I think what’s more disappointing is that I have raised these concerns with Dr. Potti before and nothing has changed.”
Perez wants to make Nevins aware of his reasons for seeking another year of funding from HHMI.
“I don’t expect a letter of endorsement for this application, but feel like it’s best to share it with you before I submit,” he writes.
Nevins quickly agrees to the meeting.
“You raise some serious issues here and I think we should talk about it sooner rather than later,” he writes. “Are you available this afternoon? I could meet around 4 p.m.”
The meeting takes place in Nevins’s office.
In a deposition, which is cited in part in court documents, Perez describes an exchange with Nevins where the scientist implored him not to send the letter describing his doubts about the science at the foundation of the Duke clinical trials that had already started to accrue patients.
According to the deposition, Nevins said that by sending a letter, Perez would harm his own career as he would not get additional HHMI support for conducting a different research project at another lab.
Nevins also says that the letter, if sent, would start an internal investigation at Duke. And he pledges to look into Perez’s allegations.
An excerpt from the plaintiffs’ court filing, which draws on the plaintiff attorneys’ deposition with Perez, follows:
In March and April of 2008, Brad Perez pointed out problems with the validation data sets and suggested that there had been “misrepresentations” by Dr. Potti and/or Dr. Nevins regarding the data and the predictors did not work.
In a letter sent to Dr. Potti and Nevins, Brad Perez wrote:
“In looking back at previous publications that claim to validate some of the predictors being used today, most validation data is either unavailable, missing clinical data or methodological methods so that validation cannot be performed, or even misrepresented. If the validation sets are not accurate, then they should not be used to make predictors…”
Brad Perez said that in a meeting Dr. Nevins validated many of Brad Perez’s concerns by referring to the serious issues raised regarding Dr. Potti’s lab as “being somewhere along the spectrum between sloppy research and a difference of opinion to research fraud,” and that Dr. Nevins confirmed that he would “go back through each and every dataset that we have posted in relation to various publications to ensure that there are no errors.”
Dr. Nevins did not, however, want Brad Perez to send the letter because he did not want Duke or any other entity looking closely at the data underlying the clinical trials.
Q: And then you say he [Nevins] asked me not to send the letter because he felt that my additional year would not get funded anyway and he thought that by sending a letter I would essentially be initiating an internal Duke investigation. I think he wants to avoid that. Is that what you said in your email?
A: Yes.
Brad Perez had access to the underlying data and methodology and he could not reproduce the results that Dr. Potti and Dr. Nevins claimed. He was convinced by Dr. Nevins to not send the full letter that would expose Dr. Nevins and Dr. Potti.
Brad Perez, to his own personal and professional detriment “made the decision to leave the lab and make these concerns known because it is important that the work be done right for the sake of our patients and for the field of genomic medicine.”
