Publishing failures to replicate without blame seems like a good way forward, similar to the practice of publishing postmortems in the computer industry.
Well, casting doubt on knowledge assumed to be true, which may lead to overturning, but is still provisional when you're at the point where you have one published experiment claiming a result, and one published failure to replicate that result. Even correct results are not always successfully replicated on the first replication attempt, including in "hard" areas like physics/chemistry. Lots of possible reasons. One frequent case is that the original results were, even if broadly correct, underspecified, i.e. there was some key component of the original experimental apparatus or method that wasn't sufficiently described in the paper, either due to an oversight or because the original authors didn't know it actually mattered to the outcome. Sometimes figuring out what that was makes it possible to replicate the result, although other times it turns out to undermine the original results (e.g. the entire result turns out to be due to some specific impurity in a specific supplier's equipment). But in any case still very useful.
Yes! Been there, done that. So much of lab technique is hard to explain in words. People do postdocs and sabbaticals in order to learn technique. And then there are uncontrolled variables. Classic example:
> When a student had difficulty in crystallizing a compound [professors] would simply shake their beards over the flask containing the offending substance. Then, lo and behold, after nucleation had done its job, crystallization set in. Gerhard believed that the beard of an Adolf von Baeyer or an Otto Wallach could indeed be a source of crystals of every conceivable space group.
Its not merely the way forward- its the whole problem. Nothing of this is at all challenging methodologically or technologically- scientists, as a group, are masters of such things. The challenge is social.
If you've failed to reproduce a result, this could be due to the original experimenter's flawed work, or your own mistakes, or unnoticed ambiguities in the experimental method, etc. There's no way to publish a negative replication without implicitly asserting, in the eyes of your fellows, that the original researcher did it wrong- its not what a negative result logically shows, but this is just how humans work. Jumping the gap between what you've shown and how your publication will be socially received takes substantial courage- scientific careers are fragile!
Everything should be published: positive results, negative results, positive confirmations, negative confirmations. Journals should only judge studies based on the quality of the science, not based on the "interestingness" of the outcome.
>'"I believe the main risk of a publication venue like the F1000 channel is that it becomes a place for “bashing" good science, because biological experiments are complex and beset by many variables that are hard to control.'
Funny how rare it is to see mention of this in original papers and press releases. When is the last time you read a biomed article where the authors said something along the lines of "we excitedly await the independent verification of our findings".
While I applaud this effort, it does concern me that they have chosen to publish only three of the claimed 47 failures to replicate.
The Begley that spawned in part this debate was seriously flawed for two reasons: the first is that the experiments they performed were not described in any detail, and the second is that the experiments they choose to repeat were clearly not selected at random.
Indeed, the limitations of their initial study may underlie the skepticism that greeted their claim that 89% of studies were not reproducible. Most scientists expected a certain proportion of studies to be difficult to reproduce, but nothing approaching 89%.
So, perhaps publishing details of ~6% of their study is an improvement, but again, ironically, it is not a statistically relevant sample.
According to the article, these three articles were not part of the original set:
> In 2012, Amgen researchers made headlines when they declared that they had been unable to reproduce the findings in 47 of 53 'landmark' cancer papers. Those papers were never identified — partly because of confidentiality concerns — and there are no plans to release details now either, says Kamb, who was not involved with that publication. He says that he prefers to focus on more-recent publications.
The Begley paper and the whole debate over reproducibility spawned a worthwhile introspection on behalf of the journals, several of which have adapted the information required from authors in response.
However, I am distinctly unimpressed by Amgen's failure to live up to the standards it expects of others; not releasing the information required to assess their original claims being their worst offense. In this series, one of the studies, as mentioned in the article, is even a repetition of 4 [1,2,3,4] earlier failed attempts to reproduce the original claim that bexarotene caused a reduction of amyloid in mice [5]; while worthwhile, it is not an impressive contribution.
If only we could prevent the news from making discoveries public before they've had a chance to be verified. Even if the news media publishes a retraction, it usually happens long after the initial results are already spread and reinforced as common knowledge.
I'm only just an armchair opinion but I get the impression that a some people are gaming the system for profit rather at the expense of intellectual honesty. The prospect of the field of science incentivizing 'junk science' is really unsettling.
I loved the part where one of the researchers who couldn't replicate other experimental results, had some of his experiments found to be not-reproducible.
