While reading this, two things kept popping up in my head:
1. Science and medicine are as much about marketing as anything else out there. If you're not making headlines, you're going nowhere.
This is not new by any means. The story of evolution and how Darwin "beat" Wallace to fame is a good example. History is littered with similar stories.
2. This is the peer review process working in full swing. Dammit it's very slow, but it works - eventually. Journal papers are supposed to be peer reviewed to catch the obviously wrong papers, and the downside is explained well in the article. However, the key is whatever bad research slips through is caught by repetition and meta analysis. And if these repetitions and analyses are themselves flawed, others will point them out.
It's exactly like open source. No one guarantees that open source produces bug-free code but we trust that the bugs will get weeded out faster.
1) I think a big point of this article is that the peer review process is misunderstood by the general public (and perhaps many doctors).
Though scientists and science journalists are constantly talking up the value of the peer-review process, researchers admit among themselves that biased, erroneous, and even blatantly fraudulent studies easily slip through it. Nature, the grande dame of science journals, stated in a 2006 editorial, "Scientists understand that peer review per se provides only a minimal assurance of quality, and that the public conception of peer review as a stamp of authentication is far from the truth."
More work needs to go into educating the public as to the meaning of scientific studies.
2) The scope of money involved in the endeavor is too large to be happy with things just slowly weeding themselves out as is permitted in the OS community.
Higher standards can and should be enforced in all medical studies, an area in which progress is being made but there is a ways to go. Let me push back and say, if the medical research community was exactly like the open source community, we're not asking much of those who are doing the research.
I know what peer review is. And to your point about the demand we make of researchers and the stakes involved, have a read of the Cochrane Collaboration website. Most of their conclusions are "no real evidence, need more studies". It's sobering how much is out there that doesn't stand to really demanding analysis.
1. Science and medicine are as much about marketing as anything else out there. If you're not making headlines, you're going nowhere.
This is not new by any means. The story of evolution and how Darwin "beat" Wallace to fame is a good example. History is littered with similar stories.
2. This is the peer review process working in full swing. Dammit it's very slow, but it works - eventually. Journal papers are supposed to be peer reviewed to catch the obviously wrong papers, and the downside is explained well in the article. However, the key is whatever bad research slips through is caught by repetition and meta analysis. And if these repetitions and analyses are themselves flawed, others will point them out.
It's exactly like open source. No one guarantees that open source produces bug-free code but we trust that the bugs will get weeded out faster.