The incentive system in science is immensely screwed up. Researchers are rewarded not for high quality work flows, reproducible results, and solid methodology but for neat, linear stories with a simple high impact result.
Imagine that a scientist designs a very intelligent experiment to test a plausible hypothesis - she has an excellent plan to analyze the results (pre-registered, to avoid p-value hacking) - and after doing the experiment, she finds no evidence of the effect that she was looking for. Why should the result (which is the only thing the scientist is not in control of) determine the reward given to the scientist? Funding should be given to investigate a particular phenomena, not to find high impact results.
While it's noisy and high variance, the approach is similar to start-ups in that we want experiments to be designed in ways that they are likely to confirm underlying theories. This makes positive results more valuable than negative results and something the scientist has some influence over, but not complete control of. Successful experiments increase the chance of future experiments being successful in much the same way that having previously had a successful start-up increases the chances of future start-up success. Particularly when committees cover huge areas, it's a lot easier to fund based on track records than to accurately determine experiment quality of an experiment well outside your expertise.
I hope I'm not reading what you wrote uncharitably, but, by rewarding people with positive results, we are inevitably encouraging people to either design their experiments in ways that confirm their hypothesis, or at worse, to falsify their data.
Thankfully, I've seen very little of the latter, but I've seen quite a bit of the former. Even unconsciously, people are afraid to design experiments that would give a black and white answer and falsify a cherish theory.
People with lots of positive results are not necessarily better scientists. They are just as often less scrupulous and better salesmen - Peter Thiel noticed that as well.
Yes, and that causes so many problems. As you indicate, all of science is easily biased by what has been done and what has been published in the past.
Negative results are essential for a complete understanding of science and when they are repressed, complete access to the globally available data set is not available.
If I'm an ethically-dubious pharma or lobby group, I love this situation. I keep funding testing until someone comes up with a positive result by chance or p-value hacking - then fanfare that result all the way to the media and to the bank. (Negative results don't need to be seen)
If I'm a cynical scientist, I take the money and p-value hack until I get the results my funding body wants, so that I have access to more money in their next round of funding. Now I'm far more successful than a completely honest scientist.
In fact, the honest scientist will either seek to replicate what the cynical scientist did, or leave the profession.
In many cases the funding is politically motivated. In turn this corrupts the results by a type of 'survival of the fittest' where fitness is determined by one's biases aligning with one's funding sources.
The reason funding goes to those that find 'good' results is the same reason why funding goes to those that start successful start-ups. Even though both parties probably have less control over the outcome than they think the perception is that they do and that betting on them again is a better bet than betting on someone that did not succeed to score.
That's how we're wired, we simply like success stories much more than that we like stories of failure.
The analogy doesn't completely work. If the data shows your hypothesis is false, or doesn't support its truth then it doesn't matter if you're the best experimenter in your field nor if you worked most competently the world is what it is (barring poor experiment design!).
An hypothesis isn't a product, falsifying your hypothesis is success - you've increased the pool of human knowledge (or increased the support for an element of it).
In business good and bad products can both lead to profit and can both lead to losses. Profit and loss can both be considered success and failure.
On your last sentence - I don't think it's as self-evident as it seems. We tend to like the successes that profit us but in general I'm not sure we humans are that congratulatory of other's success. Schadenfreude seems very popular.
False inherencies are the staple of those that wish to understand the world such that they might exploit it. If they weren't prepared to demonstrate that the pursuit of a solution was necessarily a fool's game then they'd have to give an actual shit.
It's a little hard to tease out from this particular article, but the amazing thing is that the canvassing actually happened: dozens of volunteers talked with hundreds of people at 22 minutes for each conversation. Easily thousands of man-hours of work. But the grad student running the study didn't do the easy part, hiring a survey company to poll and measure the before-and-after attitudes of the people being studied.
It's one thing to get data and then fraudulently change it to fit a story; at least then other people can go back and un-do the damage. It's a different thing to not do anything at all, completely make up data, and waste all that hard work.
As Peter Winker, an economist at the University of Giessen
in Germany who co-edited a book about poll fraud, put it in
an email: “If the faker would have been a bit cleverer in this
procedure, I doubt that the fraud would have been found this way.”
Even when data isn't outright fabricated, it often seems that studies like this are looking for a predetermined outcome that will reinforce the preconceived ideas of the authors. I feel so skeptical anytime a study is published that purports to show the truth of some idea that people would really want to believe ("if only they just met a gay person it would change their mind.")
>it often seems that studies like this are looking for a predetermined outcome that will reinforce the preconceived ideas of the authors.
Isn't this the definition of how science should work. 1) Formulate an hypothesis, 2) design an experiment to prove it 3) and then run it rigorously and accept whatever is the result.
This case (and many others) failed 3) but everything else seems more than natural. Physicists smashing particles at the LHC were also trying to "reinforce their preconceived ideas" of how particles work. They have the standard model and they want to validate it where possible.
This is my sentiment exactly, and why I posted the original article and read the refuting paper [0]. There's been a flurry of posts on HN recently about hypothesis testing and people not understanding what p-values are and what these statistics really mean, and I feel like that sits within the same critique about this paper. People who manipulate their model spec to get a convenient p-value are just as guilty as those who completely fabricate data.
I think the problem is that it takes diligence to design an experiment that is not biased towards the desired result.
I also think it's a systemic problem of positive results being far more rewarded (in terms of funding, prestige, etc) than negative results, which gives even more incentive to bias the result (intentionally or unintentionally) towards the positive, satisfying outcome.
Imagine you have two social scientists, A and B. Scientist A publishes a positive result that resonates with people; it fits the social and political climate of the day. They give TED talks about it, and the result is reported in newspapers. Later experiments are unable to replicate the result, maybe because the original experiment was a little biased somehow, but nobody can say for sure and there is no scandal because there was no fraud. Scientist B is more rigorous in their experimentation and their work fails to demonstrate a hypothesis. They either publish the negative result, or don't publish at all.
Which scientist will have advanced their career more? I think that is the problem.
Yeah, but then the problem is that the experiments are crappy not that the hypothesis generation is at fault. You used social scientists and I used physicists and that is telling. The social "sciences" are often far from scientific. If a physicist suddenly goes bonkers and claims that space is filled with ether he can't go very far because he can't generate a spurious positive result in favor of his crazy hypothesis.
I used social scientists because that is what this article was about and what my comment was about. I agree that in physics (the hardest of hard sciences) it is harder to fall prey to this kind of false positive result.
I think this is incorrect. A good experiment is designed to disprove the hypothesis. And so if the experiment fails to disprove, we've got one more vote of confidence in the hypothesis.
It's never possible to out-and-out prove any hypothesis. There's always going to be variables that we didn't think were relevant. All we can do is keep adding to the heap of evidence in its favor, by falsifying more and more objections to it.
>A good experiment is designed to disprove the hypothesis.
Statistical tests are often designed around a null hypothesis that is defined as the opposite of the hypothesis we are testing. Once we can reject that null hypothesis we have proven the hypothesis. So for the LHC you'd disprove something like "there is no particle of energy X" (the null hypothesis) by finding a definite signal at energy X, thus proving that the Higgs Boson exists (the initial hypothesis). Prove/Disprove are relative terms though as you can only have a degree of confidence X on the validity of the hypothesis.
>Isn't this the definition of how science should work.
There are two problems. First is the easy problem which is that #3 doesn't always happen. The data gets slightly cooked, the study gets redone, explanations are given as to explain away the results. The way it is done can be from outright dishonest to quite acceptable.
But the bigger problem is that there are some hypotheses that are not thought up or tested, or are done so in low numbers. Namely ones that would be political/career suicide, but even those that could hurt funding or that would have biased a person from ever becoming a researcher. For example, say you wanted to test an experiment that gay women were more violent to their domestic partners that straight men, straight women, or gay men.
Finally, even when results may be found, they will be attacked as being untrustable because the author must've been biased to even go looking for such results.
Most of what you mention, while probably true is not the issue I was replying to.
>Finally, even when results may be found, they will be attacked as being untrustable because the author must've been biased to even go looking for such results.
Sure, and that may be the problem of looking for a predetermined outcome that haberman was referring to. If your testing methods are poor enough that spurious results are likely, then the fact that all your hypothesis are somehow biased is a problem. Though I'd say the problem is more with the testing methods than with the hypothesis.
Using the LHC example again no one could attack Higgs for being biased as an argument for the boson not existing after the LHC results were in. The provenance of the hypothesis should be irrelevant if the testing methods are good enough to provide strong evidence for it.
> Isn't this the definition of how science should work. 1) Formulate an hypothesis, 2) design an experiment to prove it 3) and then run it rigorously and accept whatever is the result.
No. The critical difference lies in steps 2 and 3. You can never prove something in science, because you'll never get all the corner cases. What you're really after is designing an experiment that is able to falsify (=disprove) your hypothesis and run that. Because if you find one single instance that contradicts the original hypothesis, you know that your hypothesis in its current form is wrong, because it can't be applied universally.
You're mixing up concepts. You want an experiment that can falsify the hypothesis, because if it can't it's worthless. That doesn't mean you don't run the experiment to give evidence to the hypothesis.
So in the LHC case the hypothesis is "the Higgs exists and it's energy is X". Then they run the experiment and get evidence that says "there is a very high probability that there is a particle with energy X" and so their initial hypothesis is considered proven. If you don't like the word proven, then they conclude it's extremely unlikely to be false.
No. Science is about answering interesting questions, not justifomying desired answers.
Scientists should be invested in he question, not a preferred answer. Leave the answer preferences to engineers and inventors and businesses.
This custom of starting comments with "No." is quite frustrating. Vehemently stating that someone else is wrong does not an argument make.
Nor is the rest of the comment much of an argument. Scientists have questions they want answered and to solve them they formulate hypothesis and test them. If you don't formulate hypothesis you have nothing to test and can't advance.
There should be a Github for Academia where scientists open source their data & formulas the use to draw conclusions - people could comment and annotate on specific parts, fork & create pull requests, and file any issues they might have.
There is widespread provision of data and software on computing science papers in academia. It's currently a piecemeal approach, which is a problem, but here's an example of how things are changing:
Universities all over the world are working on repositories to store electronic resources such as data and code for the long-term. This is non-trivial, as data may need to be available for decades or even hundreds of years. Github, by contrast, has been around for 7 years.
Regarding the idea of organising research in a more change-control fashion, that's a more radical change. Academia will move slowly, and rightly so, as research practices evolve and persists for hundreds of years; to adopt an idea based on a (relatively, very) new method of software development is hugely risky.
Indeed, but it is critical to me (and I think I'm not the only one in academia for which it is so) that it uses the arXiv funding model -- donations to be sustainable -- and not the Github/Authorea model, where you pay for each private paper/repository you set up.
---
It is interesting to note that as a programmer, I do not have that many issues with releasing partial or incomplete source code on Github, whereas as a computer scientist, I do not like to release partial or incomplete proofs at all.
It might be because the academic community cares about reputation: if you get a reputation as a researcher that releases incorrect proofs, your career is over. In programming, as the names are attached to project much more loosely, such negative reputation is rare and people care much less.
This is called peer reviewed publication... It has happened for a few centuries now. It's not perfect and it can certainly be improved but it does exist.
> LaCour said he couldn’t find files containing the survey data and indicated that he would write a retraction; when he didn’t, Green sent off his. Science hasn’t yet ruled on whether it will retract the original paper.
Sure, the paper's author may want you to know that it was all faked... but Science stands by what they printed, no matter what.
If you read the report, the retraction letter was sent to Science, presumably by email, on May 19, 2015.
I'm okay with Science giving some due process here. Contacting the primary author and other parties, the peer reviewers, discussing it with their editorial staff, and so on. The world isn't going to end because Science hasn't yet made a decision on May 21 about something sent to them on May 19. The odds are exceedingly high that a retraction will be issued.
Your characterization of Science "standing by what they printed no matter what" is a gross distortion of the facts here, as Science has done no such thing.
> Your characterization of Science "standing by what they printed no matter what" is a gross distortion of the facts here, as Science has done no such thing.
I have to disagree; I think jacquesm has the comment closest to the truth.
My comment is a minor distortion of the facts -- later in the article we get this:
> Science tweeted that it was assessing the retraction request and would in the meantime publish “an Editorial Expression of Concern.”
Here's the thing. Science isn't doing any research itself. Once a paper's author has admitted the data was faked, there is no role for Science to play; they can't look into the past and determine that it was actually real. They've acknowledged receipt of the message, and they're willing to print a notice that the paper is a lie -- but they don't want to retract it, because that would damage their record.
> The world isn't going to end because Science hasn't yet made a decision on May 21 about something sent to them on May 19.
They have made a decision. They're going to print an unofficial warning. Without more pressure from the public, they will not print a retraction.
Remember, Green is the primary author. He contacted them; for them to contact him is somewhat superfluous.
Green is not the primary author. Lacour is the primary author. Green is the older, more established, secondary author; he's supposed to be playing a sort of mentor role to the primary author. He has sent a letter outlining his concerns about the paper, but he didn't do the work himself in the first place.
Lacour has not indicated he supports the retraction request and it is extremely reasonable for Science to contact him about it and hear what he has to say before proceeding.
Yes, committing to print a warning letter is a decision. The claim I responded to, "Science hasn't yet made a decision on May 21 about something sent to them on May 19", is clearly false. They have made one. They've purposely deferred a different (though closely related) one. Why? What do you think could conceivably come to light that would make the Expression of Concern appropriate, but a retraction by Science inappropriate?
Groups tend not to move as quickly as individuals because groups require discussion. Science probably will retract it, but they need to discuss it first, and they likely have protocols that they follow.
I really wish I could downvote a direct reply. There is one paper. Green, the primary author, retracted it when LaCour, the grad student, failed to do so. But just because the author says "this paper is full of lies" doesn't mean Science will admit it.
Retracting the paper would for Science be an admission that they have a less than stellar peer-review process and would cast a negative light on anything else Science has published. Never mind that not retracting the paper and publishing the retraction makes them look even worse.
I don't think that is fair to Science. They did not say what you said, and they are a group, so it's reasonable to expect them to take time to deliberate.
Imagine that a scientist designs a very intelligent experiment to test a plausible hypothesis - she has an excellent plan to analyze the results (pre-registered, to avoid p-value hacking) - and after doing the experiment, she finds no evidence of the effect that she was looking for. Why should the result (which is the only thing the scientist is not in control of) determine the reward given to the scientist? Funding should be given to investigate a particular phenomena, not to find high impact results.