Vigilante is the wrong term on so many levels for her work. For starters it implies illegality, slipshod at best due process, and a proper authority. Elisabeth Bik is essentially being a good citizen - doing exactly what she should as a reader calling out improprieties.
I wish her luck in her endeavor and note that she proves what actual journals could do to stay relevant as a service when true distribution costs aren't there - vetting the data and giving a reason to pay for the data already peer reviewed by others.
>Vigilante is the wrong term on so many levels for her work
Agreed, her work is independent peer review, an important part of science and the fundamental way science should work. The entire body of science should be constantly trying to disprove facts and remove falsehoods. Journals shouldn't have a monopoly on this. A rigorous peer review is a rigorous peer review whatever the source.
She is doing peer review. There is nothing new about this. The publish or perish culture is new and it's eating away the foundation that our civilization is built on.
If you can't trust science to be accurate you can't trust engineering to work. If you can't trust engineering to work planes fall out of the sky, ships sink and buildings collapse. After that you're back to the 17th century.
We should have a lot less science done better. The current number of people being graduated as scientists is completely unsustainable. The number should be cut down by 10 to 100 times for the current funding.
I disagree, the numbers of scientists isn't the problem. The more the merrier. There is certainly enough work to be done.
We just shouldn't expect every scientist to publish papers and use that as basis to determine their success and qualification. If you optimize on that metric, you would write on the same topics as your peers to generate the needed recitations that allegedly proove your ablility.
You cannot industrialize science where at the end is a number that determines scientific advancement.
Is is bad enough that research and economic interest are intertwined, it just plainly shouldn't be the case. But we should look for workable compromises.
I was agreeing 100% to this post until I hit the last two sentences.
WTF, are you out of your mind?
Are you suggesting we should allow less people to graduate in order to fix the problem? Sorry for the ad-hominem but this it absolute folly.
If there’s too much competition for funding the solution is not to further limit access to scientific profession but to increase funding for Chrissake!
The mere fact that there’s so many graduates is literally living proof that there’s an enormous, yet wasted potential for scientific research and discovery.
I don't want to speak for the OP, but maybe they meant we should be OK with failing people. At least in the US, if you show up to class, attempt to do the homework, and take the exams you are pretty much guaranteed to get your degree. I see this in graduate programs as well (although not the top programs like CMU or Standford). I even saw it when my wife went through med school - many of her peers partied, did horribly, even got caught cheating, but because they were spending 250k on that education no one ever failed out.
I think we should set standards and not be afraid to uphold them. This will probably result in fewer scientists, but maybe it will become clear where we are failing them earlier in their education and allow us to improve that process.
Actually I would argue this is also a thing in top programs (maybe moreso than lower ones) due to grade inflation to improve the perception of those schools.
I see so many people follow the academia career trail not because they have a longing to do research, but because they have no idea what else to do with their lives and their professors think it's a good idea.
Maybe culling the herd a bit would improve its quality. Maybe being forced out of the profession for misconduct would help prevent it. Maybe the bar needs to be raised a bit for entry so there's less competition and less pressure to cheat.
> The mere fact that there’s so many graduates is literally living proof that there’s an enormous, yet wasted potential for scientific research and discovery.
I don't see a proof in this statement - all I see is a belief. Many people believe that a sheer number of people engaged in professional science and requisite "publish or perish" job description is the cause of decreasing quality of scientific publications. Increasing the fraction of population with PhD have to run into Bell Curve limitations at some point.
How do you decide which scientists to pay? Pay them all? Then everyone becomes a “scientist” but does no work and draws a free paycheck.
Pay the ones that produce good work? What is good work? Right now, that is decided by number of papers. That metric is gamed to death, that is what publish or perish culture did. Choose any other metric, it will be gamed too.
What if the PhD meant something such that having it was enough? That was all you needed, then you would just get paid to do science. Such a gatekeepeing would require graduating a lot fewer PhDs. How to do that is open: admit fewer? Or psd fewer? If the latter, maybe master out a lot more? Anyone that makes it through gets a guaranteed paycheck, but almost no one makes it through.
Where you are mistaken is that there are a large number of graduates because there is a large demand for research. There isn’t. A given professor needs N papers to get tenure. Say that N = c*k for c researchers and k papers. N is so large that c must be >2, so for the professor to get tenure, they need assistants. Where do they come from? Graduate students. What do those students do when they graduate? The professor doesn’t care because they have tenure now.
Research funding has been decreasing over time, which means N increases, which, counter-intuitively, means c increases, which starts a negative feedback loop because the amount of researchers competing for funding increases while the amount of funding available decreases, still further increasing the number of graduate students needed. That explains what you see.
>Are you suggesting we should allow less people to graduate in order to fix the problem?
Yes of course. As per "The Mythical Man-Month"[1] adding more people to a project doesn't speed id up - quite the opposite, slows it down.
Right now we have huge overhead in training and communication. Low s/n ratio follows. You can fix that gatekeeping science a bit better - pick more talented, better educated, harder working candidates, and train & finance & supply them better.
>Right now we have huge overhead in training and communication. Low s/n ratio follows. You can fix that gatekeeping science a bit better - pick more talented, better educated, harder working candidates, and train & finance & supply them better.
None of that. A lottery of everyone who can pass a minimum literacy exam in their field.
Looking at the luminaries of any field to a very good approximation none of them are "more talented, better educated, harder working candidates", in fact being lazy, combative and self taught is usually a better predictor for ground breaking discoveries. The people who you would pick are already the ones in academia and they are the ones who created the replication crisis.
It brings to mind one larger trend I noticed with many smaller "fringe" groups but by no means limited to them - a fallacy that just because they personally cannot understand something it must be an unsustainable house of cards.
It is understandable why their heuristics could make them think that even in the absence of any egocentric self justification.
It need not be the Gold Standard people, or those who believe in a pending vague apocalypse or collapse. Some insist that fuel injection systems with computer control are "less reliable" in spite of needing less maintenance and fewer replacements than older fuel dispersal systems.
It resembles a "monkeysphere" sort of concept like a stereotypical communist refusing to see that the disappearance of every banker would cause real and major damage or an ideological counterpart objectivist not seeing that a complete disappearance of bueracrats would also cause real and major damage.
This isn't to say that current systems don't have deadweight, dysfunctions, and things which could be redone way better without the weight of inertia, legacy, and vested interests. The current cure may in fact be worse than the disease but that the cognitive "here be dragons" interferes with understanding a whole system and its alternatives.
I was doing a PhD in high energy experimental physics until I finally notice that the majority of my peers were not doing science, they were padding a resume.
Today we could stop doing high energy physics, both theory and experiments, for 50 years and we would not actually retard the progress of the field. We lack the engineering to build experiments better than the ones we have for any sane price and we are sitting around counting angels on pins. The brainpower that we are wasting on un-rigorous mathematics could be much better spent in any number of fields.
I dare you to read 100 papers in any field that you are familiar with and tell me that 95 of the papers aren't there just because someone needed to publish something. I've talked to people in astronomy, genetics, mechanical engineering and computer science and they all say the same thing.
We don't need more science, we need better science. The simplest way to do that is to do less of it. We won't even be hurting progress because you will be able to trust it, unlike today where if you try and synthesize an experiment from three papers you're pretty much guaranteed it won't work because the papers are either fraudulent, p-hacked, flat out wrong or not even wrong.
> We don't need more science, we need better science.
Exactly. We have to increase signal to noise ratio. I'm trying to stay current with progress in Computer Vision and the amount of junk being published is overwhelming. I'm pretty sure it's the same in many other disciplines.
Ex-prof here. The whole publish or perish mentality is a big problem. How many good papers can a researcher produce a year? Way less than they are expected to in the academic world.
I suspect it’s more to do with the publish or perish and funding dearth combination.
Departments need a blockbuster discoveries just to stay open, it’s obvious where the incentive goes.
Also, none of the underlings will ever think of questioning - let alone reviewing - the results. Their career, their debt repayment plans, their income would be railroaded the moment they stop to think at what they’re doing.
In this Mad Max scenario who’s going to make science and who’s just coasting in survival mode?
Yeah - a vigilante commits violent crimes to punish crimes that go unpunished by the authorities. This is... a person using above board methods to expose junk science.
Vigilante, in its current mainstream use, means a vigilant person who enacts justice without authority. The justice part is not in the etymology of the word, however. Vigilante, structurally, just means someone who is vigilant.
I think calling a watch-dog or an independent peer-reviewer a vigilante is not inappropriate. It's a stylistic choice. I think it's important in literary education to be able to distinguish between what a word is and what cultural baggage it has (how it's used). Of course, where a word ends and its baggage begins is debatable, but it's a good debate to have.
If you have not watched Peter Boghossian and James Lindsay on the Joe Rogan show you should. Very entertaining and shows how ridiculous these publications are.
Do you think Sokal-type pranks are legitimately exposing pretend science? It seems to me, the strongest conclusion you can make is that some journals are over-eager to publish studies with edgy titles or themes. This might bring into question the average level of scholarship in a field but it doesn't directly challenge whether a field is legitimate.
Moreover I don't think a journal with a title like "Gender, Place, and Culture" would classify itself as scientific. Its website certainly doesn't do so [0]. You can't expose pretend science if there's no pretension of science.
No. It's not to expose pretend science, but pretend science publications/magazines.
I believe (No, I don't have evidence) that fields of 'X - studies' have people performing research with rigorous practices. Yet, those holding to standards are marred by nonsense publications from people who disregard standards.
Edit:
This is enabled by magazines printing without... you know, Review.
Academia is almost entirely based on trust and reputation, which we're seeming to discover is not a useful heuristic if your end goal is a net gain in uncovering the truth of the phenomena around us. If you ask me, credibility should be based on reproduction of results rather than reputation of the author, name of the sponsoring institution, journal title, or a vague "peer reviewed" badge. New papers should be by-default untrusted until several reproduction attempts have been successfully executed. This would incentivize authors and scientific institutions to produce science of quantifiable quality.
Granted, my knowledge is limited to physics and chemistry, which I know are small potatoes compared to the life sciences.
With that said, a problem with replication is that a given lab tends to gear itself up for one or a small number of research programs that could span years or decades. Experimental apparatus are developed, knowledge and techniques are passed from one student to the next, and so forth.
My thesis project involved more than a quarter million dollars worth of commercial gear, plus a lot of stuff that I built. By the time I was finished, some of my tools were already obsolete.
If one lab publishes a result, another lab would have to gear itself up to replicate that result, which would probably include a capital investment plus a lot of time spent making beginner mistakes.
I don't believe strict replication is necessarily the best or only way to advance science. It produces reliable factoids, but they are still factoids. Physics has made its greatest strides when experimental evidence, that may be riddled with mistakes, supports the development of unifying theories of ever increasing power and accuracy.
Preferable to strict replication might be to let researchers study overlapping domains, so that several projects attack the same problem, but possibly from different angles.
Those who do the replication are also very unlikely to get published, both in success and failure.
My impression is that replication efforts often happen when one group tries to build on another group's work and they get frustrated enough to retry the original assumptions.
That's a problem of the current system we have, not a fundamental issue like the one he's portraying. It's one of the problems we need to solve if we want science to be better.
Totally agreed. Please fund us enough to do replication at that level.
(Although, at least in my field, replication is gives less bang for the buck than improved measurements. If I measure a thing, and then you measure it with 10x higher precision, you're not just replicating my result, you're moving the field forward.)
The issue is not on the science side, but how results are communicated to the general public. Administrators tend to add as much hype as possible, and reporters strip out all the important details.
Is this assessment (reviews over papers) based on your own experience?
As a scientist I would say quite the opposity is the case, reviews are sloppy in citations, per editorial guidelines have to be written in a positive optimistic tone, and often overstate the claims of the cited articles.
I am not suggesting they are more or even nessisarily as accurate vs individual papers. Rather, they demonstrate the untrusted nature of individual papers.
Personally, I often find them a useful starting point on a topic. At best they capture the field at a moment in time, at worst their near useless. However, that’s just me not everyone in every field.
I've gone through the phase of distrusting individual papers and relying on reviews.
Only to again realize that reviews are also often vehicles of bias perpetuated by the authors where they subtly amp up papers by themselves and their "clique" of researchers.
Only solution really is to just read as much as possible, be as critical as possible and never trust authors to interpret their results with full honesty.
Every fraudulent paper adds to the scientific noise rather than to the scientific information.
Not only are they wasting their own funding, they are also wasting other people's time and money who often can't afford to ignore prior work. At the very least such papers come up in peer review.
And apparently, fraudulent papers can get cited quite a lot in practice!
Her powers of pattern recognition must be extraordinary. I know that enhanced pattern recognition has been associated with savant syndrome [0]; I wonder if her abilities are worthy of scientific study? Or maybe it just seems incredible to me because I'm particularly bad at this task.
Also, she has a Patreon [1] which currently only brings in $40 a month. Just thought I'd bring that to people's attention.
One of the reasons I'm no longer a scientist: as a postdoc I was called on to read a paper about the functional impact of deleting genes in yeast, take their published data tables, and make sense of their results. The paper was fairly straightforward to read (a rarity) and the data was easy to access. I wrote a bunch of scripts to parse the data and ask questions (this was ~20 years ago).
The paper described experiments where the scientists created 6000 yeast clones (yeast has 6000 genes), where each clone contained "a deletion of a single gene" and they showed which genes, when deleted, had fatal effect (IE, the yeast could not grow as a viable organism on yeast medium). The data table was a list of genes that had not been previously known to be "required", and when deleted, would cause the organism to not grow (IE, it died, so you could assume the gene's protein product was required for life).
I was asked to make all sorts of tables showing that the genes that got removed belonged to certain classes, or at least had enriched population of certain classes.
Since I've always been interested in overlapping genes, I instead spent time scripting some range intersection queries on the data tables (IE, assuming all genes are intervals on a line, find the genes whose intervals intersect). What I found was, for each gene they reported as a novel important functional gene, it intersected an already known important functional gene.
So, the most likely conclusion is their table was a bunch of false positives- they deleted gene X, but also truncated gene Y, where Y was already known that if it was truncated, would lead to loss of organismal viability.
I wrote a nice letter to the authors explaining what I had found and never heard back (I also dropped that project, explaining to my PI that it was unlikely the data was of high quality). A year or two later, the authors published a new paper explaining how gene overlaps had important functional significance in yeast...
If I were rich, I'd fund an institute, but instead of trying to discover new things, I'd find a bunch of data scientists who went around looking for low-quality papers and properly letting the community know (while publishing all work so that people can verify for themselves the quality of the paper).
> Within her manual search of more than 20,000 pieces of biomedical research, 4% contained manipulated images.
Shocking. And even if she reports the lies, they are not likely to be actioned:
> Despite Bik’s work finding these manipulations, she estimates that only 30% of those papers have been corrected or retracted
And she does it off her own back in more ways than one:
> Living off savings, Bik reckons she has about a year’s leeway to work on her image manipulation sleuthing and hopes to find a way of monetizing her expertise as a science misconduct consultant to journals.
and:
> she writes and reports under her own name. In some ways, then, Bik is a surrogate for so many. She posts and reports publicly, often after being tipped off to papers from those who aren’t able to do it themselves
It's shocking but imagine you are a PhD student, you need 4 papers out in 4 years but you are operating in a field where your results are only accepted when they are positive and new (sometime even only when the confirm a certain theory!). It feels really unfair as it is essential a probabilistic process whether you will get a PhD or not, it depends only to a certain degree o how had you work and how smart you are. This needs to change.
There are moments when you feel like giving up, for example when you receive a DNA construct that has been proven to work in some Science paper. You set up your experiment and for 1.5 years you wonder why it does not work for you. And you can only conclude that the Science paper didn't in fact produce a working construct, you start to see in the images of said paper that indeed, they may be coincidental and not the result of a working protein. And it sets you back 1.5 years. Or, you write around it, creatively as you explore the borders of your own ethics.
> It's shocking but imagine you are a PhD student, you need 4 papers out in 4 years but you are operating in a field where our results are only accepted when they are positive and new.
Get the fuck out of this field. Those stupid things keep on being only because some people are willing to be abused. When a lot less people enroll for those programs maybe some change will happen. Or maybe those students could start rioting.
So you would be the brave one, the one that will tell his professor: F U, I wont stay for 3 more years on this slave wage, you're going to have to make me a PhD based on my hard work that amounted to a set of results that you deem unpublishable.
This is one of the strongest arguments why scientific publications should be open. The more people who have access to the science papers and have the opportunity to find issues, the stronger our scientific foundation will be.
If you look at arxiv though, the result has been the proliferation of a ton of really low quality, garbage papers. I wonder if a you could crowdsource the review with voting and discussion a la stack exchange style. That might actually be a good solution, particularly if you have to get vetted as being knowledgeable and constructive in order to vote.
Open access doesn't necessarily imply free and not peer reviewed like arxiv. There are plenty of good, peer-reviewed journals that are open access - they are certainly not free though ($1-2k when I was publishing several years ago).
There are also peer-reviewed “arXiv overlay journals” that are very cheap to run and consequently cheap for authors. For instance, Timothy Gowers (Fields medalist, prominent figure in the movement against big publishers like Elsevier) announced[1] another one of these last week:
> I am excited by the business model of the journal, which is that its very small running costs (like Discrete Analysis, it uses the Scholastica platform, which charges $10 per submission, as well as a fixed annual charge of $250, and there are a few other costs such as archiving articles with CLOCKSS and having DOIs)...
I use a site called "ArXiv Sanity Preserver" - it's mainly for machine learning papers based on its current audience, but it provides pretty useful filtering based on user reviews.
In any case, you can still estimate the quality of a paper with a minute of research into the authors. This is a good idea even if it's in a journal.
There's nothing wrong with ArXiv. There's a lot of bad stuff on there, but the overall quality there is higher than the average published work, which is dominated by the many many low-tier journals out there.
This kind of thing gets too much hype. I mean, these people are just sloppy, and catching them out is just a matter of how much effort you want to put in. Generally in any field you get to know quite quickly who isn't trustworthy, and largely ignore their papers. If only journal editors had the same information. What is a bigger problem is papers which seem important but actually reveal nothing due to statistical errors, mistakes you can't ever find without getting the data and redoing the analysis, irrelevant or contrived model systems, inadequate controls, inscrutable analytical methods etc
Don't get me wrong, ideally every paper would be trustworthy, but working scientists are well aware of the issues, and aren't waiting for a posse of armchair crusaders to save them.
If you aren't in the field, then it's difficult, although perhaps inconsequential.
Looking at images of crystallized materials and interpreting all similarity between small regions as proof of scientific fraud is a bit dangerous though... If I had to guess I’d say it’s more likely that that electron microscope image in Bik’s embedded tweet is genuine than not.
This whole situation is a farce and an embarrassment to science though. The motto should really be “adhere to the rigorous standards of science, or perish”.
Someone replied to the tweet mentioning exactly that, but she seems pretty sure that it's fraud. I'm closer to believing her because I'd bet the original authors would challenge her if called out incorrectly.
This cant be a serious article. This woman worked for ubiome and they recently had to shut down because they were being investigate for fraud by the FBI!
That's what I was thinking. Manually reviewing 20,000 papers is simply an astonishing feat. Heck, even searching a single paper for anything that looks like anything else in the same paper (possibly rotated, scaled, or alpha blended) might be beyond me. But it's the kind of task a computer vision algorithm should be able to do very well at. In fact, I Googled it and found (what else) some papers about it, e.g.:
I think of this scenarios like she discovered every-time someone uses a line like "studies show", the only thing the study shows is that the author wants a bump in clout.
There's a great list of examples halfway down the page. For example:
"A growing body of evidence..." (Where is the raw data for your review?)
"People are saying..." (Which people? How do they know?)
"It has been claimed that..." (By whom, where, when?)
"Critics claim..." (Which critics?)
etc...
I wish her luck in her endeavor and note that she proves what actual journals could do to stay relevant as a service when true distribution costs aren't there - vetting the data and giving a reason to pay for the data already peer reviewed by others.