These citation games are at least decades old. The PageRank algorithm came out of the literature on ranking academic papers by citation, and many of the SEO manipulation techniques work in both cases. I'd guess it's being done on an increasing scale given the article.
As a semi-related phenomenon, I also feel like mathematicians are culturally prone to under-citation. I don't just mean that nobody cites older mathematicians like, say, Leibniz or Euler, for techniques they invented. But pretty frequently I'd read papers that really seemed like they were heavily indebted to a paper that came only a few years before but didn't appear in the 5-10 citations. Maybe there it wasn't mandatory to cite those papers, but it made it hard to chain backwards into the literature to get more context on methods and ideas. And sometimes it came off as the author trying to make a tool appear ex nihilo. I'm sure this happens in other fields too.
My understanding is that citations are used to avoid having to argue for the truth value of a proposition, not to give credit, nor to establish some kind of chain of trust. A sort of "I assume this is true, if you want to argue about it take it up with this guy". If two papers are very similar in structure, but neither uses the conclusions of the other to support the truth of a dependent claim, then I don't see why the older paper should be cited by the newer paper.
Academic papers often include an introduction which summarises the context in which the paper is written. Consider https://arxiv.org/pdf/2201.03545v2.pdf where the paper says "For many decades, this has been the default use of ConvNets, generally on limited object categories such as digits [43], faces [58, 76] and pedestrians [19, 63]. Entering the 2010s, the region-based detectors [23, 24, 27, 57] further elevated ConvNets to the position of being the fundamental building block in a visual recognition system."
This shows the authors are familiar with the field, avoids inadvertent plagiarism, allows them to make it clear precisely how their contribution contributes to knowledge, shows people from funding bodies that this is a cutting edge and important field of research, and points readers in the right direction if they want to see earlier ideas, or ideas in a more applied context.
If among those 92 citations should be a few papers from your boss and your colleagues, nobody will see that as unusual - as long as they're at least marginally relevant.
This is a lot less true in mathematics research, where a question existing and not being trivial is enough motivation to investigate.
I would in fact argue that the trend, in "applied" fields, of justifying the importance of a piece of work by pointing out that a lot of people are doing similar work is in fact self-fulfilling. That makes it somewhat useless as a measure of importance.
Scientific context should be critical, not just descriptive.
Your claim about applied fields is not correct. When you do your research, you survey what other people have done in some subarea. You determine what has not been done. And then you do it.
When you write your paper you say exactly this in the introductory context. Here is what other people have done, here is a problem not yet solved in this area, we solve it. It saves the reader an enormous amount of time by not having to do a literature review themselves to see how the paper fits into the larger thrust of the field, and what the novelty is.
How can you justify the novelty other than by comparing to other people's work?
Which claim do you disagree with? The claim that the trend I describe exists, or the fact that it is bad?
Note that what you describe is what I advocate: you explain that the question exists and hasn't been answered. This is not an argument that is based on the volume of work that exists.
It is also an argument you cannot (as an author) be trusted to make. Even if you cite everything that has been published that is tangentially related to your claimed contribution, there is no way a reviewer will know all of it, and no way a reviewer will be able to go and read all of it. So they can't determine whether your claim of novelty is correct unless they already know the entire field. The only defense against this is to encourage crisp and clear descriptions of claimed contributions (to knowledge or practice) and violently reject any overinflated claims. It is not to include an entire survey paper in the introduction of every piece of work that pushes the state of the art forward.
It does mean that the average paper is less accessible to the non-expert. It also encourages the regular publication of surveys whose role is solely to critically and exhaustively compare recent advances, and of textbooks whose role is to describe historic developments and their context. This is not something every paper should be doing.
> I would in fact argue that the trend, in "applied" fields, of justifying the importance of a piece of work by pointing out that a lot of people are doing similar work is in fact self-fulfilling.
There is one filter on this, though: you have to get funding for the work, which is a different process than getting a paper accepted. There is of course some overlap, as "people are working on this so it must be important" can be useful in arguing for funding, but funders at least potentially have other criteria in play too.
You would argue for the importance of research verbally, or in some kind of proposal document, though. There's no need to include part of that document in the final publication.
“My understanding is that citations are used to avoid having to argue for the truth value of a proposition, not to give credit, nor to establish some kind of chain of trust.”
People do this in science, politics, and church. It’s a call to act only (or mostly) on that person’s testimony. An act of faith. It’s both a red flag to watch for and a necessary aid for people.
A red flag because it’s often a sign of fallacy. I default on looking that person up along with folks that might disagree with them. Every success adds credibility. Failures mean I’ll default in a different direction if I hear their name come up. Might even dig into what other claims they have which are spreading.
Those sources who keep getting the job done in specific areas can graduate from red flag to reliable enough to default on. Only in that area for claims not to far from what they’ve been good at. If outside or a big claim, still double check. Also, be able to provide verifiable evidence of why you trust the source.
Are you suggesting it's a red flag for science/mathematics to build upon previous results, to "stand on the shoulders of giants" (as it were)? What are you proposing, that researchers should reproduce all of the results leading up to the current state of the art in their field before embarking on their own original experiments?
Edit: So, if anyone wants to build on Fermat's last theorem, it's not enough that they cite Wiles' paper proving it, but they should have to re-derive the proof themselves?
I answered your first question in my original comment. People can build reputations with good science. I’ll answer the other one now.
I think humans operate on faith in others’ testimonies way more than actual science. As a follower of Jesus Christ, this doesn’t bother me since I believe we learn the truth in multiple ways. However, science is rooted in empiricism which says any belief must be proven, independently reviewed by skeptics, and we keep using it while it seems correct. Dissent is always allowed in real science since our models (beliefs) might be useful but incorrect on some level. Whatever survives the most review with the most utility is what we should keep using.
What happens is often different. People believe something just because it was reported somewhere or had a name on it. That’s pure faith (or argument from authority). Many works aren’t just unreplicated: they can’t be replicated. Many institutions have political biases affecting their work. Many don’t allow dissent on some topics. People good at one thing often fail at another. I also see big jumps in extrapolating a small success to future, larger successes.
While common (even on HN), none of that is the scientific method in action. It’s more like a mix of science, unproven dogma, speculation, political maneuvering, and a popularity contest. The ratios of these vary by field and institution. Don’t call anything science or “proof” unless it fully follows the scientific method. And builds on premises that did, too.
It's somewhat field dependent. I would generally expect citations to be used for both purposes. That is, if someone is proposing work substantially similar to existing work, they should be expected to cite that other work and identify differences with the new work since novelty is a key component generally (although not always) required for publication in my area.
Qualifications of truth are often written into the paper too tho, so e.g. in biology paper people will cite studies, but then explain their mitigating circumstances and context and how they need to be balanced with other studies.
So while I agree that generally citations are a statement of claim and the reference given allows people to see if that statement of claim is actually supported, within the paper itself good authors also explicitly weigh the "truth value of a proposition".
You're arguing against practice in mathematics based on practice in an empirical field, though. Truth in biology is fuzzy in a way truth in most mathematical fields is not.
My impression is that math is strange and disconnected. When I was a grad student the (large) physics department had a colloquium series that brought in speakers that were interesting to the whole department despite that department having major splits between astrophysics, condensed matter, accelerator physics, high energy physics, biophysics, etc.
There was a chemistry colloquium that I went to occasionally, partially because I had some friends in the chemistry department who were into the same quantum chaos stuff I was into, but it was clear that a certain faction of the chemistry department showed up there and the rest of them went to other talks.
The math department didn't have any lecture series which was of broad interest to the department.
Physics brought in senior people who had something interesting to say at the colloquium, but more than half of the people the CS department brought in for their colloquium (which I attended a lot later when I had a software dev job at the library) were job talks and often the people did not know what they were talking about and it could get embarrassingly bad. I was lucky to see Geoff Hinton speak before he got famous but there were times I would team up with "the meanest physicist in the world" (for real, not my evil twin) to ask questions at the end that would call out the weakness of the speaker because the audience was just not holding therm accountable.
All of the math departments I’ve been involved with have had department-wide lecture series. Actually, all of them even had regular lectures meant to apply to the entire college.
The situation is out of control. I've also seen that during peer review a reviewer may give a list of publications they 'suggest' for the literature review, sometimes completely unrelated fields. If you choose not to, you will likely see your paper rejected.
The truth is that almost all papers published across almost all fields cannot be replicated, full stop. For those that can, the results rarely match up, indicating that the tests are chosen to boost the performance of the method.
If you want to boost citations legitimately, purposefully leak your own paper to get past the publisher pay wall, make it colourful and accessible (if it's too dense they are unlikely to understand it), give examples on how to replicate your method, the abstract is likely the only part that will be read so make it count.
> I've also seen that during peer review a reviewer may give a list of publications they 'suggest' for the literature review, sometimes completely unrelated fields. If you choose not to, you will likely see your paper rejected.
You are severely misrepresenting how things work here. A reviewer doesn't accept or reject a paper. A reviewer gives their response to an editor and the editor makes a decision. The editor knows who the reviewer is, and has access to the reviews and the author's response to them.
If the editor sees the reviewer has suggested adding bunch of references to their own work, and has suggested rejecting the paper after the authors refuse this then the editor is going to know what is going on. They aren't completely stupid.
> purposefully leak your own paper to get past the publisher pay wall
Many papers in maths, physics and computer science appear on arXiv before being submitted for publication. In the fields I have worked in you can replace the "most" with "pretty much all".
I guess it depends on the journal. If you're publishing in some low tier or even predatory journal then yeah there might be problems like this with the editor but in good, respected journals the editors usually do their jobs properly.
I dunno. When I was in grad school I regularly would find papers in Annals of Mathematical Physics that would go on for 50 pages and have critical errors on page 17 and 45. I did a calculation like that which I know what right at the equation-by-equation level because I made unit tests for everything, but the whole thing was still useless (published in my thesis but not Annals) because the coordinate system we were using converted a periodic orbit into something like a spiral so the we couldn't compute a topological factor because there was no topological invariant...
I've published in some well-respected, but not highest-tier, journals (think impact factor ~ 5-7), where this was absolutely the case. You could very often identify the reviewer by who the corresponding author of the twenty papers they asked you to cite was. Perhaps it's field dependent, my field (chemistry) was not always as, shall we say, ethical as some of the other fields.
… I thought people in physics were pretty sportsmanlike but my understanding was that biologists would stab you in the back for the slightest advantage. I would complain about how hard it was to make it in physics but in biology it seemed almost impossible to get established since young researchers just couldn’t get grants period.
I don't know about that. IIRC the creation of HoTT was because someone was notified that one of his important papers from earlier in his career had a non-trivial flaw in it.
To be sure, reproduction is a bit easier in mathematics because you normally don't need exotic equipment, just time. However, that doesn't mean that people are necessarily taking the time to double check everything with sufficient rigor.
A number of papers on formal verification reported that restructuring their specifications for proof revealed problems they missed in the specification. Other works found through empirical testing that the specs didn’t match reality. Many theories in physics changed that way.
So, we should assume they’re all false by default until the specs are reviewed, the proofs are checked, and there’s empirical validation. If theoretical, its component theories must be built on observed truths instead of imaginary constructs. Then, we might trust the paper (i.e. mathematical claims).
Who's going to replicate an experiment that takes a $5 billion particle accelerator?
For that matter, who's going to replicate a condensed matter experiment for which a grad student took three years to build the apparatus at a cost of $250,000?
Peer review is an absolute joke. In principle researchers could publish their data sets and reviewers could see if they could replicate the analysis but they don't do it because they know if they tried they'd get different answers, see
>>> For that matter, who's going to replicate a condensed matter experiment for which a grad student took three years to build the apparatus at a cost of $250,000?
That was my thesis. To make it worse, the laser I used became unavailable as I was finishing up. My work was adopted by 2 labs but required extensive adaptation.
> Who's going to replicate an experiment that takes a $5 billion particle accelerator?
In the case of the LHC, which I assume is what you're talking about, this is why we have 4 pretty much independent detectors around the LHC looking at different but at least partially overlapping stuff. The Higgs boson was detected by the ATLAS detector, but wasn't considered to be "discovered" until it had also been detected at the CMS detector.
> Peer review is an absolute joke
That strongly depends on the field and the journal in question.
If they flew out the peer reviewer to interview the researcher and take a look at their apparatus or if the peer reviewer loaded up a Jupyter notebook and rechecked the analysis there might be some value in peer review. If peer review is just somebody looking at the paper superficially than it’s of very limited worth.
When I wrote my first paper somebody else had done a very similar experiment at another university and we wound method up peer reviewing each other’s papers. Turned out we were using the same bogus data analysis that was being used in 1000s of other papers. I was vaguely aware of this when I wrote the paper but did not feel psychologically safe at all in an environment where the American Physical Society claimed a PhD had a 3% chance of having a permanent career in the field.
Someone who was later to become a titan in the field was at our lab and was crying some nights because he had no idea where he was going to get his next postdoc wrote a paper about this problem more than a decade later after he had gotten tenure in a statistics journal that physicists probably won’t even read.
That sounds (as I said before) quite specific to the field you found yourself in. That isn't how things work everywhere.
For a relatively recent example - in my field of quantum information someone found in the course of reviewing some newer work that there was a mistake in one of the big foundational papers that this newer work was based on. They shared it with some of their colleagues and together they did some work finding out exactly where the error was and what the impact is on work that is based on it.
My point is essentially that it isn't a big shock that this error wasn't found by peer review when the foundational paper was published in 2008. Even when it is working as well as possible peer review can only detect most errors, not all errors. However when your field is functioning "normally" and everyone does their jobs properly this doesn't matter because the errors that get past peer review get detected by other people building on the work later. On the other hand if your field is not functioning properly and people are knowingly doing stuff that is wrong then peer review can't help you. If the status of the field is so broken that people who know stuff is wrong are still pushing for it to be published then you aren't in a place where peer review can save you.
My opinion is you are incorrectly putting the blame on the (highly imperfect) process of peer review here where it doesn't really belong. Peer review as we perform it today sucks for many reasons, but it isn't to blame for the problems you saw.
where there is a systematic process to search the literature and select papers with usable results. I've never seen more than 50% of papers get accepted and the other day I saw one where they found 80 invalid papers and 2 valid papers. It seems to me that peer review isn't accomplishing very much if it is passing through papers that can't be built on. For that matter, why is this work even being funded?
Yeah I wouldn't expect it to counter such problems. Peer review is by its nature review by other people in the same field. If you have a field where "bogus" methods are being used in thousands of papers then you aren't going to be able to get things reviewed properly.
This isn't an indictment of peer review as it is practised in every field.
I’ll say a good field is an island in an ocean of bad fields.
Look at hep-th. If in 100 years people conclude there is no evidence for supersymmetry and extra dimensions something like 90% of that field will be “not even wrong”. There is so much of the drunk looking for his lost keys under a lamp. I mean, there is a huge industry in 2-d gravity where it is possible to calculate things but gravity doesn’t attract in 2-d so the line between that and Newton’s apple is unclear. I’ve talked to astro-ph refugees who think that current theories about accretion disks are “not even wrong”.
Then there are all the folks who tell me firewalls make no sense because there is no such thing as the event horizon (locally) but from the viewpoint of people who fall in it is a tiny difference getting killed at the apparent horizon meanwhile the nonsensical “black hole information paradox” (sorry, no unitarity = no quantum mechanics) still gets breathless and insufferable articles written about it just because somebody famous for their disability made a bet. If young people were able to get established in the field you wouldn’t see people so fixated on bad ideas.
Yeah if one tried to find rot in academics, I think applied particle physics is one of the furthest from it. The scientists are super allergic to unfalsifiable theories that they start to doubt themselves when prediction is off by nanometers.
> For that matter, who's going to replicate a condensed matter experiment for which a grad student took three years to build the apparatus at a cost of $250,000?
If the results are interesting to anyone and there is more science to do, people will eventually replicate (or fail to do so, like with Majoranas). If not, then why would we care about replication?
Did you write many papers in experimental particle physics? I did, and in my experience, peer review was not a joke. And results from different experiments really are used as validation. Citations are not just used as a metric, but helps to leverage prior work.
Or computer science, at least when the code is also published.
(That said, performance measurements in most papers are complete junk. Most code written by academics is missing trivial, obvious optimisations which could completely change any comparative benchmarking results.)
I'm not sure that the average CS paper really tested the algorithm they thought they were testing. That comes from a lot of commercial projects where people weren't using the algorithm they thought they were using (even simple algorithms) and a few cases of working with code from CS researchers.
There was that time I got a C program from a famous ML researcher, tried to run it and it segfaulted, I pointed gdb at it and realized it was crashing before it got into main(). Understanding why was an educational experience (don't allocate a 8 GB array you never use on a 32-bit machine... It worked for the researcher because he had a 64-bit machine) but it did not increase my faith in the code quality of academic projects.
Some CS researchers are brilliant programmers but the fact is that ordinary devs make a living writing code and CS researchers make a living writing papers.
Allocating a big struct for everything and not using pointers is one of possible techniques of dealing with data in C programs. Anyway having a requirement of 8G RAM is not a good example of something "can not be replicated".
I know all about alternate ways to deal with memory, if I was just programming purely for fun I'd write assembly and not waste time moving the stack pointer around and other meaningless activities associated with calling conventions.
It's a sign of poor quality code.
He probably was using the array for something at one point, quit using it, and never bothered to delete it.
And of course it is C so there are no unit tests despite it being the kind of code that is eminently unit testable (if it wasn't written in C, not like you couldn't write unit tests for C but who does?)
> Most code written by academics is missing trivial, obvious optimisations
I agree with this, but I would pose it the opposite way. Academics shouldn't be prematurely optimizing their code: they don't have the resources for it, nor will their results be representative across all hardware. I routinely see (and publish) papers with novel cryptographic protocols where the reviewers will say things like "compare the running time of your algorithm to a state-of-the-art production system that Apple spent $50m developing and optimizing for their silicon, using hand-tuned assembly." We're not going to win in that benchmark, and we don't have the resources to even compete. Moreover, the resulting code wouldn't make for good reference code. But that doesn't mean we shouldn't publish our work, so that e.g., Apple (or other resources) can begin the process of optimizing it for production.
> Academics shouldn't be prematurely optimizing their code
While I agree with that in principle, the question of whether or not a given algorithm is fast enough to be practically usable is very useful information for the reader of your paper. Do I need to read your paper, understand it, implement it myself, and only then figure out that its 3 orders of magnitude slower than existing algorithms? I know its a lot of work, but as the reader I don't want to do that work either.
I've spent the last few years working on a novel collaborative editing algorithm for text editing. If the algorithm was orders of magnitude slower than existing CRDTs, it would never get adopted. And I want my work to be used. I want to use it, but again - only if its fast enough to be practically usable. I ended up writing a very highly optimised implementation of the algorithm - which took a massive amount of time - to answer that question. For the paper, I spent another couple of days writing a much slower reference implementation that people can read to understand how it works. (The optimised version is thousands of lines, and the reference code is a few hundred lines).
I understand how much work and expertise is needed to do things like this, but I still want someone to do this work. Even if its "We wrote a straightforward implementation of <competing algorithm> and <our algorithm> in plain C code. They perform at roughly similar levels of performance <see chart>. We suspect highly optimised versions of both approaches would also perform at roughly similar levels of performance."
It's extremely true in physics. I've published in (good) physics journals.
Computational papers: Authors leave out details because they don't want to give up the goose that lays the golden eggs. They broadly describe the technique, ("solve this PDE computationally"), and anyone who wants to replicate has to craft his/her own solution. If they fail to replicate, the authors just say "your simulation was poor and probably numerically unstable".
This was the norm.
Experimental papers: Same thing. Experimentalists are inventors. They don't buy off the shelf equipment - they build their own. They don't provide details of how they built it in their papers.
At conferences, PIs would openly discuss whether they "believed" a highly cited paper. When I was a new grad student, this troubled me. Why should belief play a role? Just replicate it! A year later: "Oh, of course these studies can't be replicated."
It's one reason I left academia. I was doing theoretical/computational physics, but the whole game reminded me of English literature: The value of your work was always seen with subjective eyes. Your paper could be rejected simply because the reviewer doesn't "believe" it.
> Why should belief play a role? Just replicate it!
If you are talking about theory or numerical simulation papers, just replicating the calculations in the paper are not enough to believe the conclusions. There are many assumptions that go into any calculations, in terms of how much of spherical cow you treat reality as. Changing those assumptions will change the conclusions. The assumptions are not always clearly stated, and even the authors might not realize all their assumptions. It can sometimes take many years or decades for people in the field to figure out how a paper was wrong.
So yes, experts do have to depend on their subject intuition to determine how much they believe some surprising result. Not to mention, if you go through historical papers in your field you will inevitably come across decades long fights where one group is coming to one conclusion in their paper and other is coming to another, before the situation is resolved one way or the other.
Except it's the same problem for experimental papers. My thesis was based on a highly cited experimental study. My role was to "support" that paper with numerical simulations. And yet, at conferences, people would openly discuss if the original, highly cited paper, was fact or fiction.
The guy built/fabricated a device, and reported interesting measurements. Want to know if it's true? Build it yourself! Except when you do and get differing results, there will be quibbling over whether you built it properly. There are always details left out of the original paper that can be appealed to. The original author is king of the field, and people will simply ignore your contrarian findings.
For theoretical/numerical results, it is for sure valid to criticize assumptions. But in the longer time frame, the question shouldn't be "Are these assumptions valid?" but "Is there experimental evidence to support these calculations?" With a lot of the papers published (including mine), I can assure you that no one will ever be able to construct an experiment to (in)validate my results. And that was true for probably over half of the theoretical/numerical results in my field. We were all publishing things that most of us believed could never be connected meaningfully to the physical world. We're not talking string theory or high energy stuff - more like material properties.
The goal was to publish, and convince others. Not to understand reality. Hence, more like literature.
If that's true it's only because most people don't understand these pure maths fields. But their results are generally very useful to the applied mathematicians (whose results are and should be used in most other fields) and in this way there is still an impact although it's not always reflected in the citations.
The fact is that these pure mathematicians could almost surely write the more applied papers but don't because it would feel to repetitive. You don't need to read Euler's work in order to indirectly benefit from it. It's the same for a lot of pure math.
As an applied mathematician, I'd say this depends on the field. Pure maths more often than not comes with assumptions that make the theory more elegant, but by doing so they also drastically limit its application to real life scenarios.
This is not even close. Pure math is often complete clueless about how it applies. For example Developing algorithms to compute properties used in the most basic pure math is a massive pursuit.
If anything, I'd think replication has gotten easier with the advent of open source and the expectation that top papers will include git repos which make it easy to reproduce. The advent of docker, locked-version-dependencies etc have all contributed as well. 10-20 years ago, we didn't have this - but I also didn't see * widespread * problems even then: if someone published a result saying that technique <x> led to <y> performance gain, I thought it usually did?
(that said, computer science research doesn't have the same money on the line as medicine and the "perish" part of the publish-or-perish equation isn't the same when the fallback for failed academics is still six figure salaries at tech companies)
What are the odds that in 2044 we will still be able to run the code published with CS papers today? If it's pure Java SE or ANSI C with no dependencies then it will probably still work, but anything more complex tends to decay pretty rapidly.
> The truth is that almost all papers published across almost all fields cannot be replicated, full stop.
Isn't it ironic that we can't replicate your assessment?...
I personally have no idea how much papers can be replicated or not, but replicating an experiment would not guarantee that the idea/conclusion in the paper is sound (ex: paper extracting some data, observing a correlation and concluding it is an implication).
I do agree that it should be made easier to replicate for the fields where this is possible, but there are many other problems.
this is overstating the case. Throwing out all publications is ridiculous. Yes, many fail to replicate, but not nearly all. A larger concern may be that many fail to generalize (e.g. mouse cancer cures fail to work in humans).
It's true that it's hard to ignore all of the citations a reviewer suggests, but that's part of the role of an editor, and the response to reviewers (while being a frustrating exercise) can make it clear what is and isn't an appropriate citation, which serves as communication to the editor. Yes, this process is frustratingly low bandwidth
This has been an issue with university rankings for a while. A lot of top US universities engage in this practice too - anecdotal, but I've heard a lot of professors force students to add/remove some citations, or even add their names to the list of people who worked on a paper to help the numbers for their university.
It would be good to see what the criteria is for deciding if a journal is "to be taken seriously". I imagine for example that Chinese or Arabic language journals wouls be published and citsd in journals of those languages. That doesn't necessarily mean that they arent to be raken seriously in the field, it's just that they aren't Western publications.
Regarding inflating the number of authors, it is especially bad in medicine, where I've observed a lot of names being added to papers for "political" reasons, despite the "author" playing no role in the paper.
Some journals now require an "Author Contributions" section to at least partially address this issue.
The worst part of this in the biomedical field is the conferences. Thats because sometimes you get toddlers with an advanced degree and a chair position picking the conference presenters, who will unilaterally reject or accept people on the grounds of whether they like them or see them as a competitor, even within the same department, no regards to what the poster or talk might be. At least with journals you have the editor who can sometimes mediate a hotheaded reviewer dispute in a level headed manner.
I've seen it go in the other direction too. Groups deliberately not citing other competing groups because it might help them. It's like the other groups don't exist.
I've observed this in multiple AI niches. In some cases I've emailed people saying they ignored very similar work and failed to cite it, and in at least some cases, they were apologetic and said they would update the arXiv version. Although of those times, they do that 50% of the time. Kind of tells you that the reviewers at top AI conferences themselves aren't that familiar with the breadth of the literature.
To be fair, there are so many publications in fields such as AI that it's really hard to stay on top of things, sometimes even within your own specific subarea. I'm not saying that to give reviewers a total pass, but I think it's reasonable that sometimes a group of reviewers might miss a relevant paper.
"All metrics of scientific evaluation are bound to be abused. Goodhart's law [...] states that when a feature of the economy is picked as an indicator of the economy, then it inexorably ceases to function as that indicator because people start to game it."
Note that the article specifies that these cartels mostly feature participants from bad universities in countries that have really whacked-out academic reward mechanisms that arose because some powerful central government force that doesn't understand research productivity came up with a dumb way to incentivize it.
Prestige is not a failsafe guarantor of quality, but it kind of is, in the sense that somebody with a reputation to protect is going to be more careful about coming across as "unserious", and doing stuff like this is a good way to get a bad reputation quickly.
We can safely assume MIT has a much higher quality output than King Abdulaziz University back in Jeddah.
In general, State Flagships and Ivy Leagues+Ivy Tier Privates are largely comparable research output wise.
The GP is right about prestige minimizing bad practices. Look at how diluted Northeastern's brand has become by trying to artificially inflating their prestige [1]
>powerful central government force that doesn't understand research productivity
[1]*
I'm sure they understand it. But their goal in doing this isn't really research productivity (though it would be a happy byproduct) it's to obtain prestige, or its illusion. And/or be able to tell the people of your country how great you're doing. "Our country is a rising powerhouse in field X, in the top 3 in the world and still rising!"
It's internal propaganda. And externally targeted towards populations of other countries unfamiliar with the academic publishing system, which is most people. So for example the vast majority of people in the US that hear these claims by China, which are then echoed in click-bait headlines or simply taken at face value after a journalist checks something like the various HCR lists.
It can even be propaganda, not from China, but from other countries. One of the studies was funded by the US State Department to an Australian group, thus obscuring a primary influence on the results of the study.
Then even a single newspaper article can spark a dozen others citing it, propagating the message. All of these directly or indirectly rely on citations to support their headlines [2]
>isn't really research productivity (though it would be a happy byproduct)
Or the goal is research productivity and citation gaming is just the unhappy byproduct. Yes some motivated actors to playup PRC capabilities (ASPI/US think tanks) exist, but unless you think every western index on science and innovation, controlled for quality of citations (like Nature) are incentivized/coordinated to carry water for PRC's massive increase in research productivity in the last few years, then the parsimonious answer is PRC research productivity has gotten really good and world leading in some domains.
Which should not be surprising because what OP fails to understand incentivizing citation gaming is a smart way to incentivize output, FAST, and at PRC scale output quantity has quality of it's own. If system spams research, and start off with only 2/10 good research, but leader is 3/5 good research, then PRC has nearly caught up. Refine to 3/10 and there's parity. Refine system to 4/10 and PRC leads. Emphasis being fast because PRC started focusing on seriously improving tertiary and R&D/S&T ~10 years ago and the goal was to develop capabilites fast by customary overproduction. You don't get that by slow careful growth, you do that by incentivizing easy KPIs everyone in sector can coordinate around and the product of said KPI is ENOUGH good research after byproduct of citation gaiming. Having a bunch of chaff leftover to get wheat is the point.
I wonder what percentage of academic work is based on cargo culting and practices like these. There are more "scientists" than ever and yet we dont really have a quantitative increase of people that are as productive as Euler, Einstein, Newton, Neumann or whoever you want to pick as a luminary
These big names are usually promoted beyond their actual achievements [0] because of the tendency to make stories about great people who changed the world.
The reality is murkier. Huge names like Lorentz and Poincare had worked on relativity before Einstein. Gregory and Barrow had proved the fundamental theorem of calculus before Newton, not to mention Leibniz's work.
If you mythologize the past it's easier to look around and wonder why we don't have immortals like Zeus or incredible warriors like Achilles anymore. But the truth is science always proceeds by steps that look small at the time and it's often only in retrospect that things seem amazing and unprecedented.
Semi-relatedly, I used to attend a seminar with a well-known Russian mathematician who would often chime in with Russian/Soviet priority over historical results mentioned by visiting speakers. The cold war created two mathematical cultures that had limited contact. So famous European and American mathematical results from, say the 40s to the 90s often had Soviet versions worked out independently in journals nobody here had ever heard of and written in a language they can't read.
So this is all just a way of saying that empirically, the big kahuna theory of mathematical development seems more fiction than reality. And it should be treated with skepticism when you hear things framed in those terms.
[0] Not that their achievements aren't great. They are all incredible. I just mean they seem more incredible than they are if we forget all the other incredible achievements. Shoulders of giants etc etc.
It sounds to me like the who of science doesn't matter as much as the where and the when. There are ideas floating around in the ether, and the discovery of these ideas generate more ether for future discoveries. To a large extent being a scientific luminary is being in the right place at the right time.
Which doesn't seem at odds with the grandparent post, given that we shouldn't expect throwing more people at the problem to accelerate discoveries.
> It sounds to me like the who of science doesn't matter as much as the where and the when. There are ideas floating around in the ether, and the discovery of these ideas generate more ether for future discoveries. To a large extent being a scientific luminary is being in the right place at the right time.
I think it's really an empirical question how it works. My own take is that it takes both. You need the ideas floating around and you need an obsessive synthesizer.
A music example might be Bob Dylan. Folk music could have been a passing local fad, but Dylan studied all of it, distilled it down, and produced a version that seems different in kind than what came before. Without Dylan, we could look back on that era as being like the swing revival of the 90s. Just a period where a kind of music was popular again but with no real lasting change.
Similarly, there could be areas of math that are ripe for a Newton figure to come in and clarify everything. But that field could wait 100 years in the backwaters of math journals before anyone took notice. So you can't directly jump from the ideas floating around to the synthesis, you do need some sort of figure who will read and understand everything and put all the pieces together into a coherent piece of writing that people can read.
Realistically, a lot of them are probably training AI algorithms to get people to click on ads and questioning their life choices.
But math keeps advancing at a good clip, there are tons of brilliant young people. Physics IIUC is largely gated on waiting for new experimental results. Einstein, Poincare, Newton and many others arrived at a time when the experimental physics data was perplexing and theoretical advances were needed to clear up the picture.
The reverse is true now if I understand. People are just waiting for more funding to build larger accelerators or whatever big expensive astronomy things they need to test the many predictions we have from the various mathematical attempts to unify quantum mechanics and relativity.
Biology seems to be moving pretty briskly. I can't speak to chemistry or any of the other sciences.
EDIT: I should add that when I left academia about 8 years ago, the mood was that academia was increasingly hostile to smart people who just wanted to do research. The move toward running universities like businesses has pushed out a lot of people who would be candidates for the next Einstein or Poincare. Poincare happened to be independently wealthy, so wealth may again be more of a factor than it was during the last century.
Problem is as always greed for fame, power and money.
Labs and researchers are given funds based on "impact".
As we all know, when a metric becomes the goal, the metric gets gamified.
This is very hard to fix.
Gonna give you an example. As soon as you move in any direction in science you're entering a niche. Pretty much everything is its own niche where there's a limited number of people really able to review your paper.
No, there's not thousands of experts dedicating their life to helicoidal peptides interactions with metal layers. It's an extremely small number of people. They all know each other and are gonna review each other's papers regularly.
Solar-powered water splitting to produce hydrogen and oxygen? Again, extremely small club.
Perovskite? Graetzel solar cells? Bigger club, but still, at the end of the day the relevant luminaries are a handful again.
You think it's much different for quaternions or advanced complex numbers analysis?
And those small clubs set the rules and standards, and there is really not much way to have oversight on those small clubs.
Plenty of terrible science is published every day, I can confidently say 90 to 95% of experiments in Physics or Chemistry are impossible to reproduce (numbers people get by taking outlier results or even more often by tweaking the data).
When it comes to softer sciences like psychology it's even worse. It's crap.
It's sad, but I think the world desperately needs a *free* alternative to the biggest publishers out there like Nature or ACM or all these things. That free alternative has to be funded by universities and governments globally. This entity should only allow papers that present clear experiments that are reproduced elsewhere or under supervision.
This would greatly enhance the quality and reliability of papers.
Yeah its very true, the incentives are screwed. I think you're touching on the key difference here. Someone like Euler would have had to been forced not to do math, it was his passion. But you have all these "scientists" going through the motions doing "science" because its their job. Passionate individuals will always be rare and the worse part is that the current scientific system in place is tailor made to root these passionate individuals out. A good modern example is Grigori Perelman.
It feels like it should be. Maybe the following is just the naivety of a novice looking at experts they don't understand.
Learning numbers, integers, rationals, and real numbers, introduced so much power with each step, to the point that every child is expected to know the basics about real numbers. Maybe not what they are theoretically, but the basic ability to do something like .5 * pi or root(2)^4. They likely never dig any deeper, most don't really get the idea of never ending never repeating decimals, but they are able to work with them on a simple level. For most fields, an introduction only requires this level of math.
When one does move to the complex numbers, it opens up far more possibilities. Fourier transform is everywhere, to the extent many use it without having the math to understand how it does what it does. More complex problems, and I mean outside of the field of math, only have general solutions if you allow for complex numbers to be used. These are difficult, mapping from C to C requires 4D to represent and thus are much harder to visualize, but people still struggle through it.
When I realized that complex numbers weren't the end, but the second step in a tower of infinite height, I wondered what otherwise unsolvable problems needed higher levels from that, just like how many problems needed complex numbers. The difficulty working with them grows, though somewhat given C to C is already 4D, we have already reached the limit of 3D viewing power.
Yet they are rarely used, and higher levels used even less. Maybe there is something fundamental that makes them less useful, far weaker than complex numbers. But from a viewpoint of a novice, I find that surprising.
> Learning numbers, integers, rationals, and real numbers, introduced so much power with each step, to the point that every child is expected to know the basics about real numbers.
The amount of additional power you get from real numbers is zero, and in fact it isn't possible to work with them because specifying an irrational number takes an infinite amount of time and space.
They give you additional power in the sense that they let you give names to the exact answers to certain equations, but not in the sense that you can manipulate those equations any better than otherwise.
> When I realized that complex numbers weren't the end
But... they are the end. That's the point. We call that the Fundamental Theorem of Algebra: every solution to every equation takes the form of a complex number.
Quaternions do not generalize from complex numbers in the sense of the progression of the numeric tower from integers to rationals to reals. That is known to stop at the complex numbers. They generalize from complex numbers in the sense of functions that scale and rotate two-dimensional images. Quaternions are a way of expressing functions that scale and rotate three-dimensional images. They are familiar to most people who have taken calculus under the name "vector cross product".
(And for completeness, while there are octonions, the full generalization of "functions that scale and rotate arbitrary space" is matrices, which again people tend to be familiar with.)
All I said is that as soon as you release a paper you fall in a niche of some topic (unless the paper is super generic) where the number of people with the expertise allowing them to review it is super small.
Quaternions or complex analysis where just examples where, even if not strictly experimental and more theoretical, if you're pushing the boundaries of knowledge the number of people able to review what you wrote and concluded isn't that big.
Like the Eiffel Tower or Burj Khalifa, the algebra tower only has 4 levels (Real, Complex, Quaternion, Octonion) before it shrinks to degeneracy, and level 2 (Complex) has the best balance of dimensionality and power. Adding dimensions adds power, but linking the dimensions adds constraints on allowed structures.
I know that at least in AI/ML there are reproducibility challenges, I even had a course during my masters where we had to reproduce a paper. Not perfect, but some disciplines try to address that. AI/ML has a nice feature of publishing almost only in open conferences/journals, there are not that many closed publishing venues in this field.
I think that also has to do with the amount of complexity. Henri Poincaré is often said to have been the last man who knew all of math. Today, you need research teams across the world working on a very narrow topic to make tiny progress iteratively in the best case. I think the time when one man could bring us a century forward like Newton or Einstein is gone.
> I think the time when one man could bring us a century forward like Newton or Einstein is gone.
I'm certain his contemporaries said the same about Pythagoras.
You're not wrong about how much more complex things have gotten, certainly.
But you don't know what you don't know. Another Ramanujan or von Neumann could be right around the corner.
Just look at all the wasted potential in our education system - it's impossible to quantify just how much effort is profoundly wasted.
This article points directly to that wastage. We've tolerated an academic system that's hated by just about everyone except publishers, in an era where publishing is as close to free as it could be.
There is not room to make education a million times more efficient and effective for one single person. A brain has extreme physical limits. Computers can do it because they can combine millions of computers to work together, limited only by asymptomatic constraints like the Amdahl's law.
“Although Pythagoras is most famous today for his alleged mathematical discoveries, classical historians dispute whether he himself ever actually made any significant contributions to the field.
[…]
The Pythagorean theorem was known and used by the Babylonians and Indians centuries before Pythagoras, but he may have been the first to introduce it to the Greeks. Some historians of mathematics have even suggested that he—or his students—may have constructed the first proof. Burkert rejects this suggestion as implausible, noting that Pythagoras was never credited with having proved any theorem in antiquity.”
What mathematical discoveries? You just quoted wikipedia saying that (1) he's most famous for his alleged mathematical discoveries, and (2) he is not alleged to have made any mathematical discoveries.
Yeah the quantity and complexity of what you need to know now is daunting. But even so I dont think Einstein for example had to know the entirety of physics and math in order to produce a groundbreaking revolution in physics.
No offence to them of course, but they already took the lower hanging fruit. It's hard to come up with fundamental axioms and laws when they have already been found. And there are definitely people around that have the same potential.
I worked as a research assistant for a bit over a year after the graduation and I had an experience that put me off. We submitted a paper and got some review feedback and one of them contained something like this
... this and that paper also worked on this research area and contain this and that stuff you can citate ...
The prof I was working with told me that this anonymous reviewer was probably the writer of those papers and asking for citation.
Why did that put you off? This is a very common thing for reviewers to do. It is useful for you, because it makes you aware of other people who work in the same problem as you, leading to future job prospects, etc. But more importantly, it is useful for the readers of the article!
The alternative is several independent communities of researchers that work separately on the same problem, but do not acknowledge the existence of each other. Now, that would put you off!
It depends on context. Often reviewers are chosen because they work specifically in that area, and of course they know their work best. I've often done this as a reviewer in appropriate situations, where I point out closely related work that I've or others have done.
Of course, I've also seen EDITORS email me saying they would like me to cite some papers from their journal after I submitted my paper. That was definitely a turn off.
> The prof I was working with told me that this anonymous reviewer was probably the writer of those papers and asking for citation.
I think this is mostly just a kind of fun way of complaining about reviewers. Having been a reviewer for many computer science conference, I've often had the names of other reviewers visible to me. Most requests for citations are not for the reviewer's own work, and truly irrelevant citation requests seem to be pretty rare.
I do perceive journal/lab-centric fields to be worse about this, though.
well, if it's that transparent... I still wonder what to do with a handful of slightly off, but relatively good papers suggested to us. 2/5 would augment existing citations, 3/5 are related but not really relevant... No author is shared between them.
So this essentially appears to be the SEO blogspam playbook oozing into academia? I wonder if any (likely well considered) techniques developed to combat this context may be able to be backported to help reclaim the usefulness of general internet search.
> I wonder if any (likely well considered) techniques developed to combat this context may be able to be backported to help reclaim the usefulness of general internet search.
For a level of an individual researcher this is easy, you just learn what is true and what is false. For a level of the whole society the battle is already lost, your (highly valuable and competent) solution to make a gigachad search just will be either lost or gamed or sued/prohibited.
SEO spam is worse than ever. The only thing Google figured out is to raise the cost of website above 0 such as by requiring ssl and well written English.
It's not just cartels, but also self-citations. For example, the original preprint of this paper [1] has 5 self-citations, with the version accepted to EMNLP increasing that to 10. This kind of behavior is sadly pretty common now and when you mention it to conference organizers they just shrug.
I have seen all sorts of crooked behaviour in academia, such as reviewers demanding authors to cite their unrelated papers. However, this article's main premise seems to be predicated on another assumption that is also terrible for academia - elitism. Specifically, this quote
> There were people that published in journals that no serious mathematician reads, whose work was cited by articles that no serious mathematicians would read, coming from institutions that nobody knows in mathematics
is really bad. It suggests one can be a 'serious' mathematician only if you went to XYZ universities and published in ABC journals. This elitism is the main reason why people try to game the system in the first place, because they know that they can't advance in their careers unless they build that perception of prestige. The single minded focus on prestige is then what smothers the real science.
I've been told who and what to cite in articles (even if it was only very tangentially relevant) as a Ph.D. student in HCI as far back as 2014. I also had to put my advisor as a co-author on papers that he didn't even bother to read...
I’ve never been a part of academia but I know it’s been fraught with all sorts of fraud for the last few decades (maybe forever idk) but this is a new type of weirdness to me. Is anything trustworthy is that space?
It's not really fraud though. If we get paid to upvote each other, the problem is not that we upvote each other but that there's someone paying us to do so.
same. I read through publications of my institute for my MS (EU, pretty big tech-focused university) and they pretty much also cite in circles (same institute and other related institutes of our university).
Now I realize why it's called a field. I always thought of a physical field but this makes way more sense. If there's enough division in a group, it is considered a field.
This is also why I think paid review won't work although many in academia advocate that. If paper mill remains a problem now, review mill will be much worse when journals start to pay reviewers.
That was my impression whenever paid review was mentioned. Nonetheless, it is also possible that more who dislike the idea were not voicing themselves.
I find it kind of ridiculous and depressing, that we are playing these kind of games in academics and risk careers of bright mind and scientific progress. What a waste of potential and time.
That's the example I often use. There are papers in prominent journals that draw negative citations.
A related but much larger problem is that citations are a measure of the number of researchers working on a particular problem and nothing else. There was one topic in particular that had many researchers making small contributions, but all of those papers cited all the others, simply because they were written on that topic. Maybe the methods weren't the best or the conclusions weren't robust, but the only reason to cite a paper is because it is prior literature. It's not like you can refuse to cite papers because you believe they are of low quality.
This seems like the early days of search engines and seo poisoning, where random sites were referencing others to make them climb up the search results.
Nah its older than SEO shenanigans. Interestingly quantitative research evaluation was an inspiration for page rank, so it make sense that they have similar issues.
Speaking as a researcher, I think the whole citation business was invented by and consumed by administrators. Researchers don't need citation numbers to discover new works and evaluate people. They are perfectly fine asking their circle of friends, using journal quality and author fame as a crude proxy, and (most importantly) just read the damn papers and reach their own conclusions.
Administrators needed some objective metric that allow them to make decisions themselves without relying on the personal opinions of researchers, whom they don't trust. This is magnified by rankings that like these metrics for the same reason, publishers trying to promote journals, and database access sellers.
Unfortunately, this has now circled back to the researchers, because the administrators are making hiring / promoting decisions based on these metrics, and people started to game the system. However, I think in many circles, there is a healthy culture of rejecting this nonsense.
What about taking them with a grain of salt? Just because something is approximate that does not mean it is not useful.
I never got this obsession with detailed rankings. Sure "above average" and "under average" makes sense, but position x or x + 1 out of 10k does not make much sense anymore (and by similarity if you are 1 or in the first 1k out of 10k could be similar)
Google has a better university ranking[1], with more experience in withstanding this kind of manipulation and no need to conform to expectations that Ivy League universities must be at the top.
Most R&D is driven by Government Procurement or Commercialization anyhow, which can validate if shit works or not.
A lot of the subfields with a replication crisis are those that aren't exactly "tangible", and as such don't have a quantitative way or vested interest in validating that stuff works as expected.
If I'm researching a compound as a MatSE on a grant from the DoD or Applied Materials, it's on the assumption that tangible results will happen, and any attempt at fraud will be detected once commercialization or productionizing kicks in.
At the end of the day, a field like Pure Math or Anthropology just doesn't have a commercial incentive that forces reproducibility because there isn't a tangible impact.
As a semi-related phenomenon, I also feel like mathematicians are culturally prone to under-citation. I don't just mean that nobody cites older mathematicians like, say, Leibniz or Euler, for techniques they invented. But pretty frequently I'd read papers that really seemed like they were heavily indebted to a paper that came only a few years before but didn't appear in the 5-10 citations. Maybe there it wasn't mandatory to cite those papers, but it made it hard to chain backwards into the literature to get more context on methods and ideas. And sometimes it came off as the author trying to make a tool appear ex nihilo. I'm sure this happens in other fields too.