First is the reproducibility issue. I think I've spent about as much time simply _trying_ to get the dependencies of research code to run as I have done writing or doing research in my PhD. The simple thing is to write a requirements.txt file! (For Python, at least.)

Second, two anecdotes where not following best practices ruined the correctness of research code:

- Years ago, I was working on research code which simulated a power-grid. We needed to generate randomized load profiles. I noticed that each time it ran, we got the same results. As a software engineer, I figured I had to re-set the `random` seed, but that didn't work. I dug into the code, talked to the researcher, and found the load-profile algorithm: It was not randomly generated, but a hand-coded string of "1" and "0".

- I later had the pleasure of adapting someone's research code. They had essentially hand-engineered IPC. It worked by calling a bash script from Python, which would open other Python processes and generate a random TCP/IP socket, the value of which was saved to an ENV variable. Assuming the socket was open, the Python scripts would then share the socket names of other filenames for the other processes to read and open. To prevent concurrency issues, sleep calls were used throughout the Python and Bash script. This was four Python scripts and two shell scripts, and to this day, I do not understand the reason this wasn't just one Python script.

Different scientific research fields are using widely different computer software environment, and have their own habits and traditions. The way a biologist uses programming has no reason to be similar to the way an astrophysicist does: they have not at all experienced the same software environment. It may even be useless to talk about "scientist" in the same field as two different labs working in the same field may have very different approaches (but it's more difficult if there are shared framework).

So, I'm not at all surprised that you observe opposite experience. The same way I'm not surprised to see someone saying they had the opposite experience if someone says "European people are using a lot of 'g' and 'k' in their words" just because they observed what happened in Germany.

One of the most poorly engineered products I work with was created by a few academic CS guys. The core algorithms are sophisticated and ostensibly implemented well, but the overall product is a horrible mess.

The incentives of academia make this obvious. You need to write some code that plausibly works just enough to get a manuscript out of it, but not much else. Reproducibility is not taken that seriously, and "productization"/portability/hardening is out of the question.

It feels like Software Developers are "brainwashed": a good software is a software that is good for what software developers need to do. But a good software is a software that is good for what people who needs it need to do. If the academic people don't need "productization" and still code for productzation, then they are doing a bad job.

Inversely, software made by software developers may be really bad in academic sector, as it is the subject of the article: they overengineer when not needed, they complicate things just for portability or future use that will never happen, ...

It's a bit like if someone says "these cooks who are preparing omelettes are bad cooks: they are mixing white and yolk together, while real cooks who do meringue always remove the yolk. So the proper way to cook eggs is by removing the yolk".

(another aspect that I've noticed is that software developers talk about unit test and code review and are offended if they see scientists not using them properly, but they don't even realise that the goal of these is fulfilled in an arguably more efficient way by other processes that exist in academia. For example, there is sometimes no unit test, but the creator of the algo is also the main user and they will notice directly if a new change has broken something. Or as another example, scientific collaboration often implies several teams working independently and writing their own implementation from scratch, so if team A and team B gets different results from the same inputs, they will investigate and find out the problem in the code)

No, it's like serving the food on the floor without dishes. It doesn't matter whether the food is fancy or simple.

We need not defend the poor practices induced by the corrupt incentives of academia (publish or perish). Ideally, scientific methods and products of research should be highly reproducible (which implies a certain level of quality, in practice).

First, scientific code is exploratory: you start building it without knowing which way will work. For example, you end up with 10 different fit functions, all using different approach (multi-gaussian, kde of control region, home-made shape based on a sum of Gumbel and Gaussian fcts, ...), and during your study, you do your test to conclude which one is the best. Because of that, "building for the future the multi-gaussian fit fct" is just plain stupid: you are spending time for something that have a high probability (>75%) of being totally dropped next week. You also need to build for huge flexibility. For example, having some "god object" allows to pivot very quickly when suddenly you discover a new parameter to add to the equation, rather than to refactorise all the function one by one to add a new parameter of a very specific type to all of them. In this context, the person who codes with strong static typing is not being very smart: in research, their code will change very often and they are just building a tool that will waste their time.

Second, the code is the tool, not the product. The scientific conclusion is the product. What needs to be reproducible is the conclusion. You apparently don't understand the goal of reproducibility. Reproducibility is to prove that 2 persons who redo it from scratch obtain the same results, even if they use different tool. By saying that it is important that a scientist B can just blindly run the software of a scientist A, you just screw the reproducibility advantage: if the software has a bug, scientist B will say "I get the same result, so, it is confirmed" while the result is incorrect. At CERN, people from the CMS experiment and people from the Atlas experiment are FORBIDEN to share their code for this reason. Even reading the code can be bad because it can bias the reader. Then, of course, there is a balance and I recommend to share the code with publication, but the reader should be educated enough to know that the code is the last resort to look at if they have a question when trying to reproduce the result.

Third, the way scientists work and collaborate creates to some extend "intrinsic" code review and unit testing. Several teams build software in parallel to check the same thing, and will investigate by comparing their result. And scientists are the first user of their software, intensively: they will never build a feature that they will not themselves use in practice later. So, some of the good practices of software dev are just not well designed for these situations.

At the end, you are right: serving the food on the floor is bad. The software developer "good practice" are serving the food on the floor because they assumed the dish was where it is not.

Finally, one thing to keep in mind is that some software dev people are just full of themselves with big ego. They love to think they are smarter than everyone else. If they are applying good practices themselves, they love feeling superior by thinking the ones not applying them are inferior and that they are so smart. It's a way of rationalising their traditions, and it's then easy to see plenty of ways that confirm this belief.

Don't get me wrong, plenty of good practices are really useful, and plenty of scientific code is just badly done. But you just need to have a balance: instead of blindly applying tradition just because it flatter their ego, people should just take a step back and think of why X and Y is better in some context and see how it translates in another context.

> instead of blindly applying tradition just because it flatter their ego

I understand this is really the core of your complaint; we agree that reproducibility is important (for more than one reason), but you don't understand software engineering as a practice, and judge it based on strawmen and perhaps bad experiences with poor, ego-driven developers. Every field has big egos, especially highly competitive ones like finance and academia.

Good engineering is absolutely not about "blindly applying tradition".

If you think that it is the core of my complaint, then you did not understood at all.

This element is just there to say that it's easy to not understand the situation of the scientists and to conclude that it's just because they are stupid or they are not doing the things right.

You were the one talking about "poor practices induced by the corrupt incentives of academia", but this is just incompatible with the reality: in region where the "publish or perish" has no impact (and these regions exist, only people who know academia superficially don't realize that), we still see the same software development "mispractice". This is why I'm talking about ego: it is so easy for you to just rationalize different practice as "being somehow forced to do a bad job".

> we agree that reproducibility is important

I don't think you understand what I call "reproducibility". In science, reproducibility means FROM SCRATCH! You know the result, and you rebuild your own experiment. If you copy the experiment of the first author, you will copy the experimental error too.

Reproducibility is one of the reason some of the thing you pretend is a good practice is in fact not ideal in the context of science (as I've said, I myself push for sharing a clean code, but it is just that some of the important goal for usual software development don't exist in the same way in the scientific development context).

> but you don't understand software engineering as a practice

I understand very well why the good practice are indeed very important in the context of the usual software development. Again, you are just rationalising: you see m not agreeing with you, so you invent that I don't understand why good practices are used. It's again an easy way to not challenge what you consider as obvious.

> Good engineering is absolutely not about "blindly applying tradition".

Exactly. People who are jumping on the conclusion "if the scientists are not using these practice, then they are doing a bad job" are bad engineers.

You keep talking about good engineering practice as if these practices are always the correct way to go. Except it is not the case. As given in some example, wasting time on a "clean" architecture on something that the author is smart enough to know that it has a high probability to disappear in one week is not a "good practice". Yet, you are still talking to me as if this practice is "obviously good". It is very good for usual software development, when you have a large set of users and you want to maintain the software for years. But it makes no sense when the software is for the author of the software only and that everyone agrees it would be better to write a proper implementation of the final conclusion rather than to mix the goal and build a robust tool before exploring.

So, yes, you are blindly following the tradition: you just refuse to even consider that maybe what you have adopted in your mind as "good" may not be the best in a different context.

So, it seems more akin to making meringue with yolks. Eventually maybe it will work, but if you knew what you are doing and cared, it would be done better faster.

The criticism though of losing sight of the goal is valid. That happens.

The goal of the software of the scientist is different, so the definition of what is "quality" is different.

The article illustrates that: a lot of "software done following the good software developer practice" ends up being of bad quality for the job, and end up wasting a lot of time.

Another aspect is the context: the iron triangle is also something built for a specific context. Of course a code that contains very very flexible function will have problems after 5 years of development and usage which will lead to drastic decrease of speed. But scientific code should not be used 5 years later (scientific code is to prove hypothesis, once the article published, you should not use this code, because this code, by construction, contains plenty of hypothesis testing that have been demonstrated not useful). So, the reason "speed" is related to "quality" is different in science.

I disagree here. I believe quality is intrinsic to the product. There may be different attributes to the "quality" of the thing that one person may value more than another, but the intrinsic quality is the same. An over-engineered solution is rarely good. Don't use a power-boat to go across a small swimming pool, and don't use a shoddy makeshift raft to across a giant lake.

> The goal of the software of the scientist is different, so the definition of what is "quality" is different.

I disagree here, but I think I see your point. The general goal for everyone (the goal of software) is to accomplish some task. Software is fundamentally just a tool. Software for the sake of software is bad.

This makes me think of an analogy where a person is trying to redo the plumbing on their kitchen sink. Compared to a firm that will do whacky crazy things and leave the situation worse than when they started, and walk away with their bills payed and job half done. Compared to this, certainly a competent novice is better. Though, sometimes it is really important to know about certain O-rings, not only would the "true" professional do the job slightly faster & more methodically than the novice, but they'll know about that O-ring that would only become a problem in the winter. At the same time, sometimes there are no such O-rings, and a simple job is just ultimately a relatively simple job.

To that last extent, hiring software engineers can sometimes be scary. When you get very intelligent people and pay them to "solve complex problems," they do tend to build things that are very flexible, FAANG-scalable, very industry latest, AI powered, for the sake of it lasting 5 years; rather than starting simple - Gall's law: “A complex system that works is invariably found to have evolved from a simple system that worked." Or, those engineers may start there because they know those are the "best practices" without yet experiencing why those are best practices, and when to apply those practices and when not to apply those practices.

[1] http://principles-wiki.net/principles:gall_s_law

You are switching definition: in this discussion, people have defined "quality" in one way, which is not relevant for the academic sector. Then you arrive with a different definition of quality to pretend that what they say is correct.

If you define "quality" as something intrinsic to the product, then the "good practice to do quality work" are not good to do quality work, because these practice leas to shit software (as illustrated again and again by plenty of people working in the academic sector or working close to them, like the author of the article here).

That's the problem in this discussion: "quality is good", "this practice is good for quality because in context X, the important attributes of quality are A and B", "therefore this practice is good in context Y even if the important attributes of quality are C and D".

I'm saying "what you call quality is A and B, and in science, quality is not A and B".

> This makes me think of an analogy ...

I agree with your analogy, but the person who know the O-ring is THE SCIENTIST. The software developers, and the "good practice rules" that they are applying, don't know anything about O-ring, they don't know how to install kitchen sink. Software developers don't know how to build scientific software, they don't even understand that scientific software have different needs and need different "good practice" than the ones they have learn for their specific job, which is a strongly different context.

It's like saying: a good practice in veterinarian is to give product X to the sick dog, so, we should follow the veterinarians good practice when we do medicine to humans, because veterinarians are doing products that help living things feel better, so it's the same right?

Your example with Gall's law is pretty clear: YOU DON'T WANT A SOFTWARE THAT LAST 5 YEARS IN RESEARCH. You NEED, and really need, a software that explore something that has a very big chance to lead to "no, it was incorrect", and that if it leads to "yes, it's correct", will be destroyed after the paper publication (because the publication contains the logic, so anyone can rebuild the implementation in a way that satisfy Gall's law if they want). Gall's law is totally correct, but they are not talking about "research software", they are talking about a different object, and the scientists are not building such object.

Some practice was best because of some issue with 80s era computing, but is now completely obsolete; problem has been solved in better ways or has completely disappeared thanks e.g. to better tooling or better, well, practices. e.g. Hungarian notation. Yet it is still passed down as a best practice and followed blindly because that's what they teach in schools. But nobody can tell why it is "good", because it is actually not relevant anymore.

Scientific code has no superstitions (as expected I would say), but not for the best reasons; they didn't learn the still relevant good practices either.

I see a lot of opinion/taste presented as something more, but I really can't think of superstitions.

Perhaps not superstition but certainly fundamentalist/hype-based thinking.

But I'd say a bit of everything listed in TFA. For instance global variables are the type of thing which makes a little voice say "if you do that, something bad will eventually happen". The voice of experience sometimes say things like that, though.

For example, Snakemake (os-independent make) with data version control based off of torrent (removing complication of having to pay for AWS, etc) for the caching of build steps, etc would be a HUGE win in the field. *No one has done it yet* (some have danced around the idea), but if done well and correctly, it could reduce the amount of code and pain in reproducing work by thousands of lines of code in some projects.

It's important for the default of a data version control to be either ipfs or torrent, because it's prohibitive to make everyone set up all these accounts and pay these storage companies to run some package. Ipfs, torrent, or some other centralized solution is the only real solution.

1. No tests of any kind. "I know what the output should look like." Over time people who know what it should look like leave, and then it's untouchable.

2. No regard to the physical limits of hardware. "We can always get more RAM on everyone's laptops, right?". (You wouldn't need to if you just processed the JSONs one at a time, instead of first loading all of them to the memory and then processing them one at a time.)

Also the engineers' tab has a strong smell of junior in it. When you have spent some time maintaining such code, you'll learn not to make that same mess yourself. (You'll overcorrect and make another, novel kind of mess; some iterations are required to get it right.)

I think a single semester of learning the basics of software development best practices would save a lot of time and effort in the long term if it was included in physics/maths university courses.

Working for corporate R&D, I once received a repo on a flash drive. The team would merge changes manually by copy-pasting.

I should've just turned around and left.

A good programmer has a very deep knowledge of the various techniques they can use and the wisdom to actually choose the right ones in a given situation.

The bad programmers learn a few techniques and apply them everywhere, no matter what they're working on, with whom they are working with. Good programmers learn from their mistakes and adapt, bad programmers blame others.

I've worked with my share of bad programmers and they really suck. A good programmer's code is a joy to work with.

It is easy to learn some coding, not so easy to become a scientist.

To becomes a scientist you must write and get your PhD-thesis approved, which must already be about scientific discoveries you have made while doing that thesis. Only people with above average IQ can accomplish something like that, I think.

It’s a bit surprising you’d have to explain such a basic conclusion here.

I honestly think most skilled tradespeople are more intelligent than me and my PhD holding colleagues.

The vast majority of papers I read on topics I know are complete bullshit. Maybe making a PhD was more elitist before, but now it surely isn't.

If we define "scientist" as anyone who publishes papers, then they have the same problem as software engineering: it's mostly made by juniors.

That being said, most scientific code I've encountered doesn't compile/run. It ran once at some point, it produced results, it worked for the authors and published a paper. The goal for that code was satisfied and than that code somehow rusted out (doesn't work with other compilers, hadn't properly documented how it gets build, unclear what dependencies were used, dependencies were preprocessed at some point and you can't find the preprocessed versions anywhere to reproduce the code, has hardcoded data files which are not in the published repos etc.). I wouldn't use THAT as my compass on how to write higher quality code.

I've found the problems that biologists cause are mostly:

* Not understanding dependencies, public/private, SCM or versioning, making their own code uninstallable after a few months

* Writing completely unreadable code, even to themselves, making it impossible to maintain. This means they always restart from zero, and projects grow into folders of a hundred individual scripts with no order, depending on files that no longer exists

* Foregoing any kind of testing or quality control, making real and nasty bugs rampant.

IMO the main issue with the software people in our field (of which I am one, even though I'm formally trained in biology) is that they are less interested in biology than in programming, so they are bad at choosing which scientific problems to solve. They are also less productive when coding than the scientists because they care too much about the quality of their work and not enough about getting shit done.

Ultimately I’d say the core issue here is that research is complex and those environments are often resource strapped relative to other environments. As such this idea of “getting shit done” takes priority over everything. To some degree it’s not that much different than startup business environments that favor shipping features over writing maintainable and well (or even partially) documented code.

The difference in research that many fail to grasp is that the code is often as ephemeral as the specific exploratory path of research it’s tied to. Sometimes software in research is more general purpose but more often it’s tightly coupled to a new idea deep seated in some theory in some fashion. Just as exploration paths into the unknown are rapidly explored and often discarded, much of the work around them is as well, including software.

When you combine that understanding with an already resource strapped environment, it shouldn’t be surprising at all that much work done around the science, be it some physical apparatus or something virtual like code is duct taped together and barely functional. To some degree that’s by design, it’s choosing where you focus your limited resources which is to explore and test and idea.

Software very rarely is the end goal, just like in business. The exception with business is that if the software is viewed as a long term asset more time is spent trying to reduce long term costs. In research and science if something is very successful and becomes mature enough that it’s expected to remain around for awhile, more mature code bases often emerge. Even then there’s not a lot of money out there to create that stuff, but it does happen, but only after it’s proven to be worth the time investment.

That conforms to my experience

It's tempting to create reusable modules, but for one-off exploratory code, for testing hypotheses, it's far more efficient to just write it.

That's not on them though. That's on the state of the tooling in the industry.

Most of the time, dependencies could just be a folder you delete, and that's that (node_modules isn't very far from that). Instead it's a nightmare - and not for any good reason, except historical baggage.

The biologists writing scientific programs don't want "shared libraries" and other such BS. But the tooling often doesn't give them the option.

And the higher level abstractions like conda and pip and poetry and whatever, are just patches on top of a broken low level model.

None of those should be needed for isolated environments, only for dependency installation and update. Isolated environments should just come for free based on lower level implementation.

For someone to shake these bad practices, they need to fight an uphill battle and ultimately sacrifice their research time so that others will have an easier time understanding and using their codes. Another battle that people trying to write "good" code would need to fight is that a lot of academics aren't interested in programming and see coding as simply as means to an end to solve a specific problem.

Also, another bad practice few bad practices to add to the list:

* Not writing documentation.

* Copying, cutting, pasting and commenting out lines of code in lieu of version control.

* Not understanding the programming language their using and spending time solving problems that the language has a built in solution for.

This is at least based on my own experience as a PhD student in numerical methods working with Engineers, Physicists, Biologists and Mathematicians.

This leads to a split between domain problem solvers, who are driven to solve the field's actual problems at all costs (including unreliable code that produces false results) and software engineers, who keep things tidy but are too risk-averse to attempt any real problems.

I encourage folks with interests in both software and an area of application to look at what Hadley Wickham did for tabular data analysis and think about what it would look like to do that for your field.

Update: I guess I misinterpreted OP's intent here, with "unreliable code that produces false results" being part of the field's actual problems rather than one of the costs to be borne.

He wrote the tidyverse package/group of packages which includes/is tightly associated with ggplot. It is an extensive set of tools for analyzing and plotting data. None of it is can't be done in base or or with existing packages, but it streamlined the process. It is an especially big improvement when doing grouped/apply functions, which, in my experience, is a huge part of scientific data analysis.

For many R users (especially those trained in the past 5 years or so) tidyverse and ggplot are barely distinguishable as libraries as opposed to core R features. I personally don't like ggplot for plotting and do all my figures in base R graphics, but the rest of tidyverse has dramatically improved my workflow. Thanks to tidyverse, while my code is by no means perfect (I agree with all the aforementioned criticisms of academic coding, especially in biology/ecology), it is cleaner, more legible, and more reproducible in large part thanks to tidyverse.

Good APIS, preferably declarative, allows the scientist to write concise code.

Win.

As ever, the best work comes when you're able to have a tight collaboration between a domain expert and a maintainability-minded person. This requires humility from both: the expert must see that writing good software is valuable and not an afterthought, and the developer must appreciate that the expert knows more about what's relevant or important than them.

I do work in such an environment (though in some industry, and not in academia).

An important problem in my opinion is that many "many software-minded people" have a very different way of using a computer than typical users, and are always learning/thinking about new things, while the typical user has a much less willingness to be permanently learning (both in their subject matter area and computers).

So, the differences in the mindsets and usage of computers are in my opinion much larger than your post suggest. What you list are in my experience differences that are much easier to resolve, and - if both sides are open - not really a problem practice.

You can't solve the first 3 issues without having people who care about software quality. People not caring about the quality of the software is what caused those initial 3 problems in the first place.

I guess we see survivorship bias here: the people who deeply care about the quality of their science instead of bulk producing papers are weeded out from their scientific jobs ... :-( Publish or perish.

Firstly, a lot of the programs people were writing were messy, but didn't need to last longer than their current research project. They didn't necessarily need to be maintained long-term, and therefore the mess was often a reasonable trade-off for speed.

Secondly, almost none of the software people had any experience writing code in any industry outside of research. Many of them were quite good programmers, and there were a lot of "hacker" types who would fiddle with stuff in their spare time, but in terms of actual engineering, they had almost no experience. There were a lot of people who were just reciting the best practice rules they'd learned from blog posts, without really having the experience to know where the advice was coming from, or how best to apply it.

The result was often too much focus on easy-to-fix, visible, but ultimately low-impact changes, and a lot of difficulty in looking at the bigger picture issues.

This is exactly my experience too. Also, the problem with learning things from youtube and blogs is that whatever the author decides to cover is what we end up knowing, but they never intended to give a comprehensive lecture about these topics. The result is people who dogmatically apply some principles and entirely ignore others - neither of those really work. (I'm also guilty of this in ML topics.)

I'm not sure what "uninstallable" code is, but why does it matter? Do scientists really need to know about dependencies when they need the same 3 libraries over and over? Pandas, numpy, Apache arrow, maybe OpenCV. Install them and keep them updated. Maybe let the IT guys worry about dependencies if it needs more complexity than that.

> Writing completely unreadable code, even to themselves, making it impossible to maintain. This means they always restart from zero, and projects grow into folders of a hundred individual scripts with no order, depending on files that no longer exists

This is actually kind of a benefit. Instead of following sunk cost and trying to address tech debt on years-old code, you can just toss a 200-liner script out of the window along with its tech debt, presumably because the research it was written for is already complete.

> Foregoing any kind of testing or quality control, making real and nasty bugs rampant.

Scientific code only needs to transform data. If it's written in a way that does that (e.g. uses the right function calls and returns a sensible data array) then it succeeded in its goal.

> They are also less productive when coding than the scientists because they care too much about the quality of their work and not enough about getting shit done.

Sooo...another argument in favor of the way scientists write code then? Isn't "getting shit done" kind of the point?

* "Big ball of mud" design [0]: No thought given to how the software should be architected or what the entities that comprise the design space of the problem are and how they fit together. The symptoms of this lack of thinking are obvious: multi-thousand-line swiss-army-knife functions, blocks of code repeated in dozens of places with minor variations, and a total lack of composability of any components. This kind of software design (or lack of design, really) ends up causing a serious hit to productivity because it's often useless outside of the narrow problem it was written to solve and because it's exceedingly hard to maintain or add new features to.

* Lack of tests: some of this is that the scientist-turned-programmer doesn't want to "waste time" writing tests, but more often it's that they don't know _how_ to write good tests. Or they have designed the code in such a way (see above) that it's really hard to test. In any case--unsurprisingly--their code tends to be buggy.

* Lack of familiarity with common data structures and algorithms: this often results in overly-complicated brute-force solutions to problems being used when they needn't have and in sub-par performance.

This quote from the author stood out to me:

> I claim to have repented, mostly. I try rather hard to keep things boringly simple.

...because it's really odd to me. Writing code that is as simple as it can be is precisely what good programmers do! But in order to get to the simplest possible solution to a non-trivial problem you need to think hard about the design of the code and ensure that the abstractions you implement are the right ones for the problem space. Following the "unix philosophy" of building small, simple components that each do one thing well but are highly composable is undoubtedly the more "boringly simple" approach in terms of the final result, but it's a harder to do (in the sense that it may take more though and more experience) than diving into the problem without thinking and cranking out a big ball of mud. Similarly reaching for the correct data structure or algorithm often results in a massively simpler solution to your problem, but you have to know about it or be willing to research the problem a bit to find it.

The author did at least try to support his thesis with examples of "bad things software engineers do", but a lot of them seem like things that--in almost every organization I've worked at in the last ten years--would definitely be looked down on/would not pass code review. Or are things ("A forest of near-identical names along the lines of DriverController, ControllerManager, DriverManager, ManagerController, controlDriver") that are narrowly tailored to a specific language at a specific window in time.

> they care too much about the quality of their work and not enough about getting shit done.

I think the appearance of "I'm just getting shit done" is often a superficial one, because it doesn't factor in the real costs: other scientists and engineers can't use their solutions because they're not designed in a way that makes them work in any other setting than the narrow one they were solving for. Or other scientists and engineers have trouble using the person's solutions because they are hard to understand and badly-documented. Or other scientists and engineers spend time going back and fixing the person's solutions later because they are buggy or slow. The mindset of "let's just get shit done and crank this out as fast as we can" might be fine in a research setting where, once you've solved the problem, you can abandon it and move on to the next thing. But in a commercial setting (i.e. at a company that builds and maintains software critical for the organization to function) this mindset often starts to impose greater and greater maintenance costs over time.

[0] https://en.wikipedia.org/wiki/Anti-pattern#Big_ball_of_mud

This part I 100% agree with. I adapt a lot of scientific code as my day-to-day and most of the issues in them tend to be making things 100x slower than they need to be and then even implementing insane approximations to "fix" the speed issue instead of actually fixing it

>"Big ball of mud" design

Funny enough this was explicitly how my PI at my current job wants to implement software. In his opinion the biggest roadblock in scientific software is actually convincing scientists to use the software. And what scientists want is a big ball of mud which they can iterate on easily and basically requires no installation. In his opinion a giant Python file with a requirement.txt file and a Python version is all you need. I find the attitude interesting. For the record he is a software engineer turned scientist, not the other way around, but our mutual hatred for Conda makes me wonder if he is onto something ...

>I think the appearance of "I'm just getting shit done" is often a superficial one, because it doesn't factor in the real costs: other scientists and engineers can't use their solutions because they're not designed in a way that makes them work in any other setting than the narrow one they were solving for.

For the record my experience is the exact opposite. The crazy trash software probably written in Python that is produced by scientists are often the ones more easily iterated on and used by other scientists. The software scientists and researchers can't use are the over-engineered stuff written in a language they don't know (e.g. Scala or Rust) that requires them to install a hundred things before they are able to use it.

A vast amount of software is written for research papers that would be useful to people other than the paper’s authors. A lot of software that is in common use by commercial teams started off in academia.

One of the major issues I see is the lack of maintenance of this software, especially given all the problems written in your post and the one above. If the software is a big ball of mud, good luck to anyone trying to come in and make a modification for their similar research paper, or commercial application.

I don’t know the answer to this, but I think additional funding to biology labs to have something like a software developer who is devoted to making sure their lab’s software follows reasonably close to software development best practices would be a great start. If it’s a full time position where they’d likely stick around for many years, some of the maintenance issues would resolve themselves, too. This software-minded person at a lab would still be there even after the biology researchers have moved on elsewhere, and this software developer could answer questions from other people interested about code written years ago.

https://us-rse.org/

This seems like a much better way to spend one's software development time and experience than, say, ad-tech... at least in my humble opinion :)

    >    * Not understanding dependencies, public/private, SCM or versioning, making their own code uninstallable after a few months

Ultimately, if there were a simple way to get data in the correct state in an os-independent, machine independent (from raspberry pi to HPC the code should always work), concise, and idempotent way - people would use it. There isn't. But the certainly could be.

The solution we desperately need is a basically a pull request to a simple build tool (make, Snakemake, just, task, etc) that makes this idempotent and os-independent setup simple. Snakemake works on windows and Unix, so that's a decent start.

One big point is matching data outputs to source code and input state. *Allowing ipfs or torrent backends to Snakemake can solve this problem.*

The idea would be to simply wrap `input/output: "/my/file/here"` in `ipfs()`, wherein this would silently check if the file is locally cached to return, but if not go to IPFS as a secondary location to check for the file, then if the file isn't at either place, calculate it with the run command specified in Snakemake. It's useful to have this type of decentralized cache, because it's extremely common to run commands that may take several months on a supercomputer that give files that may only be a few MBs (exchange correlation functional) or only a few GBs (NN weights) so downloading the file is *immensely* cheaper to do than re-running the code - and the output is specified by the input source code (hence git commit hash maps to data hash).

The reason IPFS or torrent is the answer here is for several reasons: 1) The data location is specied by the hash of the content - which can be used to make a hash map of git commit hashes of source code state that map to data outputs (the code uniquely specifies the data in almost all cases, and input data can be included for the very rare cases it doesn't) 2) The availability and speed of download scales with popularity. Right now, were at the mercy of centralized storage systems, wherein the download rate can be however low they want it to be. However, LLM NN weights on IPFS can be downloaded very fast when millions of people *and* many centralized storage providers have the file hosted. 3) The data is far more robust to disappearing. Almost all scientific data output links point to nothing (MAG, sra/geomdb - the examples are endless). This is for many reasons such as academics moving and the storage location no longer being funded, accounts being moved, or simply the don't have enough storage space for emails on their personal Google drive and they delete the database files from their research. However, these are often downloaded many times by others in the field - the data exists somewhere - so it just needs to be accessible by decentralizing the data and allowing the community to download the file from the entire community which has it.

One of the important aspects to include in this buildtool would be to ensure that, every time someone downloads a certain file (specified by the git commit hash - data hash map) or uploads a file after computing it, they host the file as well. This way the community grows automatically by having a very low resource and extremely secure IPFS daemon host all of the important data files for different projects.

Having this all achieved by the addition of just 6 characters in a Snakemake file might actually solve this problem for the scientific / data science community, as it would be the standard and hard to mess up.

The next issue to solve would be popularize a standard way to get a package to work on all available cores/gpu/resources, etc on a raspberry pi to HPC without any changes or special considerations. Pyspark almost does this, but there's still more config than desirable for the community, and the requirement of installing OS-level dependencies (Java stuff) to work on python can often halt it's use completely (if the package using pyspark is a dependency of a dependency of a dependency, wet lab biologists [the real target users] *will not* figure out how to fix that problem if it doesn't "just work"[TM])

https://dvc.org/

See pachyderm too.

Also, I'm not sure what your issue with ipfs is; If it's 'I saw something something crypto one time' - it's a really poor argument. IPFS works completely independently of any crypto - it has nothing really to do with it. The solution can also be torrent - I don't care too much - it's just possible that IPFS can run with far less resource usage on lower power, etc because it's more modern (likely uses better algorithms in the protocol, deals with modern filesystems better, with better performance, hopefully have better security, etc) and it's likely easier to implement. But it doesn't matter if it's torrent because it would work essentially the same way.

> I'm not sure what your issue [is] with ipfs

IPFS is cool. I played around with it a few years ago. I even had a similar idea during my masters (and then discovered IPFS).

Decentralization is my issue here -- it's not necessary and it would be more of a blocker than a solution.

If you want to host the data on Dropbox, Dropbox becomes a part of the network and is hosted by the community and Dropbox or whatever service you like. The problem is that it is very prohibitive to use services like AWS. For instance, downloading all of Arxiv.org from AWS is around $600 last time I checked. Not every student trying to run an experiment is going to have $600 laying around to run the experiment. But with torrent/ipfs it would be free to download.

It only adds. I don't understand how it subtracts.

1) Requirements change constantly, since... it's research. We don't know where exactly we're going and what problems we encounter.

2) Buying faster hardware is usually an option.

3) Time spent on documentation, optimization or anything else that does not directly lead to results is directly detrimental to your progress. The published paper counts, nothing else. If a reviewer ask about reproducibility, just add a git repository link.

4) Most PhD students never worked in industry, and directly come from the Master's to the PhD. Hence there is no place where they'd encounter the need to create scalable systems.

I guess Nr. 3 is has the worst impact. I would love to improve my project w.r.t. stability and reusability, but I would shoot myself into the foot: It's no publishable, I can't mention it a lot in my thesis, and the professorship doesn't check.

Here you are not improving your time to get out an article, but reducing it for others - which will make your work more influential.

Here's is where I disagree. It's detrimental in the short term, but to ensure reproducibility and development speed in the future you need to follow best practices. Good science requires good engineering practices.

Anyone who spends lots of time trying to make research-relevant code projects with solid architecture / a well designed API / tests / good documentation / etc. is doing it as a labor of love, with the extra work as volunteer effort. Very occasionally a particularly enlightened research group will devote grant money to directly funding this kind of work, but unfortunately academia by and large hasn't found a well organized way to support these (extremely valuable) contributions, and lots of these projects languish, or are never started, due to lack of support.

In practice, most research code (including supposedly reusable components) ends up getting written in a slipshod ad-hoc way by grad students with high turnover. It typically has poor choice of basic abstractions, poor documentation, limited testing, regressions from version to version, etc. Researchers make do with what they can, and mainly focus on their written (journal paper) output rather than the quality or project health of the code.

So it started as a MVC controller function that was as long as your arm. Then it got split up into separate functions, and eventually I moved those functions to another file.

I had some genuine need for async, so added some stuff to deal with that, timeouts, error handling etc.

But I hopefully created code that is easy to understand, easy to debug/change.

I think years ago I would have used a design pattern. Definitely a bridge - because that would impress Kent Beck or Martin Fowler! But now I just want to get the job done, and the code to tell a story.

I think I pretend I am a Go programmer even if I am not using Go!

There was the flawed model out of Imperial College (IIRC) during the early covid days that showed up how wrong this attitude is.

It was so poorly written that the results were effectively useless and non-deterministic. When this news came out, the scientists involved doubled down and instead of admitting that coding might be hard, and getting in a few experts to help out might be useful, actually blamed software engineers for how hard it is to use C++.

Both are real problems. Over-abstraction and over-engineering can be very expensive up front and along the way, and we do a lot of it, right? Under-engineering is cheaper up front but can cause emergencies or cost a lot later. Just-right engineering is really hard to do and rarely ever happens because we never know in advance exactly what our requirements and data really are.

The big question I have about scientific environments is why there isn’t more pair-programming between a scientist and a programmer? Wouldn’t having both types of expertise vetting every line of code be better than having each person over/under separately? Ultimately software is written by teams, and it’s not fair to point fingers at individuals for doing the wrong amount of engineering, it’s up to the entire team to have a process that catches the wrong abstraction level before it goes too far.

Scientists need code to conform to the way they examine, solve, and communicate problems. I asked for an explanation of a particular function and was sent a PDF and was told to look at a certain page, where I found a sequence of formulas. All of the notation matched up, with the exception that superscripts and subscripts could not be distinguished in the code. To a programmer, the code looked like gibberish. To the scientists working on the code, it looked like a standard solution to a problem, or at least the best approximation that could be given in code.

You see the inverse problem when it comes to structuring code and projects: programmers see standard structures, expected and therefore transparent; scientists see gibberish. Scientists look at a directory called "tests" and think of a variety of possible meanings of the word, none of them what the programmer intended.

Most of the “programmer sins” are of the type that more seasoned engineers will easily avoid, especially those with experience working with scientific code. Most of these mistakes are traps I see junior developers falling into because of inexperience.

A considerable majority of the science-non-SWE crowd are de facto incapable of writing more than 100 lines of runs-in-my-notebook code.

Hence, if a change/bug is necessary, it is much likelier to fall under a SWE jurisdiction, and hence is much more likely to be industrial code.

Add to that a further confounder (tiptoeing a "no true Scotsman" here): academia is not a first choice of workplace for strong SWEs.

I never understood nor understand people who nest their inner loops in an entangled mess of hardly distinguishable digits, which is error prone.

Same for method names.

I try to use speaking out loud to some of my methods: What do you do? And if the answer is getValue I believe it needs renaming.

E.g. getString(path) for loadConnectionStringFromDisk(configFilePath), tryConnect(30) for testSqlConnection(timeoutInSec), even the reader now knows what happens here and what input is expected.

After 15 years of writing JavaScript professionally I know that is a lie. The biggest messes are made by the majority of people hired that cannot really program.

My intent is not to put down maintainers of scientific software! It's super cool and super important.

I see the damage a person decades in an industry can do when they cluelessly and energetically start to test and implement a new shiny thing on an industrial codebase.

When the product brings in hundreds of millions a year, there is incentive to patch up the damage so you can have future releases and continue the business. I'm not sure how much resources a scientific codebase maintenance could use just to patch up a mountain of architectural and runtime damage.

The result is a complete inability to program. Most people need really large tools to do more than 80% of the heavy lifting and they just write a few instructions on top of it. The perspective then becomes you need more advanced technologies to do cool things, because everything is too scary or mysterious otherwise.

It appears that you are not even talking about the same problem as the author. You seem to be talking about people who all define themselves as programmers, some of whom have more experience than others. The author wasn’t talking about new-hire programmers, they were talking about experienced physicists, chemists, biologists, etc., who have been doing some programming, possibly for a long time.

Either way, most of my experience is with all-programmer teams, and I have to say I’ve seen the experienced programmers make far bigger and costlier messes. The people who can’t really program might always make a lot of messes, but they make very small messes, and nobody puts them in charge of teams or lets them do that much process critical work without oversight or someone re-writing it. I’ve watched very good very experienced programmers make enormous mistakes such as engaging in system-wide rewrites that turn everything into a mess, and that cost many millions of dollars, only to take years longer than they estimated, and to come out the other end admitting it was a mistake. There was also the time a senior programmer tried to get really clever with his matrix copy constructor, and caused an intermittent crash bug only in release builds that triggered team-wide overtime right before a deadline. He was incredulous at first when we started to suspect his code, and I had to write a small ad-hoc debugger just to catch it. I calculated the dollar cost of his one line of cleverness in the several tens of thousands of dollars.

This becomes immediately clear when you confront developers about this. Most of their answers will be irrational qualifiers which might make sense to them, but from a perspective of objectivity and product delivery its really mind blowing. In most cases the insanity stems from poor preparation followed by what then becomes unrealistic expectations.

Just as a real world experiment ask a front end developer to write to the DOM directly. The DOM is the compile target of the browser accessed via standard API which can be mastered in less than 4 hours of practice. Despite that prepare to be under impressed and dazzled by the equivocations, unfounded assumptions, red herrings, and so forth. The DOM is just an in-memory data structure with a standard API, but seems large data structures scare people.

---

All a person really needs to know to be good at this language:

* Functions are first class citizens. This means a function can be expressed or referenced any where a primitive can be used. This is incredibly expressive.

* Lexical scope is native. This means lexical scope is always universally on, not hidden behind syntax, and can never be turned off. This is also incredibly expressive.

* OOP is optional. The language never forces OOP conventions upon the developer, which is great because the concept of polyinstantiation, on which OOP is based, greatly increases complexity.

* The language is multi-callstack. This is commonly referred to the event loop, and allows executing externalizing instructions without locking the language.

* A casual understanding navigating data structures.

That being said anybody can build large, fast, robust applications in JavaScript using only functions, statements/expressions, events, and data structures. TypeScript interfaces help tremendously as well. Despite this most developers need all kinds of vanity to make sense of the most simple tasks and anything original is like asking people to crawl across the Sahara.

> Want to give any examples or reasoning rather than state pure opinion?

Its based upon 15 years of doing that work professionally for multiple employers. By far the biggest messes in this language come from the absence of confidence in the developers writing in it. I imagine scientists, non-professional programmers, writing messy software are at least passionate enough about their subject matter to do it well enough the first time so they aren't spending the rest of their existing fixing bugs, regressions, and performance traps of their own creation.

Perhaps the word lie was incorrect and something like wrong in practice would have worked better.

It's not even scientists vs software developers. It's people who are really into software development and clean code.

They say the program needs a total rewrite and proceed to add 20 layers of inheritance and spreading out every function over 8 files.

Ever since I make sure to repeat my mantra every week to developers:

How maintainable code is is measured in how many files you have to edit to add one feature.

Anyone who in 2023 still thinks inheritance is a good idea for anything other than a few very specialised use-cases is not somebody who seriously cares about the craft of software development, not somebody who's put any effort to study programming theory and move beyond destructive 1990s enterprise Java practices. Widespread usage of inheritance inevitably makes code harder to reason about and refactor, as anyone who's compared code in Java to code for similar functionality in Rust or Go would see (both Rust and Go deliberately eschew support for inheritance due to the nightmares it can cause).

Except, we can (fairly objectively) reason about performance and security, while 'clean code' and 'maintainability' are arbitrary, with vague guidelines at best.

Throwing out those first characteristics in name of the latter ones is just irrational.

(Not to even mention that performant and safe code still can be 'clean')

You are assuming that the only things that matter are those that can be objectively measured (and measured simply and straightforwardly, with well-known metrics today).

Developer frustration, which will increase when having to deal with messy, unmaintainable code, is a real thing, even if it's harder to measure than performance and security. Not only does it create real stress and thus harm to the developers, it also slows development in ways that are going to be much less consistent and predictable than what's needed to write clean, maintainable code in the first place.

(Also, of course, there are at least some fairly well-accepted standards of clean, maintainable code, even if some aspects of those aren't entirely agreed on by everyone, and painting them as completely arbitrary, subjective things is just wrong.)

I once got your for a couple of hours researching the origin of that famous phrase "you can't improve what you don't measure", so I could blame it correctly.

The idea is quite old, of course, and popular to the 19th century rationalists. But the format people keep teaching around today seems to be a strawman created by Deming, at the 80's, in a speech about how stupid that idea is.

Anyway, I guess we need some Othersomeoneorother's Law about how you just can't make a good point against an idea without someone taking your point, preaching it unironically, and making a movement in support of the idea.

No. Feelings do matter. But the problem is, what do You do when You have 2 people with conficliting feelings?

>Also, of course, there are at least some fairly well-accepted standards of clean code.

Are there though?

>even if some aspects of those aren't entirely agreed on by everyone, and painting them as completely arbitrary, subjective things is just wrong

Even if I grant You that there are some guidelines that are respected by overwhelming majority, that still doesn't prevent them from being arbitrary.

Hopefully, you try to work it out like adults, rather than just declaring that your way is the only rational way, and anyone else's feelings need to pound sand.

Furthermore, this isn't primarily about "feelings" in the sense of "this hurt my feelings;" this is primarily about adding unnecessary stress to developers' lives. Stress is something that is scientifically proven to increase susceptibility to diseases and cancers, and reduce lifespans, so it seems to me that this should be enough objective and rational evidence that we should be genuinely trying to reduce it.

> Are there though?

Well, I think most people would agree that putting an entire C file on one line is a pain to work with, even if skipping the "unnecessary" whitespace does save a little space.

And naming your variables alphabetically based on the order you use them in (eg, `int alpha`, `char bravo`, `std::string charlie` makes the code hard to maintain.

"Well, but that's just obvious stuff! No one would ever do that!"

I guarantee you someone would do just about any boneheaded thing you can imagine in programming unless told not to, either out of spite or because their brain really just works that way.

Just because you've made a bunch of assumptions about how people would or should code doesn't mean that those assumptions are any less arbitrary than anything else.

> that still doesn't prevent them from being arbitrary.

...But that's the thing. They're not. Just because they're not deeply well-researched to ensure that this particular set of coding standards measurably increases performance and decreases stress while maintaining code doesn't mean that maintainability is an arbitrary thing. It just means that it hasn't been adequately studied yet.

...or maybe it just means you haven't[0] looked[1] enough[2] yet, and the research that's out there hasn't yet had time to coalesce into any kind of industry-wide action.

Furthermore, it sounds very much like you're saying that coding standards like K&R, or C++ Core Guidelines, or PSR-2, are entirely arbitrary. They're clearly specified, they're written down and easy to reference, they codify plenty of aspects of coding style—but are all coding standards, no matter how well-respected, completely arbitrary?

[0] https://www.researchgate.net/publication/299412540_Code_Read...

[1] https://www.hindawi.com/journals/sp/2020/8840389/

[2] https://www.researchgate.net/publication/303870101_Software_...

Ok, how does one best reason about performance and security with messy unmaintainable code?

You barely need to try even shallow reasoning about a code base at all, before it's clean vs. messy, and maintainable vs. unmaintainable status will feel very objective and pertinent.

The same way one does it with 'clean' code: Using profiling tools. Security is a bit less straight forward, but still.

>will feel very objective.

Keyword: Feel. And while most people could probably agree on terrible code being terrible, the 'less terrible' the code is, the more this argument becomes a feeling. And then we hit up point where it's no longer possible to discuss things using objective arguments - how will a senior java developer, who is used to heavy OOP style coding reason with a senior C developer for whom such OOP heavy style is the oposite of 'clean'?

Ofcourse, the example is (too) simplistic, but even in this thread You have people arguing about "big functions" vs splitting things up. And unlike the performance, which You can alaways just point to the raw numbers, here You can't rely on any sorts of 'objectivity'

Messiness and unmaintainability does not scale. You can argue its not a completely objective thing to measure. That is true. And yet objectively, it really matters.

Not everything that is important comes with a clear number. Which is why experience and good judgement matter too. Two more things without numbers.

It is impossible to answer without all the details of the situation. Different situations will have completely different answers. Who, what, where, how, when, why, ...?

Usually what people want is an interface; ie, a somewhat generic way of saying "this thing knows how to draw itself", "this thing supports printing" or "this thing can fizzle wuzzles like all the other wuzzle fizzlers". That is essential to coding.

But Java style inheritance carries a lot of baggage in excess of that; and some of it is just bad news. In practice it is a brittle assumption that a Foo is also and always a precise superset of Bar. And usually when that is true the relationship is so shallow having a dedicated concept of inheritance is wasteful, it may as well be an interface and a shared file of code.

TLDR; Inheritance is too many ideas mixed together; most useful and a couple bad. It is a better idea to present the different facets of inheritance to be selected a la carte. Mumble mumble Rich Hickey talks.

Interfaces are good.

Method overloading for specialization or for creating mini-DSLs (Template Method pattern) is often problematic and is better replaced by composition, or by having the "overloaded" methods in a separate class.

Implementation Inheritance is certainly the worst form of "code reuse", and there's a reason people recommend composition over it since the 90s.

Using it for hierarchies (Dog inherits from Mammal, Mammal inherits from Animal) is just terrible and a joke at this point.

Composition is nice because it’s very simple and we can understand it mathematically. If you’re trying to understand inheritance mathematically then you’re basically left with using it only for algebraic structures (groups and rings and fields and vector spaces). But then you don’t really need inheritance there if you just have plain types and operator overloading.

Certainly adds to complexity in C++.

Overloading addition to mean something else entirely? That’s a problem!

It would be great if type systems could allow us to set up these laws and enforce them at compile time, but then you go down the whole rabbit hole of automated theorem proving.

How maintainable code is is measured in how well you know where to change something, and how certain you are that it did the right thing without side effects.

The fatal error of the linked article is that bad scientific code often suffers from correctness problems - not just theoretical concerns, but the "negates the main point of this paper" kind of thing.

Recently at work some people argued "things" (methods, classes, even files) should have a limit in size. I think that's valid thinking because you want to strive to having smaller components that you can reuse and compose, if you are able to do that. But what happened is that people started creating dozens of little files containing a function each and then importing those. To be it's obvious that this is now a lot worse because the complexity is still the same, just spread out across dozens of files. But most people were somehow convinced that they're "refactoring" and that they're doing best practices of keeping things small.

All other things being equal, smaller functions and smaller files are a little bit better, but what really matters is architectural and conceptual complexity. Keeping those in check is all about using the right data structures (and doing painful refactors when you realize you've got the wrong ones). It has almost nothing to do with how files and functions are organized.

Who are these people, really? I mean, I was that for a rather brief period of like a year, somewhere 3-4 years into programming IIRC. I met others like it. But in the end people seem to learn and grow out of it, and rather quickly so, because it becomes clear what the issues are quickly. Really good learning by mistake though, wouldn't have wanted to miss out on it.

mantra

That's imo just another mistake to make: wrapping a rather strict and narrowly scoped principle in a paradigm-like-must-be-followed mantra hurts the right-tool-for-the-job idea, which feels vastly superior.

Firstly the idea that unmaintainable code is necessarily an issue is already wrong to start with, in my book. Obviously it's not ideal and where appropriate - meaning nearly always - should be avoided at pretty much all cost, but I have enough examples where it does not matter at all. As in: code which hasn't been touched in 20 years and probably won't ever be touched. Does it look like a nightmare? Yes. Does it work correctly? Yes. Does it need changing? No. So, is it an issue where spending time (or having spent time) on it would make anything other than the programmer's peace of mind (well, or ego perhaps) better? Clear no.

Secondly: of course I get where you're going with such definitions but it again lacks the very much needed nuancing. I can write unmaintainable code in one file but where you need to change 20 different locations. You could than claim that your mantra still applies because the code should have been split over 20 files, but yeah, that's what you get with mantras :) Likewise depending on the feature it's perfectly possible many files have to be changed but that doesn't necessariy mean the code is hard to maintain. Could try to claim that it wasn't very well architected to start with, maybe, but welcome in the real world where not everything can be thought of from the very start except in small toy applications.

Trivially false: if all of the source code is kept in one file then you only ever have to edit one file. No matter how many lines of code are in that file, any such system would be maximally maintainable by your definition.

Maintainability is not so clear-cut. It's currently an imprecise measure of organizational clarity that makes it straightforward to extend a system in ways that are needed or will be needed.

Not if you are encapsulating and naming effectively...

Why read 100 lines when you can read 20 and find concerns in one routine you are concerned with?

Function calls can be expensive. However, optimization can come whenever you need it, and if what you need is one call vs 5, it is trivial to move that code back into a single routine.

No, and this is one of the reasons inheritance has lost popularity. Splitting some functionality across many files adds significantly to the cognitive load of figuring out what code is actually even running. After you trace that information out, you need to keep it all straight in your head while debugging whatever you’re working on. That’s even more problematic when you’re debugging, which implies you already don’t really understand what the program is doing.

And that’s in the case where things are named well. When they’re inevitably accidentally named in confusing or incorrect ways that can contribute to the bug itself and cause the code to be even more confusing.

Extreme levels of encapsulation has its own issues when, actually, the original author is wrong and you really do need some public access to some member. No one writing code is clairvoyant, so excessive encapsulation is common.

This is the crux, if your goal is to figure out what code is running, if you can keep the program in your head, if you have small simple programs splitting things up is harmful.

But there is this murky line, different for everyone, and even different for the same person from day to day, where even with the best intent, no matter how good you are, you can't keep the program in your head.

At that point, you need to give up the idea that you can. Then you change perspective and see things in chunks, split up, divide and conquer, treat portions as black boxes. Trust the documentation's, pre and post conditions. Debugging becomes verifying those inputs and returns; only diving into the code of that next level when those expectations are violated.

It depends on the change. It depends on the code organizational structures. It depends on the consistency of the code. It depends on the testing setup. It depends on the experience of the person changing it. It depends on the sensitivity of the functionality. It depends on the team structures.

Separating the code of your SQL server, HTTP server, Crypto Library, Framework, Standard Library, from your CRUD code is perfectly fine, and people understand this concept well, and even the most fervent anti-Clean-Code person won't complain about this separation existing.

But there is a good reason we separate those things from our CRUD codebase: it's because they can function separately fine, they're reusable, they're easy to isolate/reproduce problems, and they're at a totally different abstraction level.

The problem is separating code from the same level of abstraction, such as breaking a business logic class into many for mainly aesthetic reasons, such as method/class/module length, or to avoid having comments in the code (again as recommended by Clean Code), things that people are mentioning here in this thread.

EDIT: As someone said above, "20 files with 20 functions each does not cause high cognitive load, if the scope of each file and each function makes sense". In the end it's not the length or the number of methods/classes that matter, but how well separated they are. Having hard rules does not automatically make for good code, and it's often quite the opposite.

However, Can you?

You are using black box code from a web server, a sql database, the operating system, crypto libraries, and a ton more; You don't dive into that source code except in extraordinary circumstances, if you even can. In a large program, you end up treating code owned by you or your company as the same way.

In this scenario you are still making large meaningful changes by focusing on the level of abstraction you are at.

Rarely does having more functions solve “does this do what I expect.”

With a full-featured language specific IDE, it is very easy to navigate through even complicated spaghetti. It makes debugging call traces simple, with a GUI.

However, many other file viewers and editors make this much more complicated, and it can be frustrating to follow code that is making heavy use of modularization.

If you are grepping your way through a deeply modular code base it can quickly become difficult to keep track of anything.

If you need a fancy IDE to navigate around code in order to understand it, that might be crappy or poorly organized code.

Not a dig at nice IDEs, just code that requires one to navigate and understand.

> Not a dig at nice IDEs, just code that requires one to navigate and understand

A nice IDE helps you reason about code, no matter what the underlying architecture is. That is why there is a market for them.

Encapsulation is hard and a lot of what people call encapsulation isn’t. For example, taking a global variable and moving it to a class is not encapsulation. You have to actually do the hard work of removing the dependency on global shared state. Just changing everything to mutate the new global through an accessor to a “god” object that gets passed everywhere is accomplishing nothing at all. Worse than nothing, you’re complexifying without fixing the root problem: global mutable state.

A cleverly-named way of disguising global mutable state does not make it better.

When you only have to superficially skim the code, that works.

If there are incorrect abstractions, such as logging, transaction logic or manual error handling mixed with "well named function calls", then it is already very problematic even to skim.

If you have to debug, it quickly becomes torture. Especially if state is involved and shared between multiple methods or classes.

If you have to reimplement the code: you're probably fucked.

The more correct statement is that using more or less indirection than you need makes code less maintainable.

This is why I hated Fortran (77 in particular) as an applications language (for tasks like scientific computing people seemed to use saner portions of it). Computed go tos were the bane of my existence.

Consider dealing with the output of a (lexical) tokeniser. It is much easier to maintain a massive switch statement (or a bunch of ifs/elseifs) to handle each token, with calls to other functions to do the actual processing, such that each case is just a token and a function call. Grouping them in some way not required by the code is an illusory "gain": it hides the complexity of the actual function in a bunch of files you don't look at, when this is not a natural abstraction of the problem at all and when those files introduce extra layers of flow control where tricky bugs can hide. Or see the "PLEASE DO NOT ATTEMPT TO SIMPLIFY THIS CODE" comment from the Kubernetes source[0]. A 300 line function that does one thing and which cannot be usefully divided into smaller units is more maintainable than any alternative. Attempting to break it up will make it worse.

That being said, I agree that nearly all 300 line functions in the wild are not like this.

[0] https://github.com/kubernetes/kubernetes/blob/ec2e767e593953...

20 files with 20 functions each does not cause high cognitive load, if the scope of each file and each function makes sense. You easily find the file+function you need, whenever you need to look something up and the rest of the time it is as if the rest didn't exist.

Good code can come in any shape. It is not shape itself that is important, it is the "goodness" that is important.

Carmack has a good essay about it.

http://number-none.com/blow/john_carmack_on_inlined_code.htm...

It seems the author indeed is not a SW Engineer and thus does not really grok the benefit of "modules".

This of course depends on the size of the program. Small program "fits" into a single module.

And I think that scientific programs are basically small and simple because they don't typically need to deal with user-interaction at all, they just need to calculate a result.

Further I think scientific programs rely heavily on existing libraries, and writing a program that relies heavily on calls to external libraries produces simple, short programs.

Scientists produce science, engineers produce code-libraries.

This is a pretty ridiculous notion if you just cursorily glance over the page. It is quite clear that this guy is more of a software engineer than most with that title will ever be. Hint: a blog that contains a post with a title like 'Coroutines in one page of C' is a software engineer.

Overengineering is insidious - "It is difficult to get a man to understand something, when his salary depends upon his not understanding it". A team can sell a solution better than a single person fixing something without making a big deal out of it. You get organizational clout and inertia on your side when you make something big and expensive.

And then complex systems are by nature hard to reason about and by extension hard to critique.

So many things come down to "complexity is the enemy".

Wrote fancy math on a PhD economist’s laptop.

Then when we added testing when developing a platform, we found out we’d been 5% off in allocating Prime revenue between organizations — and had the correct amount gone to Retail, the 2018 hiring freeze might have been avoided. (According to a very angry Wilke.)

Whoops.

Turned out we did need a team and all those guardrails, processes, code reviews, etc.

There’s a time and a place for “move fast and break things” — but real trouble can come from taking those academic practices into the real world.

Economists from other teams looked at it and signed off.

There was manual testing — ie, trying some “choice values”.

We discovered their error in convexity when our test suite allowed us to randomly sample the models at scale. (Actually, I had questions before that — but unsurprisingly, when it was just me questioning a PhD economist, the lowly SDE was ignored.)

Good intentions aren’t enough; you need mechanisms.

Sheer tenacity is typically sufficient for scientific codebases.

Interesting switch in language here from "software engineering" to "programmers". There is of course a long history of debate on these terms, whether there is a meaningful distinction, and what qualifies as engineering versus programming.

Wherever you stand on this debate, there are a number of practices of software developers that tend to be used more towards the "engineering" side. Two of the most essential in my mind are peer code reviews and automated testing of changes (with tests, linters, type-checkers, code formatters, profilers, fuzzers, etc.).

This post doesn't talk about any of these practices or whether the so-called "programmers" messing up the scientific code are using them. I'd say if the people messing up the code are not actually advocating for using software development tools to write better code they are not actually applying software engineering practices to their code.

Not sure how much of a problem it is, but I am frustrated when "other" engineers try to teach me about how networking works and never once consider that my intuition may possibly have more value than theirs, because I actually studied networking. Not that I am always right of course, but if we have an electrical engineering argument, I naturally get into a stance where I assume they know better and can teach me useful stuff.

- Multiple/virtual/high-on-crack inheritance:

  add each function/class has 7 template specialization parameters and 3 macros which expand to templates which expand to macros
  

  think maps of maps of maps loaded from configs of configs
  

  obviously everything has to be a "plugin"
  

  a 1000 times yes! yes!!! DriverController inherits from ControllerManager which extends ManagerController which contains a DriverManager which agregates 3 ControllerManager from diferent namespaces. I've seen a DriverController function going through 12 levels of stack passed between 3 threads to eventually call back a function from the same DriverController
  

  if a template technique exists, it had to be used!
  

  Of course they define their own DDL with xmls parsed by a combination of Python and awk which generates C++ macros which are used in templates to dynamically load plugins which hold maps of maps of function pointers to create3 events dispatched on a pool of threads

and on top of it those DriverControllers and ManagerControllers keep getting depracated

Specifically when working with numbers (something scientists do), i feel it is more important to know the numbers and have an intuition of what is important to get right and what is less important. Rather than spreading unit tests and the following abstractions all over the place.

But scientists can create a hot mess too. This is when they don't care about the code enough to minify it. I.e. reducing the code to what is necessary.

Number of lines of code matter.

So the perfect blend here is not necessarily the person that follows all engineering practices. But the person has the domain knowledge but that knows enough about software practices to be able to write concise code.

If you are either missing out on the domain knowledge - or the ability to reduce the code - then it will derail.

Shipping the notebook to engineering department will not be the solution. They will break the code into pieces, follow best practices but miss out on the important.

And another aspect of this practice is that if someone finds a problem, the scientist will not be able to modify or re-run the engineered version of the notebook.

Scientific code? You just described 99% of "enterprise" java code above.

I guess it stems from the ideology that it is better to do lots of groundwork now, in case the program blows up and needs to scale. Which is somewhat true...but in the world I work in, it is only true for maybe 1% of programs we write.

So in the majority of cases, if I need to fork some software at work and do easy modifications, I do prefer the one-file programs, compared to some behemoth where almost everything is boilerplate, spread over multiple source files, folders, etc.

I think there's a genuine tension between writing good code and "shipping" a paper. At least, when I program "as a programmer" I think my code is mostly higher quality.

[1] https://github.com/hughjonesd/why-natural-selection/blob/mas...

I have learned to appreciate codebases which break some rules & yet are easier to maintain for some reason(s) Distilling the reason(s) identifies areas where I can modify my technique...or at least help identify questions & alternatives to some techniques that I regularly use.

Getting too excited about techniques is a form of scope creep

Write some code and generate immediate results can be considered a done job. If you want to reuse the code a year later and found an unreadable mess though, not so much.

Easily leads to conclusions of "you don't need this to scale" vs "build this to scale" and "you need to make this extensible" vs "just ship the hacky thing".

Just need to know what your params are and place uncertainty on it.

Once I wrote a program in a day. Did a kind of pricing. Hacky af. Just tech demo. Over time people start depending on it. One year later it breaks. Too many products priced, internal ring buffer only has few slots. Pricing stalls for all products.

"This is a company-killing issue!" Everyone yells. Sure, my mistake. I let thing get depended on without productifying but original choice was fine since it allowed iteration on something else that made money. Just when situation changes you gotta adapt.

No rules about that except just short loop on quality required and time spent.

That being said, I've also seen plenty of "scientific code" that's totally incomprehensible even to the point where the scientist who wrote the code can't debug it. So there's an extreme in the other direction.

He made it to make his own life easier, and never considered any best practices, everything was a thousands of lines of jupyter notebook, it was not modular, painful to customize but just got the work done, and was kinda a black box in way the way it generated the outputs.

Since then we've tried to re-write it with good programming principles, but still we were not able to re-create what he did. So yah sometimes done is better than perfect.

Multiple times in my career, I have seen scientific code written by academics dramatically sped up by developers who used parallelisation, vectorisation using SIMD etc.

So in terms of performance, naively written scientific code is generally easy to beat in terms of performance for a reasonably adept programmer.

That said, "enterprisey" Java/.NET software engineering shops can indeed make scientific code sub-optimal sometimes. Have come across that sometimes too but I wouldn't generalize.

Tools and frameworks encourage this. Git and VS code are build around directories. In VS code the first thing in the sidebar is the explorer. When you press Ctrl+P you see an overview of files. File-System based routing.

But directories lack a crucial feature compared to text: Ordering. If I put everything in one file, I can order it in a way that makes sense. If I put everything in different files and directories, it's all going to be ordered alphabetically.

Why bad scientific code beats code following “best practices” (2014) - https://news.ycombinator.com/item?id=12377385 - Aug 2016 (261 comments)

Why bad scientific code beats code following "best practices" - https://news.ycombinator.com/item?id=7731624 - May 2014 (168 comments)

For scientific code certain things are just not important, hence you do not deal with them:

- Observability: You just care for the result of the run, not for the state of the running system - Security: Your code is running in isolation, used by yourself

The result looks horrible to a normal programmer, even if it’s well maintainable, but it is exactly what is needed to do the job.

Sure, you can ignore security if all you're doing is processing local text files, granted. But things looking horrible to programmers isn't just about security bugs, it's about the whole span of correctness bugs. And scientists need to write code that is both correct and maintainable. The frequency with which they don't is partly why results so often can't be replicated, making the money spent on academia wasted.

The idea that science code doesn't need to be maintainable or that they have some magic way to do it that looks wrong, isn't right either. It's not uncommon to find model "codes" that scientists have been hacking on for decades. The results have become completely untrustworthy many years earlier, but they deny/obfuscate/ignore, attack or even sue people who point out concrete problems. Sadly, often with the acquiescence of the media who are supposed to be ferreting out coverups.

Scientists need to collectively get a grip on this situation. They will happily attack anyone outside their institutions as being non-expert conspiracy theorists, but when it comes to software they suddenly know everything and don't need to hire professionals. Paper-invalidating bugs are constantly being covered up and the only reason the problem hasn't reached criticality yet is that many people don't want to hear about it. But the unreliability of academic output is now becoming a political problem and a divisive culture war issue, when it really shouldn't be. A good first step to solving the replication crisis would be for scientists to stop pretending it's OK to quickly knock together a program themselves instead of assigning a ticket to a trained full time SWE. Yes it would cost more (a lot more), and that's OK. Generate fewer papers but get them right!

But, a lot of the time these scripts are used as one offs, to generate a result, then are done with, so quality doesn't need to be the same as with a server running 24/7.(Sadly the ones I had to fix were being run regularly).

Definitely getting this vibe from modern frameworks and design patterns.

I want to like SwiftUI, but the WYSIWYG editor doesn't even work for the default projects for me. Storyboards were great for creating everything UI except tables and collections, where it was still functional just meh. The reactive UI in SwiftUI… mostly works, but sometimes doesn't, and when it doesn't I can't debug it because it's a magic black box; you can get similar results with a small amount of extra code in UIKit { didSet } on your model property and each input control, and while boilerplate isn't great, it's better than magic which only works 98% of the time.

I'm trying things in JS in my spare time, no libraries or frameworks, and it's easier and faster than getting anything done in XCode. And that's despite the Swift language itself being one I prefer over JS, and that I'm doing the development in BBEdit which is a text editor not a full IDE.

But this isn't just about Apple; the reason I'm not using any JS framework and libraries is that every single talk I've seen about web development has exactly the same problem, piling on layers of stuff to fill in the gaps missed (or created) by the previous layer of abstraction.

I think the goal is to make if more accessible, so that more people with less knowledge can produce more crap with it.

People don't learn the basics, they want to write a comment in Copilot and have it assemble code that roughly does what they want. I believe that people who got into software in the 60s actually liked computers. People who get into software today just want to produce stuff, they don't care about their computer.

If so, it fails at even this: for my A-levels[0], my teacher only knew VisualBasic[1], which was very easy to work with: drag and drop widgets onto a form, (double?)-click on a widget to get right into the code block that happens when a user users the widget.

Back when JS was new and there were no extra layers of abstraction, yes the language sucks, but you could get to work with it using only what you found in a £4.99 book from WHSmith[2] and a text editor, you didn't need to `install npm` and then some library and then…

> People don't learn the basics, they want to write a comment in Copilot and have it assemble code that roughly does what they want.

Agree. Heck, I do that, and I started learning the basics when I was about 5. :)

> People don't learn the basics, they want to write a comment in Copilot and have it assemble code that roughly does what they want. I believe that people who got into software in the 60s actually liked computers. People who get into software today just want to produce stuff, they don't care about their computer.

My dad probably got into computers some time in the 60s, a one or two day corporate training program about "this new thing called 'software'". The way he talked about them, it was clear he didn't really understand them, and just wanted to get stuff done.

[0] https://en.wikipedia.org/wiki/A-level

[1] It turned out that the copy of REALbasic I had on my Mac at home was almost copy-paste compatible, the only exception I ran into was that `Dim foo, bar As Integer` has `foo` and `bar` being `Integer` in REALbasic while our version of VB had `foo` being `Integer` and `bar` being `VarType`

[2] https://en.wikipedia.org/wiki/WHSmith

One thing I've noticed is that programming practices have evolved, not so much to make them better than before, though that's conceivable. But because practices have to keep up with rising complexity of the code itself, and also of the operating environment and the social environment (e.g., work teams, open source projects, etc).

Scientific programs tend to be easily 20 years behind software development in terms of complexity, and I think we can benefit from using older techniques that were simpler and easier to learn. I learned "structured programming" via Pascal, and to this day if I hew to the same practices that I learned in my Pascal textbook, my program will probably do what it needs to do and be tolerably maintainable.

Perhaps those practices have to come from the mouths of scientists. The software engineers have moved on, and are only interested in the latest and greatest toys. I don't blame them -- they have to own their careers and follow their interests just like we do.

I mentor younger scientists who come out of fields such as chemistry, and are beginners at coding. So I literally get to explain such basic things as putting code inside subroutines, and avoiding global variables. I haven't had to tell anybody about GOTO's yet.

About reproducibility: My parents were both scientists, though my mom spent a few years in mid-career teaching programming at a community college. I learned the scientific method sitting on my mommy's knee. "Reproducibility" was certainly a guiding principle, but it was also expected that reproducing a result would require some effort -- perhaps fabricating your own equipment from available materials, and gaining skill on a technique. You might get it wrong many times before finally getting it right.

What we expect now is "pushbutton" reproducibility, meaning that a project replicates itself from start to finish at the push of a button. This is a much higher standard than any scientist is trained to expect, even if software engineering requires it. A software project has to be at least 99% that way, or it would be unworkable, due to the high degree of complexity. The tradeoff is that it also requires complexity to make things that way.

I expect my results to be reproducible, but not pushbutton-reproducible. To overcome this issue, I'd rather spend my time documenting my code and its theory of operation, than making it bulletproof. Nothing that I write goes directly into production, and I expect the theory of operation to be more valuable to a project than my code. Often, the code just automates an experiment to test the theory, so it's a middleman rather than a product.

this is why i'm a little skeptical of languages that blur the lines. sometimes ideas from application programming can complicate numerical code and sometimes numerical programmers don't fully understand the systems abstractions they're building on and end up reinventing wheels to avoid simpler solutions they feared or didn't know existed.

the moral of the story is to keep an open mind, to not be a zealot and to avoid dogmatic thinking.

Did you consider hiring an experienced software engineer as a lead?

As for hiring software devs, that's not going to change (in general, there are places where software devs write code used by scientists, but rarely are these codes themselves pushing research boundaries, it's code on top it that does) absent significant changes in funding structure and rules (which are typically a government/public service concern, and not up to researchers).

Disagree. Well, maybe still acceptable if the software is small / limited to a single paper. Having worked on a code base the people writing it learned programming on that job, guessing their intention is like archeology. And each iteration tended to add some complicated interdependence. Or like when int errorCode came from other, overlapping error ranges.

A part of the "new" code base is exactly as described by TFA. Including most interfaces having only one implementation. But while annoying, I am more able to work on it without things breaking...

My point about the rant being counter-productive is that the solution is to learn how to do the task better, not to blame people. Maybe the best way to do a task with a very low budget is to not do it at all.

I doubt this very much. Surely the vast majority of running code is some chunk of Chrome, Android, or the JVM ("billions of devices run Java...") or something. All those things were produced by software engineers with more than surface-level understanding.

Let's be honest: web browsers are really bad. Overly complicated machines made to print images and text on the screen. The web got sideways long ago, first because it was cool to add crap in websites, then because it made profit and allowed monopolies to make more profit by moving everything to damn webapps. And finally it allows companies to screw users by renting software on the damn cloud.

And they profit by making it as accessible as possible, such that everyone and their dog can produce a crappy webapp that will show their ads or track their users.

Webtech is part of what makes software really bad, even if it was created by good engineers. Because people don't want quality: they want cheap new crap.

But there is certainly more codebases out there running some python or js written by designers, data analysts etc then those produced and curated by software engineers.