Yeah, I hear you about the jupyter-is-for-draft-code. I thought that's how everyone used Jupyter for software dev. I guess this is the MATLAB-style software dev---try something in the REPL, and once you figure out the parameters interactively copy-paste that line/paragraph into your program.
As soon as I have one piece of functionality working as a function, I move the code to the "main" file for the project and import the function into the notebook. This means I never have to write more than a few paragraphs of code in the notebook and instead use it for testing and glue code. I don't think of the notebook as something sharable with others, it's more like WIP code... if I keep the notebook around it's usually because it can end up a useful test harness when code needs to be revisited/updated.
As for global state, one thing that I find really helpful is the keyboard shortcut 00. If you focus anywhere outside of a cell and you press zero twice, this restarts the kernel and you can re-run the commands from the beginning in a "clean state" so what you see matches what is. This approach works well when you're editing the source code that gets imported into the notebook---you never have to worry about re-importing since always restarting from scratch.
I think there is a big misunderstanding here. There's 2 kinds of people working with these things, and their needs and goals are very different.
You have scientists. They are trying new things, and while reproducibility is nice, most of the time it doesn't matter. Tests don't matter, to put it mildly, because we're at a stage where it isn't just "not a product", it's not worth to become product, and they are aware of this. IF there is an end product, it's a report. A report that lists in general terms what was done and what the result was, nothing more.
Their goal is to fully understand the mechanics, the minutiae and being able to explore and tinker with it (without tests breaking) is valuable. I hope you can see that most tests get very much in the way of this activity. I'm not saying there isn't anything you might test (quite the contrary), but you're not going for 100% coverage. Hell, probably not even 10%.
That's how you get to matlab, mathematica, python notebooks, ... That's the usecase.
On the other hand you have software engineers. They're also not a homogeneous group, but let's ignore that. The key point is that they do not understand the details and don't intend to understand the details. The goal is also different. The goal is not to learn, the goal is to help a company build a product. Reliable, repeatable, controllable, and something where you can pick someone of the street and tell them to fix something and actually expect that to get fixed.
Understanding everything about everything you work with, like an academic would try to do, is not practical for a software engineer.
And they get presented with an excel sheet, a notebook. They don't understand, there's no docs, but remember, they don't want to learn. Worse ... likely there's problems with the notebook. Maybe it solves one case particularly badly for some reason. Maybe it needs to run in an entirely different environment, like maybe on a phone. Or faster. Or needs to start doing something new.
So they need to change something. Quickly, and quickly means without understanding the domain behind the code. So they change something and ... everything else falls down.
So software developers want properly designed code with comments about even banal things, explanations about every last variable, with more tests than code, that cannot reasonably be changed with less than 5 people, and where you can always and without thinking get it back to when "it worked". Without understanding the domain behind it.
So notebooks are not for software developers.
You'll see this difference reflected in the languages as well. Notebooks are for languages that allow one to talk extremely high level, extremely succinctly. Because that's what academics want to do.
Software developer languages, the successful ones, are basic simple things, like (these days) Java, C and Go (even C++ and Python are considered too high level by most and the frameworks software devs work with don't exactly have PyTorch's flexibility). If you want to see how you give software devs what they want, study Ada (and resist the urge to gauge your own eyes out afterwards). But there is no language, imho, as strongly on the software developer side as Ada. I would even say it even throws basic sanity under the bus to get controlibility.
I would say to some extent machine learning and software development are enemies. You can fix a number of things, but when push comes to shove, machine learning is about "just figure it out", which is fundamentally against the software developer mindset of controlibility. Can you guarantee that a NLP model will never be racist ? The truth is ... not really. You can test specific cases, but the whole point of using such systems is creativity.
Use the right tool for the job. Although I would say, to learn programming ... even for software devs notebooks are great for this.
> Understanding everything about everything you work with, like an academic would try to do, is not practical for a software engineer.
> So notebooks are not for software developers.
You admit software developers are not a homogeneous group, but these are fascinating blanket statements none-the-less. "Notebook" systems originated from software developers, if you allow the origin to be Knuth's ideas of Literate Programming [1] as the forefather of Notebook languages.
It seems an interesting full circle of sorts that Knuth heavily promoted the idea of Notebooks as a superior programming environment in a time when the languages weren't particularly suited to the task (compiled languages that needed very delicate surgery to convert from human readable order to compilable code) and tools/environments that were effectively suboptimal to the task (text editors with no GUI/WYSIWYG abilities; macro-based compilers instead of live interpreters; etc). Now we finally have the tools to make that work, and at least in a few places developers are starting to make use of it (Netflix's Papermill in the linked article is an interesting example).
Admittedly, you might argue that Knuth was perhaps more on the academic side of the fence than software developers today, but Knuth very succinctly argued that all software development would perhaps be better if we stopped relegating the human/natural language narrative of the project to embedded comments in a programming language, and instead embedded the programming language into our human/natural language narratives.
As soon as I have one piece of functionality working as a function, I move the code to the "main" file for the project and import the function into the notebook. This means I never have to write more than a few paragraphs of code in the notebook and instead use it for testing and glue code. I don't think of the notebook as something sharable with others, it's more like WIP code... if I keep the notebook around it's usually because it can end up a useful test harness when code needs to be revisited/updated.
As for global state, one thing that I find really helpful is the keyboard shortcut 00. If you focus anywhere outside of a cell and you press zero twice, this restarts the kernel and you can re-run the commands from the beginning in a "clean state" so what you see matches what is. This approach works well when you're editing the source code that gets imported into the notebook---you never have to worry about re-importing since always restarting from scratch.