Some of this is interactive development vs static development. A lot of the IDE features and stuff you get in "static" development, you could imagine getting into notebooks one day. The thing I don't know any good way to deal with is cached state during development.
If I have to get data that takes 15 minutes to query, another 10 to preprocess, and then maybe a few more steps before it's ready to put into my algorithm and start poking around, notebooks shine. In an IDE you could (and I often do) use have cached datasets partway through the process saved to disk, as mentioned in OP. But that's a hack too. And it still takes a minute or so to load and process anyways if cached.
So the options are
1) Shitty practices, but I can write try new changes on your data immediately.
2) Better practices, wrapped in caching hacks, but I have to wait minutes between every single change.
If I have to get data that takes 15 minutes to query, another 10 to preprocess, and then maybe a few more steps before it's ready to put into my algorithm and start poking around, notebooks shine. In an IDE you could (and I often do) use have cached datasets partway through the process saved to disk, as mentioned in OP. But that's a hack too. And it still takes a minute or so to load and process anyways if cached.
So the options are
1) Shitty practices, but I can write try new changes on your data immediately.
2) Better practices, wrapped in caching hacks, but I have to wait minutes between every single change.