managing npm and package.json which has dependency issues.

With python it always has been pip install package and we are done.

Although I do share your pain with versioning. I once spent a week debugging an issue only to find out that there's a fixed version available.

As a user of an application: `npm install` just works (same with `cargo build`). For Python, I’ll probably do `python -m venv env; . env/bin/activate` and then, well, it’s probably `pip install -r requirements.txt`, but sometimes it’ll be other things, and there are just too many options. I may well add --ignore-installed and use some packages installed locally of potentially different versions, e.g. for setting up Stable Diffusion recently (first time I’ve ever used the dGPU on my laptop) I wanted to use the Arch Linux packages for PyTorch and the likes.

For managing dependencies in an existing library or application (as distinct from starting from scratch, where you’ll have to make several extra choices in Python, and where npm is badly messed up for libraries), both npm and Python are generally fairly decent, but the whole you-can-only-have-one-version-of-a-library thing from Python lends itself more to insurmountable problems.

My personal background: many years of experience with both npm and Python, but I’ve never done all that much packaging in either. (Also many years of Rust, and I have done packaging there, and it’s so much easier than both.)

The worst I encountered was a somewhat lengthy manual to build a project (cannot remember what it was), and not only did you have to manually install all required dependencies -- npm automates this by having an npm-readable list in package.json -- but after running all those commands, the last note said something like "oh BTW, whenever we wrote "pip" above we actually meant "pip3", use that instead, and if you did it wrong, there's no way to undo it."

I agree with that, its pretty painful

Give, quite a bit. The egg format was introduced in 2004 [0], and even the newer wheel, that replaced it, is turning 10 later this month.

[0] https://packaging.python.org/en/latest/discussions/wheel-vs-...

Official dependency and build tooling is not properly geared for C extensions. You'll be looking at compiler errors to figure out what dependencies you are missing.

This is why we created Linux distributions in the first place. It is not the place of every language to reinvent the wheel - poorly.

> Oh no…

That sentence doesn't mean what you think you means. It is saying that there should be "only one way to do something".

Does not mean

> There should only ever be one way to do each thing.

But somehow extremely many people read it like that.

By default, yes. It is possible to install ML-package-1.2 into your 'base' python and then have virtual environments A,B and C all use that instead of installing their own copies. However this is generally not considered best practice.

I can see that as an option in tools like venv or poetry, possibly setting a "common" location to share across environments.

I completely agree with your feeling. I been working with js, php, ruby for years, and packaging is pretty straightforward in every one of them. And the lang versioning works for every one of them (php is the most annoying, yeah, but even on php is easier than in python). Ruby has several alternatives like rbenv or rvm, but any of them works has every feature 99,9% of developers needs, and they work just fine. I want that in python ):

I don't know how I'd feel about poetry being part of pip. Didn't poetry just push out a change where they deciding randomly failing 5% of the time while running in a CI env was a good idea?

The Python ecosystem is a bit of a disaster. There are many useful packages and programs written in Python, but they're legit scary to install. I always search for Go or JS alternatives if they're available to avoid all the python headaches.

Every time I need to interact with Python, I find myself banging my head trying to figure out eggs vs wheels vs distutils vs setuptools vs easyinstall vs pip vs pip3 vs /usr/bin/pip3 vs /opt/homebrew/bin/pip3 vs pip3.7 vs pip3.8 vs pipx vs pip-tools vs poetry vs pyenv vs pipenv vs virtualenv vs venv vs conda vs anaconda vs miniconda.

I’m curious because this gets repeated a lot by many people. What specific messes do you get into?

I’m asking because my experience with python these days is always just doing “python -m venv .venv” activating it and then using “pip install -r requirements.txt”

To update dependencies I do “pip install —-upgrade dep” run tests and then go “pip freeze > requirements.txt” this never fails me. Though sometimes updating dependencies does make tests fail of cause, but that’s the fault of the individual packages not the packaging system. Even so I’d say even that is rare for me these days.

I know the might only work for 95% of the workflows out there, but I’m very curious as to what specific messes the last 5% end up struggling with and what makes people like you feel that it’s always a mess and not just “sometimes it gets messy” etc.

Personally, it feels better in some ways and worse in others.

The whole pom.xml approach/format that Maven uses seems decent, all the way up to specifying which registries or mirrors you want to use (important if you have something like Nexus). Although publishing your own package needs additional configuration, which may mean putting credentials in text files, though thankfully this can be a temporary step in the CI pipeline.

That said, personally I almost prefer the node_modules approach to dependencies that Node uses (and I guess virtualenv to a degree), given that (at least usually) everything a particular project needs can easily be in a self-contained folder. The shared .m2 cache isn't a bad idea, it can just make cleaning it periodically/for particular projects kind of impossible, which is a shame.

I think one of the aspects that make dependencies better in JVM land is the fact that you oftentimes compile everything your app needs (perhaps without the JVM, though) into a .jar file or something similar, which can then be deployed. Personally, I think that that's one of the few good approaches, at least for business software, which is also why I like Go and .NET when similar deployments are possible. JVM already provides pretty much everything else you might need runtime-wise and you don't find yourself faffing about with system packages, like DB drivers.

That said, what I really dislike about the JVM ecosystem and frameworks like Spring, is the reliance on dynamic loading of classes and the huge amounts of reflection-related code that is put into apps. It's gotten to the point where even if your code has no warnings and actually compiles, it might still easily fail at runtime. Especially once you run into issues where dependencies have different versions of the same package that they need themselves, or alternatively your code doesn't run into the annotations/configuration that it needs.

Thankfully Spring Boot seems like a (decent) step forwards and helps you avoid some of the XML hell, but there is still definitely lots of historical baggage to deal with.

Personally, I like Python because of how easy it is to work with once things actually work and its relatively simplistic nature and rich ecosystem... but package management? I agree that it could definitely use some work. Then again, personally I just largely have stuck with the boring setup of something like pip/virtualenv inside of containers, or whatever is the most popular/widespread at any given moment.

Of course, Python is no exception here, trying to work with old Ruby or Node projects also sometimes has issues with getting things up and running. Personally I feel that the more dependencies you have, the harder it will be to keep your application up and running, and later update it (for example, even in regards to front end, consider React + lots of additional libraries vs something like Angular which has more functionality out of the box, even if more tightly coupled).