This is basically what I was doing in the 80s and 90s, downloading compressed tarballs from ftp sites and compiling them. It takes quite a longer developer time than the package manager approach. That includes the time to learn which sites you can trust (probably none today) and which dependencies to use (usually listed in the README.) Furthermore there would be a big incentive to use very few libraries: this is both good and bad. Good because there won't be silly one function modules, bad because a dozen of small modules can add significant value to a project in a short time. Having to code them or build a bigger all comprising module is much harder.
The problem is that all the popular NPM packages have so much dependencies that you cannot just download a zip file and install the package on your local computer in order for the library to work, you'd need a way to track all the dependencies...
PHP (composer) or Java (Maven) are less prone to that issue because a composer package can't have 50 versions of the same dependency, unlike NPM. So even if a composer package has 20 dependencies, it's relatively easy to track down and download all of them. NPM dependency tries are often exponentials, a stupid design decision. Version conflicts should be solved upstream, not by the package manager.
But that decision allowed NPM to grow as a business which was eventually bought by Microsoft.
Why not go straight to the source code host?