One of the things I love about the Debian ecosystem is that there's a tool that's even better than what this post recommends. Debian update-alternatives, available on any Debian-derived distro.
update-alternatives allows you to compile or unzip software to somewhere obscure and safe, like /opt, then soft-link it to /usr/local with update-alternatives --install. The benefits:
1. You can slave all dependencies to the main binary, so that they can all be removed with update-alternatives --remove.
2. You can install multiple versions simultaneously (including the system version), flag one as active, and toggle between them easily with update-alternatives --config.
3. #2 lets you upgrade to a new version (say of Java, Scala, Haskell, whatever) without uninstalling or overwriting the current version, and if the new version breaks your app you can toggle back to the known working version with a quick update-alternatives --config.
4. Keep all the files together in their own directory, like /opt/java/jdk/1.7.0_07, instead of strewn all across /usr/local, /usr/bin, /usr/lib, /usr/share, etc. Soft link them instead into their appropriate locations.
I created update-alternatives install scripts for Haskell Platform [1], Java [2], and Scala [3] for anyone interested in seeing more how it works.
I remember seeing a different tool which takes care of all that for you so you don't have to write scripts, but I forget its name now. It's definitely a good idea (slotting software installs) but it needs more automation.
I only use the scripts to slave all the dependent binaries to the main binary. If you only have one binary, it's just a single line. (I also script literally everything, even single-liners).
I don't know if there's any way to do something like that without update-alternatives or your tool having some built-in awareness of the package's dependencies, but that's not really possible for a tool designed to be able to handle literally any arbitrary, user-compiled/installed package. Otherwise you get the system package manager again.
The tool I remember would basically do the equivalent of:
for file in `find /opt/package_name/version/` ; do
NEW=`echo $file | sed -e "s/^\/opt\/package_name\/version\//"`
ln -s $file /usr/local/$NEW
done
Where the base files are installed in /opt/package_name/version/ and all of them are symlinked into /usr/local/ by default with the option of changing individual file links. This way you change the entire 'installed' version of a package without needing application-specific path modifiers. But you can still build applications against specific versioned directories (if you have an app which requires BerkeleyDB 4.1 which conflicts with 4.0, for example)
(You can automate that for a pre-existing package if it's got relocatable code, but hardly as reliable as building it from scratch for the versioned directory and changing symlinks in /usr/local)
Possibly... I don't remember the tool I saw using an LD_PRELOAD, and I think it operated on post-make-install files (like a DESTDIR staging directory).
Anyway, the mini script I provided below is like half the functionality you would need after the source is compiled to a PREFIX of /opt/package/version/. Perhaps i'll write a tool when I get bored :-) The tool and method i'm borrowing from a proprietary tool I saw once, but obviously other people have implemented it too.
Paco has a great idea though, and that's a meta-format and wrapper for different system packages. If you have to support five different production operating systems and the same software on all of them, switching between package managers is a pain in the ass. An extra tool to cross-reference paco to the OS's packages could make it really handy for resolving software updates in a unified way.
> When writing scripts, specify the full path to the executable you’ve installed in the first line of the script.
No. Don't do that. Bad. Even for maintenance and admin scripts, that makes them severely less portable and harder to keep in sync across machines for various reasons. Your path is setup correctly, just use that: there are very few reasons for your shebang lines to be anything other than:
#!/usr/bin/env $executable "$options"
This will make the shebang work using the normal rules of path-resolution, and means path-override tools and methods (e.g. python virtual environments, or `PATH=$somedirectory ./script`) will work correctly.
edit: I am told not all versions of shebang/`env` support passing options to the $executable, and most notably it doesn't work on Linux. So don't do that, and i won't in the future. Still, $executable is usually sufficient
While I agree about the use of /usr/bin/env, you should be aware that some systems will perform only one word split on the shebang line. That means that if you use env, you can't pass any options to the $executable.
You can try it if you want, save this into an executable file (test.awk):
Linux doesn't support options, unfortunately. The executable path /usr/bin/env is parsed, then the remainder of the line is used as a single argument, spaces and all:
$ cat x.py
#!/usr/bin/env python -i
$ ./x.py
/usr/bin/env: python -i: No such file or directory
Coming from a c and python perspective, I'm amazed that you even can specify which ruby to run it with. Either its a compiled binary, or you call the interpreter with the code object as an argument. What is the design impetus for mixing scripting and executable styles this way?
Shebangs (#!) have been around for quite a long time. It's actually really handy to be able to write a script that you can execute as a command. See https://en.wikipedia.org/wiki/Shebang_(Unix)
It's a little more complicated than that. Using /usr/bin/env in your shebang is effectively a way to say "I don't care which executable you run this with". If the executable doesn't actually matter, that's fine - bash scripts probably don't care whether bash is /bin/bash or /usr/bin/bash, so it makes sense to just use whichever is in the path.
Once you start getting into Python/Ruby/Perl/etc scripts, the executable actually does matter and relying on env can make your script fragile with respect to path changes. Different executables have access to to different collections of installed libraries, and it's important to make sure that your scripts are run using the executable that has their dependencies installed. If you use env shebangs, there's no way to do that - you're limited to either using the first executable on the path or manually specifying the executable on the command line to bypass the shebang.
I'm not saying that you shouldn't use env shebags, because they really are fine for a lot of scripts. However, you should always be mindful of what they mean and how that could break your script in the future.
My experience is exactly the opposite wrt fragility: fully hardcoded paths assume that the hardcoded path can and will have all the required dependencies, which — at least for modern Python development — is always the wrong assumption to make. Essentially, it only ever works if the script is run on the same version of the same distribution with the same package manager and installed packages as the original developer of the script. This is the most brittle a script can get.
> it's important to make sure that your scripts are run using the executable that has their dependencies installed.
Which a hardcoded shebang path does not do, it assumes these things but does not assert them and does not "make sure" of anything.
> If you use env shebangs, there's no way to do that
There's no way to make unwarranted assumptions is about as far as it goes: using env shebangs, the caller of the utility has control over what happens and which executable gets called (even indirectly)
> you're limited to either using the first executable on the path or manually specifying the executable on the command line to bypass the shebang.
Or specify a PATH when calling the script, or run the script in overridden PATH environments (e.g. virtualenvs), or...
> I'm not saying that you shouldn't use env shebags, because they really are fine for a lot of scripts.
And I went (and would still go) much further and state that they're the right solution for the vast majority of scripts and utilities.
> However, you should always be mindful of what they mean and how that could break your script in the future.
So should you be with hardcoded shebang paths, and they are much more likely to break a script across porting than env shebangs.
I've read better articles on Unix. Just a few thoughts:
1) Software packages should probably install into /usr/local by default, not /usr as the author suggests. Distributions can always pass an option to install into /usr in their build scripts. In my experience as a distro packager, plenty of package-build/install systems install to /usr/local by default (autotools, python's disttools IIRC, etc).
2) FreeBSD (and likely other *BSD OS) ports install into /usr/local. So, if you're using the ports system for package management, you might still be screwed. (Of course, if you, as an admin, don't notice that you're installing over system packages, your system is going to end up hosed some day anyway.)
At the end of the day, go ahead and use /usr/local. But be aware what your system does with it, and that other 3rd party packages may randomly scribble all over it as well.
Regarding point 2, the ports system works a little bit differently to typical package managers, and easily accomodates running customised versions of the software you need.
Another difference between FreeBSD and (most) Linux distros is the concept of the "base system". Things in /bin and /usr/bin are part of the base, any additional packages you install end up in /usr/local. This has the nice effect of separating things you really, really need to work for the system to be functional (i.e. boot and log in) and software that is "nice to have" but not strictly necessary (your webservers and Rubies and what have you). I also find that /usr/local/etc is generally very small and tidy, which makes administration and backup a doddle.
I think the author of the post is a little bit muddled on what /usr/local is for, though - the analogy to installing applications to ~/Applications on OS X is completely flawed, as in OS X that installs things only for the current user, whereas /usr/local is available to all.
True, the linux equivalent would be ~/.local . What I usually do for software I need only for my user is to install it in ~/.local/<package name> and then symlink what I need into ~/.local/bin,lib,etc.. with bin being on my PATH. If the binary needs to see its own libs, I write a small script that sets LD_LIBRARY_PATH prior to execution and use that. This way removing is simply rm -r ~/.local/<package> and I don't pollute the rest of the system.
I see a lot of comments lamenting the downfall of HN recently and they almost always focus on poor comments. In my opinion the troublesome trend on HN has been stories like this (or yesterday's bash redirection cheat sheet) making it to the front page. This is Hacker News not Un*x For Dummies.
Meanwhile, outside your ego, some of us actually found that useful, unlike all the reposted tech crunch, naive life advice and "how to get funding like a baws" articles.
I'm sorry, I did not mean to be egotistical. I guess I assumed that no "hacker" had typed `cd /usr/local` or `cd /etc` without wondering about origins of the names enough to search out the most cursory explanation.
The thing I think your assumption ignores is that there are infinite things that we encounter on a daily basis that have interesting explanations. Different people, even different "hackers", are going to be attracted to different topics to explore. Sharing the topics that we've explored is exactly what I come here for.
We all type all day, but we haven't all dug into how keyboards work. We look at screens all day, but we haven't all learned about monitor technology, sometimes even at the "cursory" level. Not having done it yet doesn't disqualify us from being real "hackers".
I have lurked/read hacker news for a few years before creating an account, and frankly, stories like this have always made the front page, MORE frequently in the past if anything.
If you look for an issue, you will surely find one.
What is with the complaints about the "downfall" of quality recently? If you don't like the article, just don't upvote it. Writing a comment for the article about how it shouldn't be here isn't serving any purpose.
Dumping everything into /usr/local doesn't even work well when you're the only user on the system: most packages don't define an "uninstall" make target, and installing multiple versions of things is often not well-defined.
See e.g. checkinstall, GNU stow, encap, and probably several more that I'm missing for examples of people trying to fix the crappiness of ad-hoc (non-distribution-based) package management.
Thanks! I didn't know there was such a thing as $MANPATH!
But, I don't think it's necessary. I don't have `/usr/local/share/man` in my $MANPATH (I don't have a $MANPATH at all, as a matter of fact), yet `man sl` works (and by running `sudo opensnoop` in another shell process I verified that groff indeed pulled the data from `/usr/local/share/man/man1/sl.1`)
$ man man
. . .
man uses a sophisticated method of finding manual page files, based on
the invocation options and environment variables, the /pri-
vate/etc/man.conf configuration file, and some built in conventions and
heuristics.
. . .
$ fgrep local/share/man /private/etc/man.conf
MANPATH /usr/local/share/man
MANPATH_MAP /usr/local/bin /usr/local/share/man
MANPATH_MAP /usr/local/sbin /usr/local/share/man
$
(OS X 10.8.n, for some unannounced future value of n, but AFAIK this behavior is unchanged since 10.4 Tiger)
You're right. Good to know that man checks this path by default.
I'm using a custom path for homebrew, that's why I added /usr/local/homebrew/share/man to the $MANPATH.
It's a nice article and a cute history lesson... but unless you're going to be deploying that website to a production server also running OS X, you will be much better off in a variety of ways installing Lighttpd and Rails to a virtual machine and developing against that.
It's worth noticing the article has a 2005 date on it. I suspect there's a lot of stuff the author (Dan Benjamin of the 5by5 podcast network) would change now.
With /usr/local, though, you still have to use `sudo` - unless you're crazy and chown or chmod it appropriately. I've been a longtime advocate for $HOME/.local, which is (a) obviously writable by you, (b) on your home partition, so it's backed up with all of your data, and (c) a common enough standard (see the XDG Base Directory Spec) that it's generally easy enough to get applications to use it. Just add $HOME/.local/bin to your $PATH and you're good to go.
Given that homebrew tends to occupy /usr/local, I've grown comfortable with making my own ~/local folder with src, bin and app/server folders. I tend to use it for just about everything one might want in a /usr/local directory.
Can I install my own stuff to /usr/local?
Yes, brew is designed to not get in your way so you can
use it how you like.
Install your own stuff, but be aware that if you
install common libraries, like libexpat yourself, it may
cause trouble when trying to build certain Homebrew
formula. As a result brew doctor will warn you about
this.
Thus it’s probably better to install your own stuff to
the Cellar and then brew link it. Like so:
$ cd foo-0.1
$ brew diy
./configure --prefix=/usr/local/Cellar/foo/0.1
$ ./configure --prefix=/usr/local/Cellar/foo/0.1
[snip]
$ make && make install
$ brew link foo
Linking /usr/local/Cellar/foo/0.1... 17 symlinks created
And really, writing your own formula is easy if manual steps are easy, and if they're hard, well, writing a formula makes you do it only once, since you'll be able to reuse the formula later.
Anyway brew builds in Cellar, and symlinks in /usr/local but never steps on what you would have installed manually.
When you're a user installing software, you don't get to write to /usr/local. You have to use $HOME/usr or something and modify your path.
When you're an admin on a box, you can install software anywhere. You could totally './configure&&make&&make install' and your software lands in /usr/local. If you're lucky, there won't be any conflicts with existing earlier versions in /usr. But there's consequences.
Now you've built a lot of software and some of it depends on stuff you put in /usr/local. Now you need to upgrade. But is something going to break? What's the compatibility? How can I track what has been installed? Can I uninstall anything?
You don't realize how big of a mess you're getting into until you have to remove all of /usr/local just to get some working apps and start from scratch. Take the time to learn the package manager for your system, and package your software before installing. Hell, use checkinstall or some other tool to automate it.
I use my scripts called slacktools [https://github.com/psypete/public-bin/tree/public-bin/src/sl...] to automate packaging in Slackware. One command 'slackpack foo-1.4.5.tar.gz' builds me a Slackware package. Tiny modifications could make it build rpms, debs, etc. The benefit you get is actually upgrading old packages do you don't have conflicting versions, a sane default path, and the ability to remove or upgrade the package when necessary.
Disappointed to see no mention of GNU Stow there, very handy for /usr/local having multiple version of foo installed and being able to make one of them current, e.g. rewind if a new one breaks things. http://www.gnu.org/software/stow/
Linux isn't divergent, except for the common practice (though not quite a standard) of using /opt for non-distro package-installed software. All autotools-based packages drop themselves into a prefix of /usr/local by default, which makes the admonition to pass --prefix=/usr/local to configure a little strange.
Almost all software I've custom built over the last 13 years of using Linux has, by default, set the prefix to /use/local. I don't know if ruby is for some reason different, but installing software outside of /usr/local is not something to snark at Linux about...
No snark intended. I mean only to recognize BSD in the discussion as having made another choice. And yes, Linux custom-built software goes into /usr/local without a hitch.
But really, the question comes down to layers of abstraction. Is /bin the base system? Is /usr/bin? Is /usr/local/bin? Probably not on the last one, but Linux distros have chosen /usr/bin as where all packages install. And BSD treats packages as user programs and goes to /usr/local/bin.
I'm not saying one is right. At best, I'm advocating use of "#!/usr/bin/env foo" instead of some of the assumptions made by some shell scripters.
But really, the question comes down to layers of abstraction. Is /bin the base system? Is /usr/bin? Is /usr/local/bin? Probably not on the last one, but Linux distros have chosen /usr/bin as where all packages install. And BSD treats packages as user programs and goes to /usr/local/bin.
As I've always understood it, since /usr can be on another partition (to keep / small, or keep it from filling up with stuff from /usr, /home, /tmp, and the like), /bin should contain "essential" binaries for booting that weren't necessarily "system" tools (those would go in /sbin). Now that I think of it, I can't really think of binaries essential to booting that aren't "system" tools, but my Linux boxes typically have about a hundred CLI programs that are basic, but not essential (AFAICT) to booting in there. They're all from packages.
The way I understand it, the 's' in "/sbin" does not stand for "system", but for "superuser" - essential system tools which need superuser privileges (insmod, ifconfig, fsck, ...)
/bin should contain all the non-privileged essential tools for system to boot up and execute all its initscripts - stuff like echo, cat, grep, awk, sed, chmod and many other everyday tools.
In addition, it should contain tools which will allow you to repair a broken system in minimal environment (where only / is available to you). So, stuff like ls, mount, gunzip, dmesg, ...
Apparently most modern Linux distros cannot boot anymore without /usr. Therefore people suggest to link /bin to /usr/bin etc. /usr/local still has its place, though.
I'm pretty sure Debian (my Linux of choice) can boot without /usr. And I love how on that page they bring up BS such as "well Fedora has 450MB (of crap) in /, so that's not minimal". I've got at least one Debian box packed to the gills with programming languages, scientific software and who knows what else, and it clocks in at right around 129MB for /. Maybe Fedora is doing it wrong; wouldn't surprise me, as I switched from RH to Debian over a decade ago because of dependency hell.
I'm not sure linking /bin to /usr/bin is a good idea; it may have merits, but many of the reasons they bring up seem dubious ("Solaris is doing it!", yeah like Solaris should be the gold standard; "hard coded paths!" - recompile). I'm not opposed to progress, but some change isn't progress. It's interesting to me that this initiative is being pushed by Lennart Poettering - well known for his work on PulseAudio and systemd, two other controverisal Linux subsystems. While systemd is still unproven (and quite frankly, Poettering's talk at Linux Plumbers about systemd subsuming more control scared me; it's like a hydra!), PulseAudio appears to have finally become stable and usable. I guess time will tell with both systemd and /usr-merging.
Another interesting point from Poettering's talk was that he wanted to take more after BSD, especially having the one "correct" way to do it. Me, I like having options, especially in the case of "what if the 'correct' way doesn't work?" (http://news.ycombinator.com/item?id=4409768).
It's the BSD approach that's divergent. /usr/local is supposed to belong to programs that have been manually installed by the administrator. BSD's awkward and frankly unhelpful split between packages and ports means they need one more level of hierarchy than linux (rather than package-managed programs and manually installed programs, you have package-managed programs, ports-managed programs and manually installed programs); unfortunately they've taken over /usr/local for the ports-managed programs, leaving you with nowhere for manually installed programs. And while ports purports to support a configuration variable for changing where ports software gets installed to, it must not get tested because a number of programs break if you try and actually use it.
I still wish this wasn't all that hard-coded. A plethora of prefixes, the whole "/usr" directory from the start (which used to be "/home" back in the days), compiling software to depend on variant paths, huge PATH variables and the hassle to define them in the proper place, application data all strewn about etc.
Yes, there's stow/homebrew like deployment with app-specific directories and linking to solve some of these problems…
But I really wish union mounts would be more common, both global and on a per-process basis. Thus you'd just have a /bin directory and just put everything in there, no matter on what partition or under which "real" directory it actually is.
> On many UNIX systems, mounting drives has become automatic, and the mount points for the drives live in a folder called /mnt.
No, no, no, no, no. This hasn't been the case since at least eight years ago, when FHS 2.3 (released in Jan 2004) clarified the purpose of `/mnt` and added the `/media` directory. `/media` is for mountpoints for removable media, and `/mnt` is for temporarily-mounted filesystems.
"not knowing, she accepted the default paths at compile time, and the [Ruby] binary now lives in /usr/bin" wat? Ruby's default is to install to /usr{/bin,}?? Automake and cmake at least always default to /usr/local{/bin,} as the prefix in my experience. Other than not mentioning that, yes more people should know the various prefix conventions, it would save a lot of installation hassle.
No, Ruby's default is to install to /usr/local. It was just an illustrative example used to demonstrate potential complications from not using /usr/local. The submission wasn't a guide to building Ruby.
I use /opt/ld/{bin,sbin, ...} myself, on the grounds that it's short, yet just confusing enough to be unattractive to future filesystem standardizers. By sheer coincidence, it's now a backronym[1].
I remember an argument 10 years ago where a (very knowledgable) coworker swore to me that /usr/local was supposed to be world writable location for any user to install software.
He also claimed that he didn't save non-compilng code while he was editing.
update-alternatives allows you to compile or unzip software to somewhere obscure and safe, like /opt, then soft-link it to /usr/local with update-alternatives --install. The benefits:
1. You can slave all dependencies to the main binary, so that they can all be removed with update-alternatives --remove.
2. You can install multiple versions simultaneously (including the system version), flag one as active, and toggle between them easily with update-alternatives --config.
3. #2 lets you upgrade to a new version (say of Java, Scala, Haskell, whatever) without uninstalling or overwriting the current version, and if the new version breaks your app you can toggle back to the known working version with a quick update-alternatives --config.
4. Keep all the files together in their own directory, like /opt/java/jdk/1.7.0_07, instead of strewn all across /usr/local, /usr/bin, /usr/lib, /usr/share, etc. Soft link them instead into their appropriate locations.
I created update-alternatives install scripts for Haskell Platform [1], Java [2], and Scala [3] for anyone interested in seeing more how it works.
[1]: https://github.com/byrongibson/scripts/tree/master/install/h...
[2]: https://github.com/byrongibson/scripts/tree/master/install/j...
[3]: https://github.com/byrongibson/scripts/tree/master/install/s...