>Wget requires no extra options to simply download a remote URL to a local file, while curl requires -o or -O. However trivial, this fact is often mentioned to me when people explain why they prefer downloading with wget.
Funny because that's exactly the reason I installed macports and then wget. It's stupidly trivial, but there you have it.
I can type 'wget' and then do command-V entirely with my left hand, keeping my right hand on the mouse. Having to type 'curl' would require me to move my hand back and forth, slowing me down.
I love reasons like this, they make non-geeks look at us like we're insane. I've only used curl personally, so this wouldn't be enough to switch, but it's an excellent point.
I don't think it's all that trivial; whatever a program does when run without options should be a big clue towards the primary use case that it is designed to handle or fulfill.
In the case of curl, it writes to stdout; wget writes to a file. While the mechanics of each are the same, curl lends itself towards use in scripts while wget (at least in my experience) is better if you're kicking it off interactively.
Also, wget has a very handy batch mode, where you can feed it a file containing nothing but a list of URLs and it will fetch each one. I am not familiar with a similar one for curl -- although admittedly you could do the same thing with a minimal wrapper script, that's a lot more keystrokes than "wget -i <file>".
True, much simpler. I guess a larger reason would be that 7+ years of using wget has wired me to its commandline. I'm so used to typing `wget -c "<middle-click>"` or something on the command line, that I just never considered creating aliases for alternatives.
mini[~ 111] uname -a
Darwin mini.local 10.3.0 Darwin Kernel Version 10.3.0: Fri Feb 26 11:58:09 PST 2010; root:xnu-1504.3.12~1/RELEASE_I386 i386
mini[~ 112] which fetch
fetch: Command not found.
Unfortunately, it does not. I use fetch all of the time on FreeBSD, so it is a little obnoxious to have to go and install wget from MacPorts.
My main complaint about curl is that it prints a bunch of crap (well, statistics technically) to stderr by default. Especially for something claimed to be (at least in this article) "traditional unix style", that seems undesirable.
It's annoying when I do 'curl $URL | less' and end up with garbage in my pager that I have to scroll around a bit to make disappear.
It seems like default behavior should be more like 'curl -s -S' -- silent, but print errors if they occur, just like...well, every other sensible command-line tool in the world (so that's what I have curl aliased to).
wget of course prints statistics too, but it's not such "pipe-oriented" tool, as the author points out, so this isn't as big of a deal.
My version of curl only prints the progress junk to stderr if you pipe stdout somewhere. I guess I've never had a problem since I don't use a pager usually.
BTW, how do you detect whether your program's stdout is being piped to something other than the shell?
> My version of curl only prints the progress junk to stderr if you pipe stdout somewhere.
Hmm, good point -- I actually hadn't noticed that, and it seems to be the case on mine as well. (Though I rarely use it without piping or stdout-redirecting to something, so it still strikes me as inappropriate.)
> BTW, how do you detect whether your program's stdout is being piped to something other than the shell?
In the three languages I've used recently enough to remember:
C: int isatty(int fd); /* from unistd.h */
Python: os.isatty
Shell (bourne): [ -t 1 ] # exits 0 if stdout is a tty, 1 if not
Instead of os.isatty(sys.stdout.fileno()) (ew) you should just use sys.stdout.isatty(). In addition to being shorter, it will work everywhere [that Python does], not just Unix.
Actually, it is very much unix style. With any modern shell you can simply turn off stderr, or only put stdout in the pipe, or put 2>/dev/null and so on. If both went to stdout then it would be a reasonable complaint.
> With any modern shell you can simply turn off stderr, or only put stdout in the pipe, or put 2>/dev/null and so on.
Or I could type fewer extra characters by using the '-s' flag, as I described in my comment. My point was about its default mode of operation: "noisy" (which is not, as I see it, a characteristic of "traditional unix" tools).
Edit: Additionally, if both went to stdout the tool would be completely unusable -- I'd simply file that under "data corruption".
That is not really a good assertion. Look at grep: default output is filename, line number, relevant line. This is far from a minimalist output. Or du, which by default lists all the files and subdirectories and their sizes. Minimal, quiet output doesn't define "traditional unix" style -- the ability to taylor the output to the situation, and further make the output as machine readable as possible is what makes it "unixy".
This is probably going on longer than it merits, but...
Those aren't really good comparisons.
Perhaps you have a different version of grep than I do, but my grep only prints filenames when more than one file has been specified for searching (in which case showing which file the match is from is necessary, I'd say), and certainly doesn't print line numbers if I don't give it '-n':
[me@host: ~]% grep PATH /etc/bashrc
if ! echo $PATH | /bin/egrep -q "(^|:)$1($|:)" ; then
PATH=$PATH:$1
PATH=$1:$PATH
[me@host: ~]% grep PATH /etc/bashrc /etc/profile
/etc/bashrc: if ! echo $PATH | /bin/egrep -q "(^|:)$1($|:)" ; then
/etc/bashrc: PATH=$PATH:$1
/etc/bashrc: PATH=$1:$PATH
/etc/profile: if ! echo $PATH | /bin/egrep -q "(^|:)$1($|:)" ; then
/etc/profile: PATH=$PATH:$1
/etc/profile: PATH=$1:$PATH
/etc/profile:export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE
/etc/profile:export PATH
[me@host: ~]% grep -V
grep (GNU grep) 2.5.1
As for du...well, frankly I'd prefer if du's output were less verbose -- a default more like 'du --max-depth=1' would be more to my liking. Compare to 'ls' vs. 'ls -R' (though I realize du does need to operate recursively regardless of whether it prints a line for everything it finds, so perhaps that's not an ideal comparison).
But the main difference here is the nature of what they're printing. In both of the above cases, the extra output text is still much more immediately relevant to the program's real task than what curl prints to stderr. For a more direct analogy, it would be like grep printing to stderr '50MB searched, 974MB remaining...' as it searches through a 1GB file.
* curl for any bash scripts
* wget for downloading via command line
The error codes that curl output make it very powerful when creating a script. Something that gets run on cron & you want to know why it failed if it did.
Wget is simple, and for a shell it's what I always turn to. The only time I would use wget in a script is if I was making some sort of simple web crawler script.
same here. I use wget to do stuff, and I use curl to do complicated/fancy stuff. One is simple, one is powerful. I'll use simple whenever I can get away with it.
I use curl for almost everything. Especially for debugging caching and whatnot, using curl -D - -o /dev/null http://example.net/ makes it extremely simple to see the headers the server is sending for a certain file.
Not only that, but curl is installed standard on Mac OS X whereas wget is not, also, most attacks that attack PHP tend to use wget to download further files to the target system, which means that when wget is not available no extra files are downloaded and it generally stops the exploit cold in its track as the attacker does not try again.
> most attacks that attack PHP tend to use wget to download further files
Nowadays, neither Wget nor Curl is required for downloading in PHP, because functions like file_get_contents() are able to handle URL arguments! So a missing Wget protects only against script kiddies who don't know PHP very well and copied over some outdated code snippets.
Also, not installing Wget "protects" you just from those kind of primitive attacks to which your system shouldn't be vulnerable in the first place.
Security is achieved by sealing entry doors, not by removing tools.
Also, installing wget is just one more way to defend a server that is hosting multiple sites with multiple scripts all of which can not be trusted all the time. I don't know if tomorrow a new bug is found in phpBB that allows a remote exploit that runs as the user php runs under, if not having wget stops one such exploit it is a win in my book.
The other thing I have in my firewall is all outgoing access is blocked for the user that PHP runs under so that in the event something does happen and something is executed under that user it won't be able to make an outbound connection or get inbound connections.
Yes, security is achieved by sealing entry doors, but removing an entry door in the first place may be even better, wget being an entry door, unfortunately the front door is a huge gaping mess you could drive a truck through, but that is unfortunately required for the business I am in, administrating a server with multiple websites running various PHP scripts.
> file_get_contents on most hosts does not allow external URL's
This is still not really a gain in security, but ...
> The other thing I have in my firewall is all outgoing access is blocked for the user that PHP runs under
... this is! And since you do that, there is no need for strange measures like providing Curl but not Wget.
Wget, Curl and allow_url_fopen are just tools to access the network API. So you need to block that API, rather than blocking some tools which merely use that API.
Although I fully agree that it improves security if you remove all services that aren't needed, I don't think the same holds for tools. (except if they are SUID binaries, but such a program is in fact more a service than a tool)
Good points all round, X-Istence is suggesting defense in depth, and you (vog) suggest making sure the outer perimeter is secure. Both good ideas - even if you fill the hall with barbed wire, you should make sure to lock the front door. Beyond that, it's a trade off.
Having outbound access from the PHP user blocked is not always possible due to client needs to interact with remote services or API's (Flickr, Google, Twitter). Generally those run under a different user (yay FastCGI) but still can leave holes exposed that may not be easy to fix.
If having no wget installed removes one or two possibilities for an exploit it is a win in my book.
> Especially for debugging caching and whatnot, using curl -D - -o /dev/null http://example.net/ makes it extremely simple to see the headers the server is sending for a certain file.
Working on web stuff, I'd needed to see server headers many times, and I have always used the HEAD command, that is part of the libwww-perl package on ubuntu. It's just: HEAD http://example.com.
The libwww-perl package also comes with GET (which is similar to curl in that it sends the retrieved content to standard output) and POST (which sends a POST request to a server) commands which are very handy too.
That's a nice breakdown... I've been a curl user, and never really knew what the major differences were. And it doesn't seem overly biased to me. Many thanks to the author!
I find wget is almost always what you want, unless you're interested in development details like headers. If I want output to a pipe, I use wget -O -.
In scripts, I use wget -o <log-file> (or -a) so that I can tail -f to keep track of it. I don't see off hand what curl's log file support is, so I can't comment too deeply on it.
When downloading via HTTPS curl seems to barf error codes often so I try wget first, and use curl only if wget doesn't work or curl is easier (which isn't often).
wget has a better-designed command line API and documentation, although its inability to PUT or POST is rather unhelpful. For example, curl uses --include to output full headers (wget uses --save-headers) and, more inexplicably, --request to change the request method, even though --method is available.
As someone who have always used curl and wondered why I often see wget quoted in READMEs (as how to obtain some remote file), I quite enjoyed reading the write-up.
My take-away was that a) I am not missing out by staying with curl and b) the use of wget is likely due to being part of GNU.
You did notice this was written by curl's author? I only noticed after I read it and without that knowledge thought it was rather a pathetically biased article.
But you can't expect developers to be impartial about their own code and it was my fault for not noticing the disclaimer the first time. It reads completely different with that framing in mind, but still if I was looking for the definitive reason that one tool was recommended over another I'd consider the point of view of the developers as a starting point rather than an open and shut case.
Funny because that's exactly the reason I installed macports and then wget. It's stupidly trivial, but there you have it.