Yes, it's a thing, and aside from the lack of HTTPS it's not much worse than your standard "./configure", "make", "make install" install flow of the last 20-whatever years of open source. If you don't trust the developer to serve you a safe install.sh script you shouldn't trust the rest of their code either. If you don't trust them you should audit the code no matter what the install process looks like.
That said, if you're going to do this, you should do the following:
1. HTTPS only. No excuses.
2. No URL shorteners, which make or may not use HTTPS, and introduce another point of failure.
3. Ideally serve directly from Github repo. Eliminate another point of failure. The user can be reasonably confident that the install script is the same one that's in the Github repo.
It's a lot worse than the standard configure, make and make install.
For one, it's only requiring one simple script to get compromised and replaced to compromise your entire machine. At least when I download the tarball, unpack it, etc. I've got several steps before I even get to the make install that I would run as root. And I might not even do that depending on my install target. If I do, I have ample opportunity to verify the package in question and examine the script I'll be running via sudo before I do so.
With the instructions provided by this site, not only is there no opportunity to verify the script I've received is the one I wanted to (HTTPS alone is not enough for this, the site or domain may have been compromised) but they're instructing me to just run it immediately as root without any examination.
This is a terrible practice, and while yes it has been around for a long time in the past it was something only done and encouraged by idiots, it seems to be gaining popularity.
This is a bad idea, stop doing this people.
Provide an install tarball and provide a checksum of it that we can verify.
I'm a Windows user. Virtually every piece of software I install comes in the form of an executable, that I download-and-run directly from the browser. I often give this executable administrator priviledges. The executable can then do whatever it wants on my computer. Usually it will install said piece of software (except when it was made by Oracle, it'll install said software and the Ask.com toolbar)
This has been standard practice for decades. It's very comprable to that wget oneliner. Why is it bad? Or, why is the wget oneliner bad but not Windows installers? Do Windows sysadmins compile tarballs and verify checksums? Or are Windows sysadmins simply dumber/less paranoid than Linux sysadmins?
You're giving those executables full control over your computer. Yeah, you won't get burned very often (depending on where you're getting these executables.) But when you do get burned it's going to be catastrophic. That's the problem.
How would you then recommend installing non open source software on a production system? E.g. a freeware (but proprietary) FTP server or a database engine.
1. Don't install software from untrusted sources.
2. The trusted sources should provide checksums so you can verify the package before installing it.
This protects you from a compromised package so long as the page you accessed the package from has not been compromised (if that page is compromised, the checksum could be changed to the new one).
I have no idea how you would verify closed source software if the checksum and package are both corrupted...
"2. The trusted sources should provide checksums so you can verify the package before installing it."
If the checksum is published to another trusted location, in addition to where you download the files, that would help in the case that an attacker compromises both the checksum and the package on the download site.
The developer could post the checksum to Twitter at the same time as making the package available for download. An attacker would need to compromise both Twitter and the download site.
If you are taking recommendations, I wouldn't recommend installing them at all. You could ask the authors to support a platform which has a better security model.
(Given that there are open-source and robuster-than-Windows platforms freely available.).
That was a terrible recommendation. Proposing a solution when you know nothing about the situation nearly always results in a pointless solution.
Suggesting that the best way to securely install Windows packages is to use a different operating system smacks of immaturity, and is obviously not a practical solution in Windows environments.
A better suggestion might be: If you do not trust it, run it in a VM/sandbox.
Blindly installing windows executables and giving them admin privileges is at least as stupid. This being common practice is probably the primary reason botnets are easily able to exist. This being common practice is why windows sysadmins disable users from having admin privileges. Because apparently you're too dumb to be trusted with them.
My point was the people who are going to blindly run curl/sh commands are also going to blindly download the source tarball and run configure/make/make install, and the people who want to do more verification are certainly free to do so.
Yes people are going to dumb things. But that doesn't mean you need to encourage and facilitate it. That's my point. If you know better you should be facilitating the proper way to do it, not the worst way.
True. Though for a bit of benefit, if the download was hosted separately from the site, that'd be two things to hack. The downside, of course, is that script updates would require new SHA1s. It would be best, then, to insert version numbers into the installer scripts, and a version check. That way if a new version comes out, the installer could warn about it (y/n), while still verifying the SHA1.
No, they're not depriving you of any opportunities to verify. They're offering you a shortcut if you decide that you trust them enough to forgo examining thousands of lines of code. If you decide you want to review the code, just wget it down and then install it after you review it. And if someone doesn't know enough command line to do that, are they really going to be in a position to audit the code?
>>>>Provide an install tarball and provide a checksum of it that we can verify.
This 100%.
When I first started experimenting with Linux after being a lifelong Windows user, this was the same advice I got from a seasoned Linux developer: ALWAYS COMPILE FROM SOURCE - NO EXCEPTIONS.
He just told me even though Linux is secure and not as big a target as MS, he said you want to establish good habits from day one and not get lulled into a false sense of security.
What? You don't need a tarball to be able to verify the package via a checksum. In fact, a good packaging system will do checksums automatically, plus offer public key signature verification. I know deb and rpm files do that, and I believe many Windows packaging systems also provide checksums and public key signing. So do apt, aptitude, yum etc. On most of these systems, you even have to manually disable this option if you don't have the repo's public key.
In fact, of all the install methods I use regularly as a Linux user, the only one that does not offer automatic checksums and public key signing is make. You can certainly verify checksums and public key signatures, but you have to do it manually.
Do you look through all the code you build from source? Highly unlikely. If not, what's the point?
Security is a lot more social screening than many people in the tech world would like to think. Know and trust the people you are running code from. If you don't know and trust them and still need their code, that's when you want to spend a couple of hours reviewing the source.
Why? If you don't trust your Linux distributor then you need to find a different one, but I don't think the answer is compiling everything from source; specially if the upstream project doesn't support 100% your target distribution.
I think he told me this so I would get into the habit of doing it. I was so used to just clicking on an executable file and just installing the program without ever thinking it may contain malicious code.
I can tell you I don't do this every time. Now if I'm installing a 3rd party extension or plugin, I usually do it in a sandboxed environment, run some pen tests on it to check how secure it is and see if there's any malicious code before I install it on my machine or server.
And yes, even then, it is still possible to install malware on your machine.
The install.sh goes off to github and pulls the latest - it's not a method for keeping the same version across machines. It may be fine for your needs, but it'd still be nice to have 'stable' or suggested versions to use.
I don't see any rational basis for this. If you weren't vetting the code you downloaded before, you don't lose anything by blindly running the commands on one line instead of three. And if you do want to vet the code, you know what to do. Practically no installation instructions include a "vet the code" step, so you aren't losing anything here.
The lack of TLS is worrisome, but that's not so much "becoming a thing" as it is "a bad thing we're just finally starting to get rid of."
Although this isn't new, it should make you throw up in your mouth a little bit.
Malicious compromise is not the only way that an install script can become broken, or corrupted, on a remote site. The ability to checksum what you're running is a very useful sanity check and you should be doing it with all downloaded scripts/packages.
> Malicious compromise is not the only way that an install script can become broken, or corrupted, on a remote site. The ability to checksum what you're running is a very useful sanity check and you should be doing it with all downloaded scripts/packages.
Yes, and you still can do that. But the standard installation instructions almost never include it. For example, nginx's build instructions boil down to "run configure, possibly with some flags, and then run make and/or make install". That includes the exact same vulnerability you complain of here — it never tells you to checksum anything. This is true for essentially all the software I've ever downloaded. So again, still not "becoming a thing."
The assumption is that if you're the sort of person who customarily checks code before they install it, you know how to inject that step.
Your assertion is that if we want to vet the code, there are steps to do that and no conventional installation includes these steps, correct?
Well, getting the "trust us, this is easy" type installation instructions here isn't terribly comforting from a brand new project. That's the sort of thing I'd expect from a source I know. Not from this project that I didn't even know existed before this post.
It's unrealistic to expect those who take security seriously to do so in the first place. Regardless of how often it's done elsewhere.
OTOH, there's a well known and trusted software distributor that does the vetting for you, which this project doesn't have, and I'd much rather do the "vetting-skipped install" if you will for something they distribute instead.
That's all true, but I don't see what it adds to the conversation. It's just a plug for nginx. "Conservative, security-conscious people like older projects with large, trusted organizations backing them" is something of a tautology.
This conversation started when you said you don't see any rational basis for the criticism of running wget/install as root when no vetting takes place on conventional installations. My post wasn't a plug for Nginx, but an imploring to know what you install.
Since I can't know everything that's running on my system, I'd trust that from some place that actually does vetting. It doesn't need to be OpenBSD, but some competent source with an established record for security.
That's not tautology. It's sane practice that I wish more people followed.
The wget/shell install as root is presumptuous in the extreme for a young project with no history. In fact, many young projects, whose security we know nothing about, have similar installations and that's a bad precedent to set and continue.
Yes, but at this point, of course you won't want to use the OP's software, or any other newish project linked on Hacker News. The method of installation (which is what we were talking about here) isn't the problem, because you wouldn't use anything from him because he isn't OpenBSD. And that's OK — you can be as conservative as you like. It just had me confused as to why you were bringing it up.
because you wouldn't use anything from him because he isn't OpenBSD
My sarcasm detector must be malfunctioning. Who said I don't want to use his software or install anything linked on HN because they're new? I've checked out and installed plenty of things linked on HN, not because they're new, but because they're interesting. But lax procedures like the root shell install aren't good indicators.
In fact, I was just going over his install script to see whether it's worth installing and test, but seeing things like this :
# The umask is ridiculous, or mkdir does not conform to POSIX,
# or it failed possibly due to a race condition. Create the
# directory the slow way, step by step, checking for races as we go.
case $dstdir in
/*) prefix='/';;
-*) prefix='./';;
*) prefix='';;
esac
> My sarcasm detector must be malfunctioning. Who said I don't want to use his software or install anything linked on HN because they're new? I've checked out and installed plenty of things linked on HN, not because they're new, but because they're interesting. But lax procedures like the root shell install aren't good indicators.
Oh, my apologies. I think the context of your other comments caused me to misunderstand that one. Your initial comment was simply "Nginx is part of the base OpenBSD install," and I read your later comment as an explanation of that, which made it sound to me like "I'm OK with nginx because it comes with OpenBSD and not OK with the OP's server because he's not a trusted organization." I understand better now.
And I apologize as well. My posts were venturing over to the pushy/rude side and I don't want that to be the case.
Truth is that I do enjoy seeing new projects like this because it forces me to look at things with a fresh perspective. Yes, the install method did give me some pause, but maybe I've been spoiled by things that I'm comfortable with so I welcome the chance to pour over the code for something totally different.
This has been a thing since the mid-90s when you could execute 'telnet sci.dixie.edu 1 | sh' to install an IRC client. It preys on the clueless and one of these days, some project that uses this install method is going to get owned and a bunch of machines will get malware.
Then again, most of us blindly run what we assume to be GNU autoconf scripts and never read Makefiles before we 'make install' as root. Nor do most of us read or audit source code, even for security-sensitive applications.
as well as the header count (which i came here to post), there's another suspicious hard-coded size limit in the number of servers (1000). although you could only crash the system in that case by configuring too many.
there's very little error handling. good c code returns error codes all over the damn place. this hardly has any.
What's a "static web server"? I haven't seen this term.
A webserver for only static pages, i.e. it won't talk to a runtime? But it's "for serving web apps", this means all the dynamic content is pulled from client side JS? Then you would still need programs on the server to answer AJAX requests.
I did not find an explanation on the page or by Googling.
A server that only serves static files. It doesn't enable the use of dynamic content like CGI[1]. Compare this to something like Apache or nginx. Both can serve static files but also support a number of ways of either directly running scripts (ex: mod_perl or mod_php in Apache) or proxying to other servers that handle dynamic content.
Still not much of an app unless the JS does AJAX, but then it's talking to a server that's running some dynamic process, maybe with DB access etc. - right?
thttpd is probably the most well known "simple" webserver, and is very, very fast[1].
In fact, I think the op needs to explicitly mention thttpd on the page for shortfin and provide a performance comparison, since that's all anyone really cares about ... "is it faster than thttpd ?"
The notes very clearly state, "The benchmark test system is a 297MHz Ultra Sparc with 256MB RAM / 512MB swap running Solaris 2.6".
What you're referring to is merely an observation about the actual capabilities of web servers in general at the time: "Most of these servers have enough oomph to keep a T1 saturated, running on a lowly 100MHz Pentium or maybe even a 486."
Something I keep thinking I'll build is a fast, high connection count, limited HTTP server, something that is essentially a wrapper around a program that works like 'regular' and emits HTML. It is a corner case in a custom corner, but the target it something which is essentially a 'transponder.'
In the 'Internet of Things' I want to build a wrapper/environment such that my program can be
Yes, "static CGI" if such a thing was a thing :-). I've got a thttpd tree that I do this sort of hack in (these are great for monitoring large numbers of servers for example).
I don't understand (and am interested). What's "static" about it? With Apache and mod_rewrite and CGI, I could make the functionality you want (which is, of course, completely not low footprint, but I don't understand how what you want is different from CGI).
Static in the sense that it always executes the one function (doesn't load it from disk, its part of the executable) and so there is never any risk that some other path might get you 'out' of the docs directory and into the cgi_bin directory. So in this case static is code for 'compiled in' versus 'dynamically loaded.'
Like a statically linked HTML document? I know you're talking C code, but I could imagine some kind of HTML to C "compiler" that could you then statically link into an executable. Then you could have a full website that ran as a single static executable, including CGI-esque calls.
That's cool! I wonder if this would be (much) faster than just letting nginx or this embedded httpd read the files directly and serve them. I would hope that the OS or server would cache things enough to make it unnecessary, but it would be interesting to find out.
HTML isn't that big, so if you used linked resources, it would be pretty simple to make a C program and compile it to a binary for any purely "static" site.
That's how the Kore web server works ( https://kore.io/ ); it makes you generate a static.h file at compile time that contains all of you your static content.
But like you say, I have to wonder if it actually makes a difference performance-wise. The kernel will cache the static file data anyways. The only improvement I can see is maybe to avoid having to look up file metadata (length and mtime)? Or to avoid a syscall?
The apache module to do this is like a screenful of code. Really, mod_perl or mod_python or mod_wsgi can do this with dynamic languages, the code is only loaded once (per child in prefork apache). Your apache config can avoid reading the filesystem all together.
As long as you have a web server that can load modules and be invoked on specific URL requests, you can embed this.
I'm more familiar with Windows, but Linux should be similar. If your application is executed often, then it's pages should be in memory already, and not necessarily need to be read from disk. In terms of execution, if it's a compiled binary, it should be fast enough, if you need more, what you want is a lightweight application server anyway.
The problem with regular old CGI is that for every request the web server gets it has to run some binary file and setup a new thread for that every single time. Also, that's ignoring the filesystem overhead which isn't negligible even if it's already cached. You're basically turning your web server into a glorified shell script and even though all of your CGI scripts are compiled it's still horribly inefficient. FastCGI on the other hand is basically like giving the web server a socket to a persistent process that runs the associated code in an existing thread instead of having to spawn a new one on every request.
Basically FastCGI is like having another server just for your dynamic content while the web server sits in front of it and takes care of hosting the static content and deferring to the FastCGI server when appropriate.
I'm specifically saying that the File System overhead is negligible for files/executables that are frequently accessed. Windows uses RAM to cache these files... Setting up a Process to execute said file vs a service that stays running is a different story.
I've used Snorkel[1] for a few C/C++ projects in the past, you may want to check it out; it sounds like it could be what you're looking for (or quite close). Link it with your code, pass it a few config parameters and let it serve static files -- or you can overload GET/POST functions to provide dynamic functionality.
I had an idea like this when I thought about creating a small server on an Arduino that I can use to control/get status updates for a pellet stove for my cabin (TBA).
I host more websites on the same server and as kajecounterhack said I'm using nginx as a reverse proxy. The server is fast and have no problems at all with the HM crowd ;)
On my tests it's a few 100 requests faster per second than nginx but the average should be around the same. The kernel is the bottleneck in this case I believe.
There isn't any "selling point" ;) Anyone could use it as they like. The main usage for me has been in other projects where I needed a fast and lightweight http server. Such as tracking servers etc. It's pretty "experimental" but I think I will use the "original" server to serve a web app that I will be launching soon. The problem for me has been the lack of a reverse proxy.
But why do you need reverse proxy at all if your server is as good as Nginx? And if the speed is comparable and Nginx has more features why did you write your server then?
If I understand, it's "lighter" than Nginx. Why not writing how much lighter and in which aspects? Of course I can compare the code bases, but you know exactly as you wrote yours, so it would be really nice if you'd write what you know. Thanks.
I was going to say that there are *~ files in the repository, they should tidy a little bit because things like that don't look good.
Then I thought may be I was being a little bit to picky and unfair, so I checked the code of some random files and... well, it looks like it was part of a learning experience but it definitely needs work before being considered a viable option.
That's completely orthogonal to what we're talking about here. If you're already not doing it, you're not doing it any less by being more efficient. And it doesn't really matter if you're running it in a throwaway VM like I do.
... and especially not if you're suggesting run it as root. You should never run anything as root unless you know what it is. Having it over HTTP vs HTTPS makes things really bad though.
Time taken for tests: 12.125 seconds
Requests per second: 8.25 [#/sec] (mean)
Time per request: 12124.501 [ms] (mean)
Time per request: 121.245 [ms] (mean, across all concurrent requests)
Transfer rate: 38.46 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 210 1114 1517.2 360 4041
Processing: 5017 8407 2034.0 8715 11762
Waiting: 169 1084 3054.8 188 11408
Total: 9057 9521 1068.7 9058 12123
Time taken for tests: 5.790 seconds
Requests per second: 17.27 [#/sec] (mean)
Time per request: 5789.949 [ms] (mean)
Time per request: 57.899 [ms] (mean, across all concurrent requests)
Transfer rate: 82.97 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 171 249 35.7 263 290
Processing: 172 1798 2345.5 212 5524
Waiting: 170 1055 1825.1 212 4949
Total: 344 2047 2368.2 469 5789
1. Shortfin is not a replacement for nginx. I use it myself to create other projects, e.g. fast API servers with one purpose only and a REST server that uses redis as storage.
2. See #1.
3. I don't try to sell anything, it's just an open-source project that I think maybe someone could benefit from. I've learned a lot while coding it.
While I appreciate the time and effort that went into this, I'm more concerned with security than just speed alone. It's still relatively easy to throw hardware at the load problem (up to a point, naturally), but safety shouldn't take a back seat to performance.
The wget shell script installation does make me nervous and I'm glad the source is available separately. Blind installation of scripts was never "a thing" with me.
Of course, I'm still glad people are writing proper web servers (as opposed to simple 2-10 line ones). That creates an opportunity to explore the field with fresh ideas.
I can't find anything related to the response headers the server can send to the client, and the OP site just serves responses with content-length, content-type and server. Anyone?
This code is far off from being production ready. Skimming through the code quickly, there's barely any error handling. HTTP parsing is not compliant either.
I will perform some benchmarks and compare it to nginx (and maybe apache) in a while.
By the time (2010, 2011) I was working at a company that needed a tracking server for tracking clicks and views in email. So I started to mod lighttpd and thought that I could make it faster. So in true "challenge accepted" spirit I started to write my own HTTP server. And since then I've made it better and better. It's not meant to be an nginx killer. It will always be very basic, fast and lightweight. It's great for projects where you need a web server included.
As I wrote in another comment I have used the code base myself in various projects such as:
I've never used it as a static web server in production yet. Mostly because of the lack of a reverse proxy and other great features that nginx has and shortfin not.
The author claims it's 'high-performance and open-source' - sounds good to me. It's not as if we have a hundred web servers all competing as we do static blogging systems, the more web servers the better in my opinion, just as with web browsers.
The zip file is 312KB as linked from here, so it's pretty compact, possibly suitable for embedding in apps, etc:
A lot of you wanted benchmarks so here's one comparing Shortfin with Nginx. The tests was performed with a 56.1 kB PNG image with keep-alive turned off on my laptop. The best result out of 3 tests is shown below.
tl;dr:
Shortfin: 18 914 req/sec
Nginx: 15 603 req/sec
SHORTFIN
sudo ab -n 100 -c 100 http://127.0.0.1:40/timothy-johansson.png
Server Software: shortfin/0.9.5
Server Hostname: 127.0.0.1
Server Port: 40
Document Path: /timothy-johansson.png
Document Length: 56089 bytes
Concurrency Level: 100
Time taken for tests: 0.005 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 5618000 bytes
HTML transferred: 5608900 bytes
Requests per second: 18914.32 [#/sec] (mean)
Time per request: 5.287 [ms] (mean)
Time per request: 0.053 [ms] (mean, across all concurrent requests)
Transfer rate: 1037701.56 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 1 2 0.0 1 2
Processing: 2 2 0.1 2 2
Waiting: 1 1 0.2 1 2
Total: 4 4 0.1 4 4
ERROR: The median and mean for the initial connection time are more than twice the standard
deviation apart. These results are NOT reliable.
Percentage of the requests served within a certain time (ms)
50% 4
66% 4
75% 4
80% 4
90% 4
95% 4
98% 4
99% 4
100% 4 (longest request)
NGINX
sudo ab -n 100 -c 100 http://127.0.0.1:41/timothy-johansson.png
Server Software: nginx/1.2.6
Server Hostname: 127.0.0.1
Server Port: 41
Document Path: /timothy-johansson.png
Document Length: 56089 bytes
Concurrency Level: 100
Time taken for tests: 0.006 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 5631000 bytes
HTML transferred: 5608900 bytes
Requests per second: 15603.06 [#/sec] (mean)
Time per request: 6.409 [ms] (mean)
Time per request: 0.064 [ms] (mean, across all concurrent requests)
Transfer rate: 858015.83 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 1 2 0.7 2 3
Processing: 1 2 0.5 1 3
Waiting: 0 1 0.7 1 3
Total: 2 4 0.9 4 5
WARNING: The median and mean for the processing time are not within a normal deviation
These results are probably not that reliable.
Percentage of the requests served within a certain time (ms)
50% 4
66% 4
75% 4
80% 4
90% 5
95% 5
98% 5
99% 5
100% 5 (longest request)
I don't know how valid a test of five milliseconds is to one of six milliseconds (much less one requesting a single known resource with no keep-alive). Though practically, even if those results held in a realistic test, is it still a viable alternative to nginx? You eliminate an enormous amount of flexibility (I admit -- I ♥ nginx) and proven trust for a margin-of-error theoretical speed advantage?
I understand the desire to get software out there, but webserver is a ridiculously hard nut to crack. nginx broke in through a new architectural paradigm.
Nope, I'm not trying to replace nginx. I love nginx too ;) The "challenge" was to build a very lightweight server that was faster than nginx - which I did. Then I've used it as a base for a various of projects (see my other comments).
Nice, in what language? I've really learned a lot from creating shortin! Shortfin is written in C so it's a nice project to spend some time on when I get tired of writing javascript and html ;)
