>It was and it wasn't. It was indeed a bunch of patches on top of the
(also younger) NCSA http server. That's where "a patchy http server"
came from. But that was '90s already, some ten years after GNU
started.
This was an interesting security patch that marked the first time in my memory that updating Apache led to an immediate regression. A few hours after taking this upgrade many systems experienced such strange timeout errors. Connections were low and couldn't pinpoint the misleading behavior that looked like a slowloris attack, with no connections.
Half a day later with no resolution in research a new patch [1] was available and problem resolved.
It might be because they had patch cycle commitments. Ideally you want this stuff to be tucked into the regular release cycle. It costs companies a shit load of money to release and out of band update, esp when security related
I’m replying to you because the parent is flagged.
I’m not a security guy, but I am a serious generalist and I’ll try my hand at anything.
Except security.
As finely as I try to hone my craft, as much discipline as I bring to it, and as many great results as I’ve produced: computer security is a completely different ballgame.
The guys and gals who are as sophisticated, and diligent, and dedicated, and ultimately humble enough to be serious security pros are a breed apart, and my hat is off to them.
What I find interesting is that when browsing lists like these, more and more often I see names that are not from Europe or a rich English-speaking country. Nice to see previously-poorer countries getting online, the internet a bit more diverse, and the world a bit more connected. Though of course this is only empirical, perhaps it has always been this way and I just never noticed.
It has always been this way. It's just that in the latest years the internet became more localized due to everyone wanting to take control of it. It is just sad.
Oh boy, once Apache goofed with incomplete error reporting on simply why .htaccess failed, most of who i know has moved on to other kinds of HTTP servers.
You can check the status of vulnerabilities like this with a quick Google of "Debian" plus the CVE. Example for this one is https://security-tracker.debian.org/tracker/CVE-2021-33193 . When a patch is released, the package may not say 2.4.49, but that page will have the exact version number with the fix.
"Heap buffer overflow in Autofill in Google Chrome on Android prior to 91.0.4472.77 allowed a remote attacker to perform out of bounds memory access via a crafted HTML page."
While bugs like that aren't "confirmed" to affect desktop versions of software, the underlying bug is still in the Desktop codebase and needs to be treated as a security issue.
Could be the case. Could also be that the bugs were never in Debian if they were introduced recently as Debian stable is relatively conservative about package updates other than for security purposes.
Lots of companies that have been around a while and haven't replaced legacy applications still run those behind the web server they were designed to run behind. Moving to another server seems simple until you actually try to do that with a handful of 15-year-old sites with proxy rules, rewrites, custom headers, and such in one server's central config file and a bunch of per-directory access files. Technical debt is a real thing.
Shared hosting is still huge for small sites, and the major control panels for that (cPanel, Plesk, DirectAdmin, vDeck) still support Apache as the primary web server.
This. There's nothing wrong with using mature technology that is still being developed. LAMP is still a perfectly fine choice if it's what you want to use.
So putting these config in .htaccess which live in project folder is a technical debt, but putting all these logics in nginx master config file is not?
When the reason a team can't unbind from one technology in favor of county and that reason is old decisions that we won't make today, that's a form of debt. If you're choosing to stick white Apache for a good reason it's not debt.
I don't think we disagree here. If old systems stay with a legacy technology and do other work instead of changing that foundation, then staying with the legacy technology can become technical debt - not the decision itself, back in the day.
But Apache is not a legacy system and still a completely valid choice, next to alternatives like nginx and caddy that have their own advantages and disadvantages. There is no reason to swap Apache out by default, apart from maybe sometimes in specific situations.
Not by default, but since nginx is pretty flexible you can implement support for it, and it seems like people have.
There are some downsides with .htaccess files though; since it can change you need to at least check if it was modified since the last request, and since it's recursive (i.e. /foo/.htaccess applies to /foo/bar/file.txt) you may need to check several locations. Basically, it's a lot of stat() calls for every request.
The upside is that it's convenient; i.e. "tar xf somephpapp.tar.gz" and it "just works" without mucking about with configuration to disallow certain files, but this convenience comes with a fairly hefty performance impact, and it's only certain types of PHP apps that really take advantage of this.
I don't agree with the downsides - what you are describing is simply a bad implementation of this mechanism. At the very least, the files could only be checked every 5 seconds, or inotify or similar mechanism could be used.
The main upside is that it is possible for sysadmin to allow 3rd party users to change parts of http server config via ftp (!). And last I checked, php was dominating web landscape - I would argue that the convenience of deploying plays a large part in this.
Plugins... If it is not part of original app, can I trust it will stay available? I might as well use Apache instead.
You seem to not be a fan of it, do you have a feature-complete replacement?
For example, Nginx's standpoint on htaccess is basically "find yourself another web server" (https://www.nginx.com/resources/wiki/start/topics/examples/l...). I don't fancy giving users access to /etc/nginx and the power to reload the config just to add aliases on their own subdomains. Much better if they can add password protection to a directory by themselves. It also breaks that each site can live in its own project directory: without htaccess, if I copy over a project I would also have to modify my global nginx config to set per-directory (or at least per vhost) aliases, error pages, etc. Right now, I can say that deploying a project is "download the directory into the desired location and configure your database credentials and a pepper in the config.php". No global configuration changes are ever needed, the global config only does global stuff: tls ciphers, fallback error pages, compression, log files, and more such things.
And that's ignoring the fact that, even if there was some objectively better web server, I'd have the task of rewriting a decade of htaccess files into their fancy new format while Apache2 is still alive and properly supported.
About a decade ago nginx stormed onto the scene with event based multi-processing, and even though mpm-event became a thing not too long after that, Apache retained mpm-prefork and then mpm-worker as defaults for a very long time, and lost a lot of market share.
A lot of people still retain the notion that nginx is "just faster" or "just better" which is not necessarily the case. Apache with mpm-event is just fine for most applications.
There are other reasons to use nginx, and there are other reasons to use Apache. Both are fine, and I hear https://caddyserver.com/ is coming in hot!
redbean is a pure fork() web server and according to ab it is much faster than nginx. I feel like we've come full circle with history. Apache started off with fork() since it offers the strongest security versus other i/o models. The problem is that code bloat makes fork() go slower so as Apache ballooned over the years there was this race to the bottom in terms of i/o models, caused by tragedy of the commons. https://youtu.be/1ZTRb-2DZGs?t=761
To be fair, the event worker in Apache and nginx are roughly the same age, 2004.
Smaller web servers have been event based since they first showed up, it is the natural way to build them. There was no epoll() available, but a select() loop is pretty much the same thing. The super useful thttpd had been around a long time at that time.
What caused people to start using non-forking web servers for regular public web applications was probably that PHP finally got a stable FastCGI mode going about ten years ago. Because, honestly, most of the web is PHP and it was even more dominant back them.
And when people posted their web server configs in forums at that time, nginx configs were much more readable and concise. The easier config format is what caused its popularity to soar, not that it used a particular syscall half a decade earlier.
I agree wholeheartedly that php-fpm rapidly speed up nginx adoption.
Before nginx, lighttpd was a thing. It also comes with `spawn-fcgi` to handle fastcgi, but it's a little complex to set up. Back then people were also using `spawn-fcgi` with nginx, too. I remembered reading some tutorial and decided that I will continue to use Apache with mod_php due to complexity. When php-fpm come, I immediately switched.
> At the expense of the memory footprint because developed with Go. Hi Caddy creator!
Makes sense to me that something built in a language with more memory safety than C would use more memory, but is it really significant in practice? I use go every day and memory usage has not been a significant issue for many applications.
I don't use Caddy, but I've written plenty of Go apps (which usually implement their own HTTP server with net/http) with plenty of users, and memory usage has never been much of a concern.
There are other performance concerns of course, e.g. GC latencies, and at times the GC may bite you. See e.g. the "taming Go memory usage" story from a few days ago[1], but this usually isn't a huge concern for many use cases.
I'm guessing you mean the about section of your profile. I'm just saying that it's nice to make that more visible when you have that sort of involvement in the topic and it is not obvious.
It's more the other way around, when saying things about your own product it's nice to disclose your involvement since that definitely colors your view. Especially in cases like this where it's hyping the product.
I never cared much for certbot because "sudo certbot" and it works by rewriting config files. I don't really fancy running it as root (which can probably be avoided with some muckery, but still) or a program automatically rewriting my configs.
Overall, the entire thing seems very complex automagic black box last time I tried it (already years ago). Sometimes trying to make something easier can make it harder.
Nothing really. Historically Apache HTTPD was slower than the newer web servers like NGinx when Apache HTTPD 2.2 was the mainline version. This is no longer the case with Apache 2.4 using the latest APR libraries. 2.4 has been out for a very long time, but some Linux distros were slow to uptake it.
Probably not the answer you want to hear... But I push that stuff to AWS S3 website and put a CDN in front for SSL... that's my solution for "easy secure static site" when I don't want to invest more than an hour or so a year on the infrastructure... But obviously that's not for everyone.
Edit: specific to your request, I would probably set up a container pointing to the latest apache build, and write a script to pull the latest image then roll over the container... Don't need to wait for your OS of choice to have the fixed versions... Your never more than a day behind
I've been running darkhttpd for years. I was recently complemented on how fast my site loads; although part of that is probably the lack of JS tracking bloat.
Note: several folks have pointed out that I was totally wrong about Caddy. Sorry! I should look at it again before I spout off any more opinions about it.
Unlike nginx or Caddy^H^H^H^H^H, the Apache design and community encouraged writing modules (roughly akin to "middleware" in the modern server-side web stack) that hooked directly into the web server, rather than just using it as a static host + L7 proxy.
Furthermore, these modules could themselves expose language-specific APIs; ergo `mod_perl` and its ilk, which provide a bunch of useful building blocks for a full-stack webapp but are entirely specific to Apache's module API (vs. something more standard like FCGI, WSGI, etc.)
I've worked at some shops with heavy investments in Apache-based application servers and the cost of moving away from HTTPD modules and towards a more standalone, Apache-free implementation was generally quite high. Usually you need some other forcing function (rewrite in a new language, microservice decomposition, etc.) to justify the investment.
> Unlike nginx or Caddy, the Apache design and community encouraged writing modules (roughly akin to "middleware" in the modern server-side web stack) that hooked directly into the web server, rather than just using it as a static host + L7 proxy.
Huh? I must be misunderstanding, because everything in Caddy is a module, even its HTTP server, and all its middleware handlers are equally modules that you can plug in as much as any other module you could write and publish.
Some Caddy modules also expose dynamic scripting / language interpretation capabilities as well. Caddy is used for well more than just static files and HTTP reverse proxying.
I use it. We probably have 50+ websites hosted on Apache HTTPD.
It's very stable, supports that one special thing many sites need (special headers, stripping file extensions, caching etc) and has excellent documentation.
It's not fashionable, but it will still be unfashionable in another 10 years, and we won't have needed to update the configurations in that time.
It appears so [1]. It's surprising how far behind most non-tech industries are when it comes to legacy software (not sure I'd call Apache "legacy", but still). I'd also be interested to see the stats for IT/developer positions in tech vs. non-tech fields. My intuition suggests that there are a lot more Apache sysadmins than you'd think.
Sometimes I think custom writing your own http server might not be a bad idea after all.
Lots of security holes in your custom write? Yes! But the hacker needs to be dedicated to exploiting your one server specifically to find it.
In exchange you are safe from of all those : vulnerabilities in the wild => script kiddies => mass exploitation => your are now hacked type of situations.
Relying on obfuscation as part of your security posture is a fantastic idea IMO. Just be prepared for what happens if a human figures out your specific puzzle box.
We like exposing our private B2B web services in such a way that an attacker would be led to think the web server is totally fucked up or otherwise mis-configured if they don't understand the proprietary protocol. It's not that they could never understand it, but it would take a significant, targeted effort to even begin attacking our system in a direct fashion. Even if they figure out the obfuscation, they are still going to have to fight through a moat of PBKDF2 & hardware tokens.
To me, obfuscation is all about minimizing the impact of automated, mass, 0-day assaults. Some person will always be able to walk back something another person built. You can mitigate legions of scripts by cleverly dropping TCP connections when a resource isn't requested in just the right way.
Another fun option is to build a honey pot around your web service which has the ability to ferret naughty TCP pipes off to the hacker matrix where you then simulate all manner of PII-rich data sources for them to worry about (instead of your actual business).
Security through obscurity is great for preventing low-effort attacks at low-scale, yes.
The market for security skills tends to be more interested in protecting targets that can draw high-effort attacks, or low-effort attacks at scale, so there's good reason for the "security through obscurity = bad" meme. You won't get in trouble by incorrectly assuming "security through obscurity = bad," but you can definitely get in trouble by incorrectly generalizing "security through obscurity = good enough." It makes sense to err in the direction of least damage.
I don't see how this could be true for an HTTP server. If we were discussing some custom protocol, sure. But with HTTP, you are far more likely to repeat well-known mistakes and vulnerabilities that have long-since been eliminated from the popular HTTP servers.
This is a great idea where people can submit their own web servers that have some basic functionalities and are on the side would be users who gets points/karma for exploiting them under some threshold like 3 hours. Maybe something like DevCon's Capture-the-Flag but online.
> In exchange you are safe from of all those : vulnerabilities in the wild => script kiddies => mass exploitation => your are hacked type of situations.
Not sure about that... You might commit some of the same mistakes that they did
Presumably you'll have less features. I wrote a web server (in the 90s, so forgivable?), and I only implemented GET and query string for the API data, and nothing fancy, no cache headers, no redirects, no multipart, no gzip, no content-encoding negotiation, etc. Some of the APIs it hosted were ultimately hacked (thru encryption of the encrypted query string via plain DES, and the key was determined), but, so far as I know, it was not itself ever compromised.
Not impossible that your web server was 100% secure, but I doubt it because I think that I have never seen a server that didn't eventually need security patches.
I left the company and then about ten years later it was turned off. It was in C but was pretty mature framework, not C style strings. And the code was just “validate assumption for parsing, else return error” then “parse the next few bytes else return error”. Very simple small number of parameters, explicit length limits everywhere. And it was a public AOL end point so widely and continually attacked by people. I was there for maybe 7 years after deploying it and nothing (known) happened in that time.
https://www.mail-archive.com/fedora-list@redhat.com/msg06924...
>On Jul 16, 2008, Les Mikesell wrote:
>> Alexandre Oliva wrote:
>> Apache wasn't the original name.
>It was and it wasn't. It was indeed a bunch of patches on top of the (also younger) NCSA http server. That's where "a patchy http server" came from. But that was '90s already, some ten years after GNU started.