Not a fan of this guide: No explanations, no context and worse: missing critical stuff for your typical hosted $webapp server usecase:
run services as normal user and confine them using apparmor (or selinux if you are masochichst)
the NSA will probably hack you anyway, that drive by script-kiddie that kills your company JIRA instance by DDoSing some gameserver while exploiting some old Wordpress plugin can be stopped by apparmor!
What's missing is some kind of kernel live patching - if someone is not a script-kiddie on your machine that Wordpress shell is a nice tool to elevate to root - your kernel is rotting and exploits are plenty... so consider some kind of live-patching if your are paranoid.
Also no discussion about updates, the need to restart services for reloading updated .so files (like openssl)...
You can obey that guide and be busy with life and some skiddy root exploits your box anyway...
so /rant - there must be some better guides? found those cis benchmarks (https://www.cisecurity.org) a mixed bag - is there anything better out there?
Yeah, getting strong 90s vibes from this doc, perhaps because most hardening manuals like this are just collections of advice from other (bad and old) hardening manuals.
Lists like this exist to give administrators and their managers (1) a false sense of security when implementing it and (2) a subconscious excuse not to actually have to think about what you actually have to defend against. The list of topics is almost entirely useless if you are running any sort of containers on top of your Linux install, just as one example.
I'm sorry to be sounding so harsh, but I sincerely detest the security-by-checklist approach. It barely protects against any real-world threats, needlessly complicates matters, and gives credence to the perception that "security slow things down" through over-engineered solutions to non-existing problems. Moreover, it's a subconscious excuse to actually stop thinking about one's actual risks. Reasoning about actual threat models may not be easy but it is what one should be doing.
There's nothing "practical" about this at all, in my view.
> I'm sorry to be sounding so harsh, but I sincerely detest the security-by-checklist approach.
Checklists are essential (or becoming so) in all professions that require a level of safety or assurance in their operations. They don't replace careful though and action, but they do supplement it be making sure there's a minimum level of items that were checked and not omitted because someone thought they didn't apply when then did, or someone just plain forgot. A checklist is a good way to start some rational thinking about what's required for your specific case (especially if it's so restrictive that you have to (and are expected to) selectively alter portions of the checklist just so your current use case functions.
Checklists are well known and save lives in some professions (such as aviation), and are being applied to others even when there's push-back (Surgeons and emergency room operations[1]) because the benefits are just so large.
I think it's fair to dislike how people and organizations adopt a checklist in lieu of careful thought about security, and to dislike poorly defined and reasoned checklists themselves, but I for one would be much happier, and feel much safer, if security checklists were much more common overall.
Checklists are fine if you have a fixed problem space. Just to go by your example - aviation safety is a mostly fixed problem space you can make mitigations in. Any item on the checklist would be there to mitigate safety issues almost any aircraft would encounter (and I'd imagine there are specific checklists for e.g. passenger vs. cargo aviation).
"Linux security" is not a fixed problem space. Not at all. And that's precisely my problem with this - this checklist pretends that it's a fixed problem space, and therefore grossly misrepresents the problem.
My point is that most problem spaces can be split into fixed and non fixed portions. Aviation safety has plenty of problems that require a real person to make a call and react intelligently, which is why we still have pilots. Checklists are used to cut down on the entirely avoidable problems that might be missed some percentage of the time otherwise.
There are plenty of things in Linux security that are static solutions that can be employed almost all the time, such as not allowing direct access to root accounts, always running a local firewall, making sure remote services aren't run as root without dropping privileges, etc.
Encrypted boot really bit me in the ass one time, that's one afternoon spent in the dust underneath the stairs in my student housing that I won't get back.
Would've been so much easier to repair that machine if the boot partition wasn't so weird.
CIS and the various NSA STIGs are about the best actual prescriptive guides.
But threats occcur on a continuum - at a point (which is focused organised crime/nation state actors) you will lose...but there is still a lot of low hanging fruit you can win on.
> missing critical stuff for your typical hosted $webapp server usecase:
It looks like work in progress. Selinux, updates, deployment and many other topics are on the table of contents, but not present in the document. (Yet?)
It's got some context which is better than almost any other guide. Ideally they'd include some threat models applicable to each point, but that could come in the future (also you can send prs)
I have recently completed a project applying CIS to a clients estate. Whilst CIS is great, particularly if you are a large Org having to deal with auditors and the like, it can only be regarded as the bare minimum.
Nah it looks like author just made table of contents and counts that there will be bunch of suckers that will collaborate on his repo by submitting pull requests.
Really, fire protection and locking racks with configuration of Nginx and Apache in same guide? If I am going to harden Apache, I am going to find Apache only guide.
> Maintaining a stable temperature and humidity within tight tolerances is critical to IT system reliability.
Data center HVAC engineer here. The part about tight tolerances is not entirely accurate. IT equipment can handle a reasonably wide range of temperatures. The ASHRAE guidelines [1] have a range of 18 to 27 deg C for all classes of equipment.
What is an issue is rapid changes between temperatures. For a data center with any kind of devices (eg tape storage), ASHRAE limits this rate of change to 5degC within a 15 minute period and obviously you also want to avoid any situation where condensation could occur or static build up (high or low relative humidity).
Data centers do try to keep the temp stable towards the top of the thermal envelope for economic reasons but there's always some fluctuation due to server load.
Wow, your gear is treated so nicely. The stuff I work on has to deal with anywhere between 0C and 80C while having the crap shaken out of it and occasionally being hit with a high pressure hose.
OK, high pressure hose in a data centre piques my interest 8)
I stuck a camera on a stick at the southern boundary of my property, about three feet above the stream. The stream rose four feet for a while. I moved the camera.
Now, 0 to 80C. That is just plain odd. Here (southern UK) the temp range is something like -10 to +35C. Nowhere has 0-80C apart from a weird oven.
This is minesite automation stuff, so it usually lives in a stainless steel cabinet, ambient air temperature can be over 50C, sometimes sharing the space with high power switchgear. It's not unheard of for the inside of the cabinets to be over 70C if it's mounted in direct sunlight, so most gear is rated to 80C. Then at other times in the year it'll get down to freezing.
Aye. The old rules are ... old rules. Nowadays we have rather a lot of temperature sensors - inlet, outlet and each subsystem has multiple sensors - they will also report max and min and current.
Rapid changes in temperature and interactions with certain ranges of humidity can be interesting.
> For secure rooms make sure that the walls go beyond the false ceiling, and below the raised floor, large vents should also be covered with bars if possible.
Secure room means there's no way for anyone (big or small) to sneak into the room undetected. Hence, having false ceiling/floor and ducts need to be secured - usually the fire/aircon zones are also separated and isolated.
I have seen colo providers do this when they are hosting banks/financial institutions.
What, are you trying to say security theatre doesn't make everything safe???
Surely if we grope everybody's balls/boobs on their way in, and have a secret list of "people with names that are vaguely similar to middle eastern sounding names" that we won't let in, that'd make up for the walls that you can just crawl under of climb over? Right?
It does, but there’s also what I call stupid security that doesn’t really add any measurable improvement, but does decrease productivity. Users out smart these systems all the time (e.g. forced password changes where you can’t use the last 10 passwords, users just change their password 11 times so they can continue to use a password that’s been configured on their devices. It’s stupid then to force changes like that when there are much better ways like mfa)
Smart security allows users to do what they need to do efficiently and safely.
Yeah I can't stand the forced password changes where you can't use the last X passwords, or passwords that expire every 30 days. A lot of times it's security compliance entities that push this down to companies, for instance PCI, etc all require those. I think even the new NIST standards address these practices, but the compliance entities are slow and far from pragmatic.
Yep. I can tell you that currently, leading practice involves such measures. I don't agree, but CIS, which is essentially the current gold standard, says so. People who get paid to do this often don't bother griping about it, because they are being paid to harden to a standard and that standard is what it is.
This unfortunately leaves a disconnect between the people who harden (who might actually hear about issues), and the people who write. Even if the writers do hear, it won't be implemented until the next revision.
Yeah PCI or FedRAMP have this 10 char password requirement, which of course no one can remember a 10 char password. So companies just make the password a pattern with some variations, effectively reducing the complexity to a tenth of a random 8 char password and the people who know the pattern leave the company so it’s effectively public. So much for math.
Right. The first rule of password security: if you have a large enough user base, the odds of a user writing down a password increase, and as passwords become sufficiently difficult to remember, the odds approach 100% at some point that _some_ people are writing down passwords. No amount of defense in depth can protect the "I have a Post-It note under my keyboard" problem, if people can get into your building.
We've handled this by mandating password manager use and pushing length requirements to absurd levels to where it truly is easier to just use the manager, which has two factor.
Absolutely, but I prefer not to leave 22/tcp open to the world. If I do leave it open it is only from a restricted IP set, otherwise it is behind a VPN, probably OpenVPN.
Sure, especially when you VPN into a sacrificial subnet and need MFA to continue elsewhere into locked down application domains. OTOH I would leave ssh listening on a non-descript high port with MFA (key and OTP) enabled. No use worrying too much about that.
Usability is a component of security because of human factors; if your “secure” process or system is not convenient for use, people will in practice find ways to work around it instead of using it as intended, which will defeat security.
It shifts convenience. It's less convenient for me to have to unlock my door when I get home, but it's more convenient to not carry all my valuables with me during the day. And I really like that tradeoff.
Good security measures are like this. Add sandboxes so you can let users do what they want. Add authentication so people know who they're talking to. Support security keys so people don't have to worry ad much about being phished. And so forth.
That's why for any office that has internal wifi I encourage also having a "guest" wifi network (small offices, not using anything with enterprise-level management). People are going to want to connect devices to wifi instead of LTE even if they never approach their billing limits, and if there's an available option that they're allowed to use it cuts down on attempts to use the one they're restricted from (and complaints about "I tried to connect to the wifi but I can't get to the Internet!" "Was it the internal one or the guest? Internal is locked down. Connect to the guest.")
On that note, I would like to recommend the most complete book I've encountered on the topic:
"UNIX and Linux System Administration Handbook" by Nemeth et.al.
Just because I wish someone had recommended it to me earlier.
Great book, although when I was a young teenager in the late 80's yearly 90's I lived/owned on her sun servers (anchor.cs and the like) undetected for about 7 years. Anyways, I always had great respect for her and I was sorry to hear she was lost at sea a few years ago.
This looks good. I like that they have explanations / reasoning included, rather than just "do this". If all the planned topics get filled, that will be something to recommend.
Also, first time I hear about polyinstantiated directories, so TIL.
This was one of the neatest features of the computer formerly known as AS/400: each job (batch or interactive) had its own temporary library (QTEMP). You could create objects there knowing only you could see them, and the library was destroyed the moment the job ended. Definitely will try polyinstantiated directories.
What do you mean former? There is an AS/400 in our data centre that we look after for a customer for their off site DR box, with constant DB replication. It is about four years old, the platform is rather older. Bloody noisy beast especially on boot up (IPL?) sounds like a helicopter winding up. It was switched on after racking and that was it - I'm told it is simply available.
A lot of this can be eliminated (Not the physical security or install) by doing the following:
1. Run centOS or Redhat
2. install openscap-workbench
3. Use the centOS stig and choose which profile (I recommend US govt configuration base)
4. Uncheck the firewall rules (they set it to deny all incoming; change to DMZ with basic rules)
5. Click remediate and apply.
6. OpenSCAP does the work for you to harden the system
You'll have to use other security appropriate tools for appropriate servers, but you'll know which service and its ramifications.I know that MySQL has a comprehensive security script to prepare. Other tools have similar built in functions.
Also note you can download Nessus and get a 7 day free trial as well. It's not perfect since the ticket price is $2400/yr . You could also use OpenSCAP for compliance, and metasploit as a substitute for application. There's also websuites like Burp and OWASP.
But regardless you pay or not for automated testing, you need something to automatically find bad things so you can fix them.
As with a lot of other enterprisey software made by RH, it should all work in theory, but it rarely works as well. I've never been able to get stuff like openscap to work reliably with satellite (which is one of those things that should probably not exist at all). I don't remember which openscap profile it was (I think C2S), but it would just hang on nfs servers with lots of data. So you can go and hack on these profiles, but it just gives the impression that none of this stuff is that widely used/production ready.
So here's something I've always wondered. If you use linux as a desktop/workstation in enterprise, how do you deploy it securely ? Other than chromeos, I don't see any other distro trying to tackle this. With all traditional distros, you'll likely need to give the user sudo on their machine to be productive (for eg., installing packages). However, all bets are off once you do that. You can restrict sudo to certain programs, but that also isn't very practical (for eg., sudo for yum is basically sudo for the whole system since you can install arbitrary packages). You can create a chromeos like system, but is there no interest in it ? There are some developments like fedora silverblue and nixos, but how is it that this has gone unsolved till now ?
You run your own Software repo(s) on site and you only allow software to appear in that repo that you trust. You use applications like puppet to automate installations from your repos. LDAP can store users,groups and attributes that can be used into determining which groups get what software baselines.
Not really all that different from what goes on in the Microsoft world with WSUS/Appstores/etc. Just the implementation is different and much more customizable.
So the agent still needs to hook into something with elevated privileges to install packages, just that instead of having granular control in the package manager, it has moved to server side with an RBAC/HBAC tie-in. I agree that it solves practical issues, but it would be nice to not have privileged access at all. Also, doesn't it create friction if you don't have a package in your repos ?
I mean 'your repos' typically are a superset of what's available in your distro's repos so it's fairly rare for someone to complain that software isn't available.
"you'll likely need to give the user sudo on their machine" - no we have sysadmins and application management systems to obviate that. If a user needs an app then they ring up the helldesk and get added to a group (after authorisation), which causes the app to appear.
>If a user needs an app then they ring up the helldesk
That seems very inefficient and friction prone. We have stuff like the nix package manager that can install stuff locally without sudo. It also has other nice features from what I've read. What I'm saying is that we need general purpose distros where you don't need sudo to use them.
Interesting you bring this up.
Regardless of the provenance of packages and facility for installation you seem to be hinting that there should be a
method to install specific service/feature sets to a system
securely and without systems access.
I designed such a system which runs with appropriate permissions to a task, is modular, and accomplishes the provisioning via 'job' files submitted to a service
endpoint. A user could describe the software they want installed. The SA performs due diligence on vetting software
and designs a standard install module then generates a skeleton jobfile and provides it to the requesting user to fill in necessary details and submit. Wash , rinse, repeat.
That's the way its done in general: User makes request, request is approved, action. I was noting that a Linux system does not get in the way of that work flow by being a Linux system. If the box belongs to a company then the company gets to decide what is on it.
If the box belongs to the end user then obviously root rights are in order. However, my wife never gets to use anything requiring root rights on her Arch Linux laptop and does not care and would not know what it is anyway. I keep her laptop functioning effectively by watching disc space and other metrics and regularly patching/updating it for her.
I'm sure this is possible in some places but in others it would be completely impractical. At a university, specifically in the STEM faculties, not giving users sudo/admin would create an endless workload for IT. We end up being nothing more than professional software installers.
Any reasonably secure windows systems have restricted user accounts that do not allow app installs. (I'm talking ofcourse in a corporate environment) The help desk / it call for any app install approvals is common in my experience
For installing/using Linux applications without root access, use Docker containers. For security, use rootless containers. Most major graphical apps have a Dockerfile now.
it cuts down the drive-by noise by 97%. which lets you not overlook anomalies in the remaining 3%.
we also had a bunch of scanners probing for common php scripts on the http server (we don't run php). simply returning a 404 header and disabling logging in nginx for \.php$ uris cut down a bunch of garbage (mostly from China). additionally, using ngx_http_limit_req_module reduced the effects of numerous DoS-type scanners.
so far, this + iptables for only 80/443/ssh has been a superior solution to instaling fail2ban that continually scans bloated log files and performs constant iptables banning/unbanning foo.
"Log noise" is a legitimate security concern all by itself. It is actually disappointing how rarely it is discussed, and how in this world of "AI everywhere" there's no great smart AI log analysers to reduce the signal:noise.
Moving the SSH port might "only" reduce noise, but that's a legitimate security goal in its own right.
> "Log noise" is a legitimate security concern all by itself. It is actually disappointing how rarely it is discussed, and how in this world of "AI everywhere" there's no great smart AI log analysers to reduce the signal:noise.
That's because after many people learn about not using "security through obscurity" they apply the rule without thinking as if obscurity never has any benefit for anything.
That's because people can't agree on what is or isn't valuable.
For instance, if I (or my software anyway) see a bunch of sshd login attempts from some IP, and then that IP decides to try imap ... yeah, that's getting insta-blocked.
And that's hard to do if you decide to just ignore the sshd attempts.
Could additional honey pot on 22 help? Run sshd on non-standard port and honey pot on 22. Then automatically ban all the hosts that would connect to it. There could be a concern for a mistake, so then ban for limited time at first or something.
Additionally having a non-standard port does help defend against 0day exploits before patches are available or before you are able to disable the port.
> Putting SSH on a nonstandard port is pretty silly.
what's the alternative while still keeping ssh logs that aren't filled with thousands of login attempts from bots per day? afaik there isnt any solution that is both as simple and as effective as changing the port.
Access behind VPN connection. Or deal with the logs in a better way. If SSH noise bothers you, you can either direct it to a different file, or setup a log aggregator where it's trivial to see what you want (or filter out what you don't by default)
fail2ban will cut down on log noise signficantly, while still allowing you to operate on the standard port. It also thwarts brute-force login attempts, if you're using password authentication. (Which you shouldn't, without good reason. Use public key authentication.)
You control what fail2ban does with said log data and what actually gets logged.
There are better ways than fail2ban of course but it's not the worst solution on the planet.
i think the main argument for leaving it default is [1] for any additional/new sysadmin that has to touch the system. but imo this goes both ways; i'd rather not have my house key hidden under my doormat, where every burglar can check for it in a few seconds. if i taped the key to a random tree branch (even in plain sight), it would be a much safer bet.
If you're going to insist on a security analogy involving keys and doors in the real world, a better one would be that the attacker has to search every single atom of the doormat to find the key. If you have PubKeyAuthentication enabled and PasswordAuthentication disabled, the world will end before the attacker finds your key (in expectation).
Changing SSH to a non-standard port is an inelegant solution and doesn't actually bolster security.
So standardize on a non-standard port, document it and when you do have ssh probes assume it's targeted rather than just another l33t child running du-jour crack(x).
Moving to non-standard means more sophisticated discovery
and possibly a more sophisticated attempt.
One reason is because setting it to a random high port allows non-root users to turn on the SSH server if the admin turned it off. Probably not super likely to happen, but it's possible.
That's really terrible recommendation. You should add firewall rules and allow SSH access only from trusted hosts, ideally over VPN. It's not about login attempts but about possible bug in SSH (and other network services), because it's all just software, written in C, so believe it or not, but it's there, it's not if but when.
If you're not prepared to expose a service to the world, you probably should run it at all. Ad hoc, non-standard configurations add substantial complexity and maintenance burdens. Complexity is the enemy of security, and having less time to manage more complex configurations is not a good recipe.
If you really need a service, then choose the best one and move on. The rule of thumb is that if you can reach a service, you should assume anybody else can, as well. This is especially true regarding SSH. I've seen plenty of servers p0wned via SSH, but never by breaking SSH. Instead the vector was always through an SSH user's computer infected by malware.
You want secure SSH? Disable password authentication and force everybody to use smartcard authentication like a Yubikey. I do rate limit SSH access using OpenBSD PF, but only because the authentication failures fill up and pollute the logs.
OpenSSH is solid piece of software. It hasn't had any (public) authentication bypasses that wasn't configuration related since it started to look like it looks today. Multiple skilled teams have looked at it thoroughly since. It has also been hardened with things like privsep and sandboxing.
This is not OpenSSL. Language alone does not make a product insecure. For practical purposes it is certainly not just a matter of when.
Contrast that with your VPN product, which probably falls over if someone looks at it funnily. Unless your product is called Wireguard, and we are ten years in the future, perhaps.
It's fallen out of favor these days, but if you're running on firewalls that can be configured for port knocking consider doing that. A non-sequential knock pattern that unlocks the remote IP for X amount of time can prevent a ton of grinding attacks.
Just use a jump server and put the rest of your servers on the private subnet for port 22 inbound. Expose your jump server to your vpn only and use 2fa on vpn.
Also cut down on the host keys (e.g. don't have an RSA/DSA/ECDSA host key if all of your clients support Ed25519 hostkeys) and key exchange algorithms (KExAlgorithms in sshd_config); almost all clients support curve25519-sha256 now, so get rid of all the DH fixed ones. Similarly ciphers; OpenSSH and PuTTY support ChaPoly, so you can get rid of aes-ctr, et al.
I'm not saying decrease your supported algorithms to the point where you break things you're using, but you'd be surprised at how much you can get away with not having and never notice it.
At the end of the day this means all those horrible libssh bots and worms can't even begin an SSH session with you, much less attempt any sort of exploitation or bruteforcing. It still generates an entry in your logs (key exchange algorithm mismatch or the like), but decent syslog daemons can filter out such messages by regexps, and redirect them to another debugging log file that you hardly check unless you're having an issue connecting somewhere. Metalog is an example.
This would be detected by SSH, as the host key wouldn't match. (The impersonator wouldn't be able to use the same host key, as they would need the private key, and that requires root privs.)
Or someone can simply own your key holding device as you sip your beverage at the local hotspot or home wifi. Simpler to
go after the secret privileged client than the server.
Do you sync your ssh keys to all the machines you login from?
How do you ssh-copy-id without some allowance for passworded logins? Especially if you're doing it from such distant machines that you can't scp over a .ssh/config file.
So, I just ssh to whatever machine I need to work on, but occasionally need to rsync or scp files to/from that server.
Sure, I could have puppet push a ~/.ssh/config file everywhere, just seems more effort than it's worth. After all if a hacker wants to know if you are running ssh, it's not hard to scan all ports for ssh.
Why inconvenience yourself more than the attacker?
Right, but say you have a dozen machines you login from regularly and maybe a few 100 that you login from occasionally. You want to maintain a ~/.ssh/config on all of them?
Seems like a big inconvenience for minimal extra security. Not like scanning for all open ports that have an ssh listening is hard.
If you’re already typing in the full address for the server every time, since you don’t have an ssh config file, then is it really so much more work to additionally type the port?
Yes because I will forget to type -p 5223 at some point. If you set up a non standard port against my recommendation and I get banned for trying 22, you better be on call 24/7 for free to unban me.
I do similar but I personally recommend OpenVPN on 443/tcp rather than 1194/udp. It looks like SSL traffic because that is essentially what it is. Make sure you use your own locally signed CA and certs and enforce CA trust for a successful connection - to avoid any silly MitM tricks.
There is a reasonable argument against using tcp for the tunnel instead of udp due to what can happen to latency when you have a tcp inside tcp stream when one or the other or both are dealing with retransmits (bear in mind that tcp has some guarantees about delivery, whereas udp does not).
In theory you can get an exponential stand off instead of linear due to tcp (say RDP or http) in tcp (openvpn). In practice I find that it does not generally matter these days and I can generally stream BBC iPlayer when abroad with minimal fuss over OpenVPN.
It's actually very easy to detect OpenVPN on port 443 because it isn't really TLS traffic. Much better to use OpenConnect with ocserv, it opportunistically uses DTLS over UDP on 443, and if it can't do that it can use normal TLS. Much harder detect and gives you the performance boost of UDP when possible.
I have the same , but also have an IP ACL on the VPN ports / protocols (only on new connections). When I do need to Connect from random remote ip , I have a port knocking rule that adds ip for 5min to acl
I actually have the port knock action on a blank website w a unusual subdomain. So we need i it I pull up website and can then vpn in temporaryly
The only benefit of this is clean logs. We had auditd running and the bots blew up our SSH logs. Was so ugly that I killed SSH on all but a single entry point server, which is better security anyways. Then I finally just killed logging failed attempts. Since we use two-factor authentication with a key (you have) + password (you know) I was not very worried about this. Prob should just use something like fail2ban though eh?
Less attempts of something that has zero chance of succeeding (password login against a system that requires keys) doesn't really translate into less risk.
>Less attempts of something that has zero chance of succeeding
All attempts have a >zero chance of succeeding. Honestly...struggling to see how this concept is even remotely contentious or how someone could credibly argue against it.
If the number of brute force attempts drops substantially then the chances of success drop too. This is true for a high security setup and a low security setup. In both cases there is going to be a proportionate drop.
You can install Ubuntu with UEFI and secure boot enabled without any problems (personal experience). I believe that several other distros work as well, out of the box. All distros can do UEFI with minimal effort.
To be honest I consider things like BMC and the like and whatever goes on inside the BIOS and the like as far greater opaque problems than some of the more esoteric security "assurances" like measured boot.
While not part of the OP - there is some information on a common TPM use (LUKS unlocking) with the later TPM 2.0 options (TPM 1.x was not amazing with Linux).
Even if you have a single server to run a bunch of services (web, mail, ssh, file server etc), best way to raise the security bar quickly is to isolate these services from one another. Use one of the modern virtualmachine/container technologies with minimal footprint to do this.
Ideally, you would want physical separation between a node running firewall/loadbalancer and the servers behind them.
If you are running a large cluster of many servers, then you definitely want to plan and separate your architecture into different layers of separations.
Don't forget your own laptop from where you login to administer all this is the best chance an attacker has to breaching all this security. So, ensure there is good role separation for administrators and good security hygiene on all the individuals machines (use password managers, do only official work on these laptops, don't visit random websites, run vetted authorised packages only).
For highly sensitive stuff (like CA or any root of your chain of trust), setup a SCIF and use only terminals inside a SCIF to access/administer them.
Since this guide started with datacenter and hardware security, it is important to discuss the merits of having a secure enclave (hardware security module) on each server to hold the security keys.
Securing the chain of trust is a critical element of securing any system. If the root of the chain of trust is not securely bootstrapped and continuously verified to be still secure, then all bets are off.
The provenance and integrity of binaries, config and data running on your servers should be fully known – their integrity verified with cryptographic methods tied to your chain of trust.
There should be no way for these entities to be mutated without cryptographic hardware tokens also being broken. This raises the security bar significantly. (mostly targeted supply chain attacks mounted by highly motivated likely state actors can overcome this bar).
Without this, you are always standing on shaky suspicious ground no matter how much you harden the layers above.
Suggested additions: 1. Don't use a CPU equipped with Intel ME or equivalent, or other system-level management engines like HP's iLO. 2. Look out for keylog devices.
Also, from a practical point of view: sshguard is very useful and easy to install.
You mean, you want to drive to your data centre every single time something fails to boot, or have the awkward debugging session over the phone with remote hands? Or come in with a physical copy of the firmware update every time? Remote management is necessary in some situations. KVMs, ILO, and others have their issues, but it's a tradeoff. "Don't use" is not a great suggestion.
I do exactly this. Granted I only have 50 chassis to deal with and can emergency boot to rescue via PXE. After 1st generation ILO and the enforced insecurity of vendor KVM options I saw the light.
> Don't use a CPU equipped with Intel ME or equivalent, or other system-level management engines like HP's iLO.
This is a nice dream, but absolutely not practical in the real world. All the major hardware vendors have an equivalent, eg. CMIC for Cisco UCS, DRAC for DELL, etc etc.
Not to mention the impracticality of not having these when your equipment is locked inside a cage in a datacenter on the other side of town.
After thinking about it some more, I guess there are very special cases in which you might want to avoid Intel ME. Mostly if your adversary is the US Gov.
Why put /boot on a separate partition? It becomes a pain on Ubuntu if the size is too small, if you forget to run apt autoremove (or whatever the exact command is) that thing fills up with old kernals. Updates can fall because of this and your system breaks in weird ways.
I know this because the original developer of our system used a /boot partition of 500 MB. That's right, 500 MB. I noped out of that. Don't even get me start on 2 GB total disk space for everything else on our API server. This was built in 2013 mind you, plenty of disk space by then...
Have fun getting smartcard logon working on a desktop Linux machine (unless you've got a full-blown enterprise PKI in place -- and, even then, it's still gonna be fun).
Seems to be in progress. In the future it will cover 'automatoc security updates'. Anyone have insight into wether this is a good idea and how it's done on something as rolling as an ubuntu server?
I've had problems on some servers where unattended-upgrades would install new kernel versions without removing the old ones, and ending up filling my /boot partition.
Ditto, for this reason I leave it off. Also, I want to be able to verify patches in staging. Kill me if you want, but we only patch once a month for this reason. For critical things like heartbleed, meltdown, spectre we fast track them obviously. I find it helpful to subscribe to all security lists of core services to know if something really nasty is out there.
It grabs all the security based updates, and installs only those. And it reports the reboot required status. I watch that and make a reboot ticket if it hits the true endpoint.
Edit: the server part is just Node-red watching for those endpoint GETs. Once it sees a true, it fires a Jira ticket in the Change Management space for a security based reboot.
ASLR is implemented in the kernel; although some unscrupulous programs are able to bypass it entirely (or force it to be disabled system-wide as is the case for Dropbox on mac)
I think what they were getting at is that ASLR doesn't work at all (for the main program) if you're not running a PIE binary. The libraries still get it (they usually have to be PIC by design) but that's not much benefit when the application using those libraries has entirely predictable addresses.
So yeah, your binaries need to be built with CFLAGS="-fPIE" and linked with LDFLAGS="-pie".
I have to agree. Table of contents is a bit too ambitious. Anything that wants to cover "Air conditioning", "Fire protection" and configuring Nginx and Apache is not going to be practical.
This is the beginning of a great list, but to apply everything without careful understanding / adapting to your situation would just end up confusing people and causing them problems. Some things are also very distro/environment specific (like the mounts layout).
There are tools which will verify the situation for you, but not force a specific solution.
There are already some automation packages around, in dev-sec project we have them for ansible/puppet/chef including automated compliance tests for inspec.
Does anybody knowledgeable know how hard would it be to patch the same linux kernel with per-application permissions? I know there is some 'OS' doing it but I am looking for a WSL like solution- 'patch' the existing kernel so I can run the binaries et al unmodified.
There would be a permissioning matrix where I can list allow-all-permissions and deny-all-permissions by default and then change the granularity
You understood exactly and apparmor clearly fits what I need (and possibly the other two too but I haven't had time to look...). I used to (incorrectly) believe that SELinux only dealt with networking...
run services as normal user and confine them using apparmor (or selinux if you are masochichst)
the NSA will probably hack you anyway, that drive by script-kiddie that kills your company JIRA instance by DDoSing some gameserver while exploiting some old Wordpress plugin can be stopped by apparmor!
What's missing is some kind of kernel live patching - if someone is not a script-kiddie on your machine that Wordpress shell is a nice tool to elevate to root - your kernel is rotting and exploits are plenty... so consider some kind of live-patching if your are paranoid.
Also no discussion about updates, the need to restart services for reloading updated .so files (like openssl)...
You can obey that guide and be busy with life and some skiddy root exploits your box anyway...
so /rant - there must be some better guides? found those cis benchmarks (https://www.cisecurity.org) a mixed bag - is there anything better out there?