From helping on the Cloudflare Community forum for a while, this seems to be a fairly common issue[0] - users set up Cloudflare and then continue to get attacked since the firewall isn't properly set up to only allow connections from Cloudflare IPs.
Something I don't like is how Cloudflare themselves don't really suggest you firewall off connections that aren't from CF ips, as there's only a support article on whitelisting and not blocking[1]. This is an area I hope CF can improve since any competent, targeted DDOS attacker will know the IP the server had before the owner went to CF, and/or can use a tool like curryfinger to figure it out.
I haven’t set this up in a while but I think you can set up a “fake” cert on the origin to Cloudflare portion and then also pin that cert into Cloudflare, so you get protection against MITM on top of protection from scanning (and from accidentally serving directly from your host). They should probably support “secret name” http headers instead of the normal host, too. So e.g. your site is set to serve for fjeiiejdndjs.dhdjdj.com and publishes via Cloudflare as www.riskysite.com
Cloudflare also has (had? I haven’t kept up) some special accelerated serving products which would de facto protect from this. Doesn’t help if you just have https vs a full vps though.
It would be awesome to have some standardized containers/ami/etc which were set up for “concealed hosting” via cf, ipfs, tor, etc.
My understanding is this shouldn't be necessary. If you only start using your SSL cert after your origin is protected behind Cloudflare, and don't serve traffic to anything except Cloudflare's CDN IP ranges, your real IP can't be discovered through Shodan, Censys, or any other technique.
And even if you don't lock it down to their CDN, it may still never be discovered if your origin only serves the relevant content when a specific host header and SNI are passed (rather than served by default regardless of host header or SNI), which Censys/Shodan may never try. Someone could still scan a huge chunk of the Internet to try to look specifically for your origin, though. Anyone using Cloudflare or a similar CDN should always spend the minute or so it requires to restrict inbound 80/443 to only Cloudflare's published IPs at https://www.cloudflare.com/ips/
I'm talking about the specific case where www.mosthated.com is sufficiently hated that people will scan the entire Internet looking for it. At some point, you'd use other intelligence to develop info on which VPS providers someone prefers for their hidden middle nodes, too. Knocking a few random VPS providers offline and then observing if the site goes down might be sufficient.
IP acl is best practice but the absolute cheapest web hosting options don't make this trivial or even possible. Plus, you could conceivably scan close to the hosting provider candidate by jacking a CF more-specific.
In the general case you are looking for server farms (digital ocean/amazon/etc) and passing the host header to get around reverse proxies. That could still be several million ipv4 numbers
Utilize SNI and serve up a fake cert when someone scans you without a matching hostname. Censys is scraping you by IP, so it'll just see the fake cert.
You can do this in nginx by making the fake cert the first server block.
To me it looks like this tool (and cloudflair.py) can only ever find servers which are configured in a very specific way:
* The traffic between server and CDN is encrypted using a valid certificate
* The server's firewall is not properly configured
Apparently there are indeed servers with this configuration, but I just find it odd how someone would go through the trouble of setting up HTTPS (instead of terminating it at the CDN) and then not bother to block traffic from anywhere but the CDN.
Imagine someone with a fully configured HTTPS web server (without a front CDN), that adds Cloudflare in front of it when the load gets higher, and does not bother configuring their firewall.
Could be a shared hosting platform behind the CDN where the user doesn't have the ability to ip-acl but does have https. Common among low-end users and in the particular case of a "hidden" site, easily purchased cheap VPSes which are de facto anonymous.
Because I imagine someone who would expose their mongodb instance to the web would also just terminate HTTPS at the CDN, rendering curryfinger/cloudflare.py useless.
"One; Python has its uses, but writing highly performant multi-threaded scanners is not one of them"
Is that still true given Python 3 asyncio? My understanding is that it's really well suited to writing things like network scanners, without needing to run them in multiple threads.
It hasn't been a problem for most of our things at Shodan. I think there's a lot of confusion about when to use multithreading vs non-blocking sockets. Most security tools tend to use multithreading when they should just use non-blocking sockets. With Python3 you can also swap out the scheduler if you want better performance:
asyncio is still bound by the GIL. It's not like Go's continuations which can parallelize on the CPU. There still can only be one TCP payload from your Python process at one time. The author wanted to spam the network and use all his bandwidth. Python multiprocessing would have worked for this though (probably faster than the solution using shell commands, but obv this is network-bound much more than CPU bound so who cares).
While asyncIO hugely helps, this + interpreted language wont yield better performance vs custom native code doing true native threads which themselves are also using async methods.
They're websites that scan the entire internet to see which hosts are up and responding, and which ports are open, and then make the data available to researchers.
Easy solution for hiding yourself from most scanners (Shodan, Censys) - only allow requests with proper SNI, don't serve your server's cert by-default. Firewall is also nice, but you could make a mistake at some point.
Curryfinger seems to work by querying Shodan and other scanners. Those scanners seem to work by just connecting to an IP address's port 443 and look at the certificate. If you always require correct SNI (the domain you host) then that scanning stops working (you literally disappear from Shodan for example). The fix (to scanners) would be to try and resolve every domain name you know of or scan every IP with every domain name you know, that's unfeasible. Only replying to correct SNI is not a defense mechanism by itself, but it does make it more difficult for attackers.
It turns out that if you have a targeted domain you have a good chance of finding it in one of the popular cloud hosting ranges. Masscan + curryfinger work well together. Alexatop + masscan + curryfinger makes an interesting dataset.
Something I don't like is how Cloudflare themselves don't really suggest you firewall off connections that aren't from CF ips, as there's only a support article on whitelisting and not blocking[1]. This is an area I hope CF can improve since any competent, targeted DDOS attacker will know the IP the server had before the owner went to CF, and/or can use a tool like curryfinger to figure it out.
0: https://community.cloudflare.com/search?q=firewall%20cloudfl...
1: https://support.cloudflare.com/hc/en-us/articles/201897700-W...