DNS TTLs are fraught with peril. For one, a lot of clients ignore them. [0] Secondly, a lot of ISPs ignore them too. At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high.
When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut it off at that point. I've had similar problems when using DNS based traffic management. Sometimes when we say 50% to one IP and 50% to another, it can end up at 70/30 because of ISP and client caching.
And lastly, my friend who ran an ISP in Alaska told me that they just set all TTLs to 7 days minimum to avoid traffic back to the lower 48.
It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake. It's a catch-22. No one plays nice with TTLs anymore because everyone sets them low, and everyone sets them low because no one plays nice.
We should do the right thing even if other people do the wrong thing.
Traffic signs shouldn't be taken down just because some people speed regardless of the posted speed limit.
Fundamentally however, giving clients control over when failover occurs means giving up control over failover.
Literally just a couple of hours ago, I was patiently explaining this to some Azure techs from Microsoft.
The conversation went like this:
> "Just use Traffic Manager for active-passive fail over!" they piped up, helpfully trying to solve my problem.
> "Client software often ignores DNS changes."
> "We designed Azure so that Traffic Manager does failover and load balancers are always active-active."
> "That's nice, but client software ignores DNS changes."
> "Umm..."
Fundamentally, the DNS protocol is just too old and crusty for the modern Internet, much like SMTP and FTP. It's best avoided to the extent possible.
The big providers seem to be converging on Anycast IP routing, where a single address can be routed to multiple data centres dynamically. You're all probably familiar with 8.8.8.8 and 1.1.1.1, both of which are Anycast IPs, but anyone can create a global static IP like that in minutes via a service such as Azure's Cross-region load balancer: https://learn.microsoft.com/en-us/azure/load-balancer/cross-...
> Fundamentally however, giving clients control over when failover occurs means giving up control over failover.
Fundamentally the problem is that you want to direct a client to particular datacenter and it is not always shortest part (shortest != lowest latency) so anycast DNS isn't helping here.
There is reason biggest CDNs use it to direct to particular DC instead of "
just anycasting everything" like you're proposing, it's suboptimal.
> You're all probably familiar with 8.8.8.8 and 1.1.1.1,
And both companies behind that still direct you to particular datacenter, not anycasted IP, for the actual content.
Anycast IP are also "expensive", you need to use entire /24 route to get one, and it takes routing table space in every router's memory that is limited, and requires costly replacement if it runs out everywhere, not just where you distribute it.
They aren't really cached. Routes are kept in the routing table as long as peers are responsive, if they stop responding to hello's or a link goes down to directly connected peer the routes are removed (or replaced with a less preferred one). While its possible to configure a router to keep routes in the table for a peer that is down for some time its typically not used unless you have a good reason for it.
routing tables usually aren't "cached" in the same sense that DNS caches entries.
however. Inside ISP;s routes are usually distributed by an IBGP with a couple of route reflectors, and route reflectors will drop routes if no router within the IBGP domain is advertising the route.
Routes between ISP's are usually fairly static thanks to mechanisms like graceful restart, which makes sure routes do not get dropped, even if BGP has a restart between routers. THere are various other mechanism in which to achieve this. (look up route flap dampening if you want to learn more).
The service I worked for in AWS faced this as a routine pain. No matter what the TTL, stuff completely ignored it. In the end the only sensible thing we could do was accept that we were going to impact customers and that we just can't realistically account for misbehaving DNS infrastructure. We had something like a 5 minute TTL on our public record, and would wait 15 minutes after removing the IP from the DNS records before we would take action that would potentially break connectivity.
By setting the DNS TTL, we had established the terms by which we offered the service. Failure to accept those terms is not something we can control.
What we generally found was that clients disconnected would reconnect almost immediately to one of the other IP addresses. Any disruption to the client appeared to be almost negligible.
It doesn’t help that Java’s default configuration is to cache DNS entries forever for “security”[0]. Any long running server that makes external requests will eventually hit an issue where one their dependencies no longer works because DNS has changed and the TTL is ignored. I’ve seen this bite teams for 15 years and it will probably keep doing so well into the future.
Only if you have a security manager enabled (=are running in an applet/webstart). The entire security manager concept has been deprecated, so things are looking better
I'd say about 80% of traffic moves within 2, maybe 3 TTLs. That 20% of traffic though... I've never really seen traffic on old servers go to zero real looking traffic, even after waiting months. Sometimes there's a dropoff when you send RSTs to SYNs, but sometimes not.
If you're using a DNS service that charges per query, set the TTLs high! Otherwise, 60s is probably good enough for most failover. I wouldn't go too far under 30s, unless you have a good reason; some recursives do stupid things with low values, and why tempt fate. Otoh, a lot of high profile sites and services do have < 10s ttl, so it probably works?
From the CDN & authoritative name server side Ive mostly dealt with eyeballs and a smattering of devices. With ~1min TTLs itd be 80% in 5min, 95% in 20-30min, 99% at an hour, 99.9% 1day, the remainder entirely variable. Literally stopped announcing BGP prefixes for years, turn them back up, get immediate client traffic to 80 & 443. Have done down to 10s and lower and its fine (as far as we were concerned).
> "When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds"
Likely because the clients just weren't doing DNS lookups. I would often run into clients written to resolve a hostname and then query the resolved IP for the duration of the process (often this is done as part of a library or runtime). Our customers wouldn't restart their apps for weeks or months so there was always a long tail of clients which never picked up dns changes.
DNS wasn't designed for failovers. We can use it for that, but it's never going to be perfect by design.
Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way.
You sure can. There are many details and caveats and it takes a lot more infrastructure and technical sophistication. Many small sites don't control their own routing at all, for example.
DNS is very simple, easy, and mostly good enough. And DNS failover is now well-enough known that we can mostly expect clients to accept fault if they don't engineer to expect it.
What you can't do with the above is more fine grain load balancing. If your LAX node is getting overloaded, you can use a combination of geo and DNS to start splitting some amount of traffic away.
>It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake.
IMO that is a little misguided. 60-second and higher TTL will be honored most of the time. And I don't really get what that ISP in Alaska was doing: DNS is about 0.1% of Internet traffic (according to some quick googling), so decreasing that will save you only a tiny tiny bit of bandwidth at disproportionate inconvenience for your customers.
When it comes to TTL I try to imagine the perspective of the person on the other side that implemented the resolver at a university (which copies might be anywhere) or isp a couple decades ago. What odd behavior did they have to manage and what quick hacks might they have done to fix them?
From this I have chosen a minimum of 5 minutes of TTL. Anything that might have caused a large amount of dns traffic in the early day of the internet to go off every minute, or worse ever few seconds, might killed their server and taught the young programmer to sanitize inputs.
>Sometimes when we say 50% to one IP and 50% to another, it can end up at 70/30 because of ISP and client caching.
That doesn't really matter. Once a server becomes saturated you just remove it from being returned. Having multiple records being returned just slows down how fast a server can become saturated. You don't need the multiple records to be saturated at the same rate.
I think you're misunderstanding. If I have a lookup for foo.com configured to return 1.2.3.4 half the time and 5.6.7.8 the other half of the time, I want half the traffic to go to each load balancer. Maybe I'm running different software on each, or behind each. Maybe I'm using different networking gear. There are a ton of cases where getting close to 50/50 is what I need to get good experimental data.
Why would you think that it doesn't matter that the traffic doesn't match the ratios I picked? I picked them for a reason.
I think charcircuit suggests returning the inverse of the actual measured traffic: if the target is 50/50 and backend A gets 70 while backend B gets 30, one solution could be to return A in only 30% of the requests and B in 70% of them, leading to an actual 50/50 distribution.
I am suggesting that the returned servers are the servers with the least load. Yes, there may be uneven load being assigned to these servers, but clients are being given the choice of selecting a server with low load.
I'm suggesting to use more horizontal scaling. So if you have 1.2.3.0/24 and you advertise 2 servers at a time you can start with 1.2.3.1 and 1.2.3.2 and then you would cycle out servers as they increase in load. If one server is getting less traffic then it just takes longer for it to cycle out.
People without cached records will always get the returned servers with the lowest load.
I was referring to load balancing and not splitting traffic for an experiment.
You're going to tend to have unevenness there because you're returning results to recursive DNS servers, not clients; so maybe you get unlucky and return A to more large ISPs; their recursive servers serve more clients per lookup and you've got lumpy results.
Then, if you return both IPs, you're very likely to eventually come across some people who sort multiple returned A/AAAA records and will prefer one that's "closer" to their current IP. Because if you're in 1.0.0.0/8, it's certainly going to be better to connect to 1.2.3.4 instead of 5.6.7.8. Same thing happens in v6 land, of course.
If get unlucky A's load will rise quickly and it will no longer have a record for it. As the TTL expires the resolvers will get a new set of servers with low load.
Lumpy results just means the times for an IP to be rotated out will be lumpy, and not that the server load will be lumpy.
Yes, that is how you scale DNS load balancing. There is a pretty low limit for the number of records you are allowed to return in practice. Putting 100 A records works with 1.1.1.1, but will break other's resolvers on the internet.
Sure, but a lot of people use other people's DNS services. The number of times I've been allowed to build a DNS service for my employer is zero. I've used static DNS, where you just put in your X number of A records and hope; dynamic services where you put in X and have them serve only Y of them in any request and hope. I've used services with load feedback, but those are always a much higher tier.
>Sure, but a lot of people use other people's DNS services
There are many of DNS services which offer an API for querying and updating records.
It is very easy to write a service / script that just fines the X least loaded servers and then call an API to set those as the available records.
In practice you will also want to implement some monitoring that it is actually working.
> At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high.
DigitalOcean does this, or at least some of the collocation/hosting providers they are using for their infra. They appear to redirect :53 connections to their own resolvers that cache NXDOMAINs.
It's an ugly situation. DNS seems like the wrong way to do load balancing. Meanwhile web browsers seem to have decided that it's a great idea to send out DNS requests for every domain on a web page (of which there are dozens, thanks to adtech, CDNs and other stuff.)
If you're in a corporate environment with control over the infrastructure, low DNS TTLs allow you to mostly guarantee a failover. There will always be clients that ignore TTLs, but most of them are fixed by IT rebooting the box.
As the poster says, when it comes to the public internet DNS TTLs are more like guidelines.
Luckily most client DNS caches are cleared on a reboot.
FYI, if I remember correctly Windows (and MacOS?) used to cache negative responses for "a while".
> It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake.
That's a weird conclusion. My conclusion would be that you can't force low TTLs for everyone or even most users to you don't can't use them to allow for quick changes and may as well let everyone else benefit from caching too.
The scenario I think about is, what happens if the DNS entry I just pushed had a typo in it? If my TTL is 60s, then I push a fixed entry and my site is back up quickly. If the TTL was 1hr, then anyone who cached that bad entry can't connect for about that long. 60s is better for disaster recovery, 1hr is better for performance.
I've seen this happen on internal networks where IT support tries to guide end users through flushing their local DNS cache so the users can get back to work before TTL expires on a broken entry.
The model I've been trying recently is to make DNS changes in three parts. First: change existing DNS entry TTL from 1hr to 1s. Wait 1hr as the caches clear. Update entry to the new address, keeping 1s TTL. Wait, test, and monitor for a while. Finally, increase the TTL back to 1hr. It takes more planning, but I avoid a 1hr partial outage if I mess up the address.
Most sites I've worked with change DNS <1 time per year, so outside infrequent, planned maintenance a long cache time is best for performance.
If you have DNS typo for a fixed time, the longer the TTL the fewer clients will ever see the error. E.g. lets say your typo is up for 10 mins. With a 5 minute TTL everyone will be seeing the error. but with a 60 min TTL, 5/6 of your customers will never see the typo (assuming there's enough traffic for caching)
Slow onset of impact is probably more useful in minimizing incidents than fast onset of fix (edit: or rather this is a trade-off and one might consider both directions).
I think the amount of people that would see the error either way would be the same, assuming constant fix time and traffic. This means lowering the TTL as much as possible should be the right solution, because those affected will only have the issue for as long as the fix takes (hopefully less than 1hr) whereas if it is kept at 1hr those affected by the error will have it for an entire hour.
Testing strategy is also pretty important here. If you can run integration tests that bypass DNS caches immediately after the change, then you should know quick if things are working. If both old and new DNS settings are valid, the slow onset works great and minimizes impact of an error.
I've handled DNS changes where the underlying host was changing IP address. In that instance the old DNS entry immediately became invalid, so a quick DNS cut-over was needed for all clients.
> The model I've been trying recently is to make DNS changes in three parts. First: change existing DNS entry TTL from 1hr to 1s. Wait 1hr as the caches clear. Update entry to the new address, keeping 1s TTL. Wait, test, and monitor for a while. Finally, increase the TTL back to 1hr. It takes more planning, but I avoid a 1hr partial outage if I mess up the address.
This is my preferred way of doing it, especially for MX records.
I wonder what those clients are, none of these warnings ever give examples in my experience so I can't tell if this is old out-of-date information being repeated or it is an active modern problem.
I know there was a DNS resolver that did worse than impose a minimum like that, it treat short TTLs (<300s) as errors and "corrected" them by using its default (often 24 hours). I actually encountered it and its effect in the wild, but this was back in the very late 90s or very early 00s. It was a DNS cache daemon on an old Unix variant (I forget which one, my university had a few floating around at the time), so old that if someone is still using it they have far greater issues than my low TTL value!
My current attitude is that if I have the need for short TTLs then I'm going to use them (though 300s is more than short enough for anything I envisage doing ATM & 300s seems to be reliable) and if someone's DNS cache is broken that is their problem, much like I don't pay any attention to issues Internet Explorer users might have looking at some HTML+CSS+JS I've inexpertly strung together.
HughesNet used to run a caching DNS resolver on their customer equipment circa 2016 (not sure if they still do) which overwrote TTLs to 12 hours.
Java/JVM historically ignored TTL and just did a single DNS lookup and cached the result unless configured otherwise.
PGBouncer ignores DNS TTL and has its own configuration.
I've seen lots of software that will create a "connection" object and re-use that with TCP reconnect handling. In practice, that means the DNS resolution happens when the connection is recreated but not necessarily when the TCP connection reconnects due to timeout/failure. I've also seen some issues with long lived connections (where the connection greatly outlives the TTL). Long lived connections aren't necessarily a DNS TTL problem but they mean DNS load balancing/redirection don't always work how you expect.
That said, the solution is usually to just fix the issue in the software and not ignore DNS TTLs.
As counter examples, AWS ALBs and Kubernetes service discovery both rely on short TTLs with fairly good success.
Some heavily loaded ISP's have modified daemons that set a lower and upper TTL threshold. Anyone running Unbound DNS can also do this. Some versions and distributions of Java do all manor of odd things with DNS including ignoring TTL's though I do not have a current table of those misbehaving. The same goes for some IoT's but I have no idea what resolver libraries they are using.
Yep. Just checked the docs for that and “cache-min-ttl” is a thing. For those unfamiliar, unbound is a pretty common resolver: it is the default for most BSDs and things based on them (such as pfSense) and used by the popular PiHole and its forks. How common it is to use this setting I can't comment on.
> Some versions and distributions of Java do all manor of odd things with DNS including ignoring TTL's
This at least is less of an issue if all you are futzing with the DNS for is services intended to be consumed by a web browser: if common browsers and DNS resolvers behave OK then you are good, if someone consuming your stuff through their own code hits a problem they can work around it themselves :)
It goes without saying that only a fool would play trick like this for email as that is already a mountain of things that can be easily upset by anything off the beaten track.
The reason I mention Java is that when updating DNS in a business, there may be business to business flows that are critical. This is becoming more common than ever with cloud-to-cloud-to-cloud dependencies. Mishaps in cloud services can affect a service used by a large number of people. That is why I always tried to teach people to design apps so that TTL could in theory be 1 week and it would not matter. e.g. apex domain anycast IP used for authentication, then direct browsers, API clients and cloud daemons to a service dedicated domain name that is also anycast and uses health checks behind the scenes to take nodes out the picture rather than relying on DNS.
Getting DNS right is non trivial, and UDP will be dropped at times. If you have to lookup the name every two seconds means you have a high risk of getting failed lookups even if your infra team is competent. (10 minutes should be ok)
One cool thing about DJBs tinydns is that you can configure TTLs if you plan to roll over to another system at a particular point in time.
I.e. you can have multiple records each with its own time range where it is valid.
From the tinydns-data documentation:
--
You may include a timestamp on each line. If ttl is nonzero (or omitted), the timestamp is a starting time for the information in the line; the line will be ignored before that time. If ttl is zero, the timestamp is an ending time (``time to die'') for the information in the line; tinydns dynamically adjusts ttl so that the line's DNS records are not cached for more than a few seconds past the ending time. A timestamp is an external TAI64 timestamp, printed as 16 lowercase hexadecimal characters. For example, the lines
specify that www.heaven.af.mil will have address 1.2.3.4 until time 4000000038af1379 (2000-02-19 22:04:31 UTC) and will then switch to IP address 1.2.3.7.
Note this data many no longer be correct. I just checked and numbers for some of the websites are much longer now:
raw.githubusercontent.com has 1 hour TTL. github.map.fastly.net is 5 minutes TTL. detectportal.firefox.com is 60/120 seconds TTL.
Another thing missing from the post is exactly _how_ bad is it -- because latency matters. If there is a choice of 30mS extra latency on initial site visit vs a chance of 1 hour of downtime in case of hardware failure, it is not clear what the best choice is. Especially things like detectportal.firefox.com which are asynchronous and normally do not introduce extra user-visible latency at all.
Fascinating read. Of course I'm no big business but for the last 20 years I've done it like this for non-critical stuff:
- NS Records at 86400
- A and CNAME at 3600
- 1-2 days before moving stuff to other infra (ISP, DNS reseller, etc), switch to 600 or lower, then 1-2 days later switch back.
Can't even say why, it's cargo culting but it has always worked out fine, but ofc I'm not talking about production APIs.
And every single time there was some problem, setting the TTL to < 1minute would not have helped, so I guess I'll stick to it.
The way of measuring this affects (and skews) the outcome. DNS queries with low TTLs are requested more frequently and because of that, the author seeing more of them pass through his patched DNS relay, which was only left running for a few hours instead of for at least the max TTL they wanted to measure.
That's what I thought of as well. Data can be presented in many ways to show many meanings at first sight. If charts would be generated in number of unique NAMEs in relation to TTL, readers would probably get a diffetent feeling. Or maybe not.
I don't know, but this issue probably has a name ... selection bias?
Yeah,that was one of my first thoughts as well. I guess the raw.github example gets not only more traffic (irrelevant here) but probably also many more requests than hundreds of other sites combined.
Also no clue how actual RR DNS load balancing (without anycast) is still widely practiced, because I guess that also influences it a lot, you may be able to fix a DC going down on a completely different level than DNS. And my experience with cheap home routers is basically that they might cache hostnames for a month (we had stuff with tracking pixels in 2013-2017, EOLing some stuff was wild, we switched DNS and days and weeksl later still got valid requests to the old IPs).
One reason to use low TTL’s is for a DNS-based load balancer that is typically used for DR such as Azure Traffic Manager. That and paranoia about incorrectly updating a record causing your service to break for a period of time with nothing you can do about it.
There are use cases for low TTLs and use cases for high TTLs
The statement here is misguided.
Use whatever TTL serves your purpose.
DNS was designed to be distributed and scalable. And in my opinion it is THE MOST scalable protocol as it basically allowed the entire internet to work that way. Any user in the world can get any public DNS record for a site.
I interpret their data differently: only 47% of DNS lookups were for expired entries due to TTL. If some of those were for longish TTLs, some for actually changed records, then an even bigger majority of requests would be needed anyway, and longer TTLs would not have stopped them.
My uninformed model is that it's all pull based, and for changes to propagate you basically have to wait for cached entries to expire. It seems to me that if there were a cache invalidation mechanism you could have fast updates, but long cache TTLs.
(I know that if this doesn't exist, adding it would be basically impossible, since some servers won't update to new protocols)
There's DNS NOTIFY (RFC 1996), which is commonly used to trigger a pull. But to minimize malfeasance it's usually ignored unless authenticated (e.g. via DNS TSIG) or appears to originate from a known address. The typical scenario is a master nameserver notifying a list of [known] slaves about a zone update.
There's also DNS Push (RFC 8765), but it's a very recent mechanism. I'm not familiar, but I doubt it's widely supported.
There is the NOTIFY extension, but it's for master servers to notify secondary servers that they should do a pull. Not really usable for unknown general clients, of course.
An outage at a large (but not mega) company will likely have a 5-10 minute period to get looked at, a 5-10 minute period to understand the problem, and a 5-10 minute period for resolution. Any DNS related problem with a TTL higher than 5-10 minutes could greatly extend an outage.
It makes a lot of sense, if we think about how the protocol works. Without caching, I assume every client would send every query in it's entirety to the root server, right?
Since a single zone can have multiple labels ({abc.def}.ghi.example - abc.def is a NAME in ghi.example), the server needs to receive the entire requested name in order to properly respond (yeah, root queries usualy optimize this, because all NAMEs there are one label long).
Not really, because infrastructure is much more dynamic today. Maybe if your IPs never changed long TTLs are OK, but when you're running behind a cloud load balancer, with changing IPs and failover to different regions/providers, low TTLs make more sense.
I like changing my IP address from time to time. I host my own VPN and connect to it through a domain. When I update my IP, I have no internet access for whatever the TTL length is.
If you use a pihole, you can force the minimum ttl for downstream clients (upto a maximum value of 3600s) by setting, for example, min-cache-ttl=2400 in /etc/dnsmasq.d/ Never had a problem with this.
Of course, you can force a flush of the cache using `pihole restartdns` in case you need to.
I'm using Netlify DNS for most of my websites. Usually I set TTL low (~600 seconds) in order to quickly change configuration in case I fuck up. When things are working though, I remember that Netlify doesn't have an edit function for DNS, so I just leave it as is.
IME DNS TTLs are low because debugging DNS during development is a pain.
So you set them low the first time you run into a DNS issue where you had to wait an hour or a day to resolve some bad configuration and your "fuck this" meter went from 0 to 100.
Even if the DNS update is available to an end user, if you're using HTTP2 then web browsers will keep the connection to the old IP address until they receive a GOAWAY frame.
Anyone going “just use low TTLs all the time” doesn’t have a lot of traffic to deal with.
I have a commit of ~15bn queries/month with my DNS provider ATM, and if just one busy record has its TTL set erroneously low, it can cost an extra few thousand $ a month in overages. It’s not a great conversation to have with the CFO when that happens. Caching matters.
If your 15B queries are generated by 15B website visits monthly, your web properties should be generating tens of millions of dollars in monthly income. So a few thousand dollars is nothing.
Unless your traffic is non-revenue generating or somehow poorly monetizable...
In any business doing proper budgeting and trying to make a profit, a few thousands in UNEXPECTED costs, is huge. It can be the difference between your department having the money for new equipment, or begging for a budget increase because of unexpected costs to get new equipment.
It seems like the problem is expecting a fixed infrastructure budget while trying to create ever larger customer engagement. Is it just modern companies that don't realize you can't count your profits until you've got your AWS invoices for that month?
It's a fixed _overhead_ issue, not a fixed _budget_ issue.
There are many AWS services for which you cannot make reservations. You can get better per unit rates by pre-purchasing, but even with instances, network transfer is a per unit factor.
Your only way out is the built in tools to limit service in these cases or to build your own circuit breakers and implement them, or make your usage so small as to not truly require the cloud in the first place.
Your average consumer router is a pile of garbage that barely works.
I'll admit I don't have the stats in front of me, but I'd bet over half of them either don't attempt caching at all or do a horrible job at it.
Of course, your upstream resolver (hopefully) does do some caching, but as most of the latency is getting to this resolver in the first place that doesn't really matter.
Consumer router doesn't seem to need to given that they typically have such low client counts. Your computer is doing caching, so there's not much benefit for your home router to also do caching.
Maybe you live in a country where ISP routers are well designed and low latency, but where I live in the US the primary latency in DNS lookup is almost always the round trip to the horribly configured and slow ISP DNS.
It seems the only reason given is "some clients are broken"?
They mention latency but I wonder what the real world impact is. On a home network, most users are using ISP supplied DNS which /usually/ has pretty low latency and will have a huge cache already built up. Corporate users will have DNS cached to all the common host they're visiting.
Anecdotally, AWS ALBs (HTTP load balancers) have a CNAME with a 10 minute TTL (maybe it is 60 seconds) and can have their IPs change 100s of times a month and this works fine as a default configuration for many use cases.
When I got most of my TLS certs from a commercial CA via OV or EV processes, I could have my cert in place before I repointed DNS.
Now most of my certs come from Let’s Encrypt via DV, which checks DNS. So I have to repoint DNS first, and risk users seeing a cert error before certbot finishes getting the new cert. So I keep my DNS TTLs a lot lower than I did before.
Also, DNS service is a lot cheaper than it was years ago, so it doesn’t hurt my budget to send more requests back to the name servers.
That sounds like a server setup issue. Nothing about LE prevents you from getting a cert with the old server and moving it to the new server before switchover.
Well if you don't like people using low TTLs maybe you should have put validations in the original spec. Oh you didn't write the spec? well sucks to be you then.
Another reason to set a low DNS TTL, is if you are running a service with very limited resources and still want some kind of HA / Disaster recovery. For instance you setup your web app in both Digital Ocean and Linode, with a system that takes the database (for instance mysql replication) and the web app synchronized in the "replica" VM. When something bad happens and you can't recover, you switch DNS and that's it.
As an admin, I use low TTLs when I do some major DNS maintenance. I wouldn't mind upping the TTLs after, but... what about if I need to do some emergency maintenance, like the IPs on a load balancer change and I can't get the old IPs back? I don't want to wait an hour for clients to pick up the new IP.
It would be great if there were some reverse notification to let revolvers know to refresh their caches.
I've been bitten in the rear far too often with high TTLs. Especially stuff like AWS Beanstalk which tends to delete and recreate resources if you're not extremely careful... thanks but I'll stay with 60s or 300s TTLs.
When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut it off at that point. I've had similar problems when using DNS based traffic management. Sometimes when we say 50% to one IP and 50% to another, it can end up at 70/30 because of ISP and client caching.
And lastly, my friend who ran an ISP in Alaska told me that they just set all TTLs to 7 days minimum to avoid traffic back to the lower 48.
It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake. It's a catch-22. No one plays nice with TTLs anymore because everyone sets them low, and everyone sets them low because no one plays nice.
[0] https://www.ctrl.blog/entry/dns-client-ttl.html