Fastmail tells you very explicitly what to do and how to do that, I recommend doing that, at one day you may be hit with a very important email going undelivered.

I'm using the recommended Fastmail setup with my own name server, which doesn't support DMARC out of the box. Is this something I should look into to improve deliverability?

I haven't bothered with anything beyond the recommended SPF and DKIM records, and am not aware of any issues. IMO it will be years before not having DMARC is a serious issue for delivery, and I expect Fastmail to tell me when that time is getting close and/or take care of it for me.

DMARC is just a TXT record. Which DNS server doesn't support TXT records out of the box?

I meant that the recommended guide doesn't include anything regarding the DMARC setup. Technically, it should be possible of course.

If you point to Fastmail as your name server it will even setup the records for you.

Have you tried it with this service? My fastmail managed custom domain doesn't have DMARC set up according to this site. The other two are set.