I am doing a pilot of the same thing.

so far I am mostly using tailscale + firewall. Using a firewall directly on the host as you mentioned seemed a bit dangerous - although we are trying it on a few servers. For now cloud provider firewall + tailscale.