conntrack timeouts don't just apply to UDP:

  net.netfilter.nf_conntrack_dccp_timeout_timewait = 240
  net.netfilter.nf_conntrack_frag6_timeout = 60
  net.netfilter.nf_conntrack_generic_timeout = 600
  net.netfilter.nf_conntrack_gre_timeout = 30
  net.netfilter.nf_conntrack_gre_timeout_stream = 180
  net.netfilter.nf_conntrack_icmp_timeout = 30
  net.netfilter.nf_conntrack_icmpv6_timeout = 30

and even for TCP, there is a timeout after the connection is closed. The fact that UDP has no state and therefore no 'connection' doesn't mean that just because TCP does, that conntrack only tracks it while the connection is open. Besides, you could sever a cable and TCP wouldn't know that anything happened. So you do need timeouts for anything in a NAT table.