You can include all the bulky math libs in the executable. That way if you want some version of Tensorflow and some other executable wants some other version of Tensorflow, there isn't a conflict.
IMO it is no problem if your software is several hundred megabytes in size, I have 8 TB of hard drive. Storage is cheap as long as you aren't using Apple.
So have both CUDA and non-CUDA versions bundled, along with the specific CUDA and cudnn version it needs. Still not that large. Maybe a couple gigs at most for everything, and a LOT of headache saved in return which I think is a fine deal.
I mean, very often the Tensorflow version a piece of software wants doesn't actually have a version available for my currently-installed CUDA version. I'll have CUDA 10.1, 11.0, and 12.6 already on my system for other pieces of software, and your software is hellbent dependent on Tensorflow 2.3 and CUDA 11.1 and I'll end up having to install 11.1 in addition anyway just to run your software.
IMO it is no problem if your software is several hundred megabytes in size, I have 8 TB of hard drive. Storage is cheap as long as you aren't using Apple.