I would go SaaS only. The ones who insist on having everything on their servers are probably going to be high-maintenance customers you don't want to deal with anyway. Remember the 80/20 rule. The 20% of your clients who want the download will take up 80% of your time.
You have to convince them that SaaS is the better deal - regular, automatic updates, tight security, redundant backups, and reliable access - and they won't want to consider the downloadable version.
You should prefer SaaS because you can charge them more in the long run if you get them dependent on you, in addition to reducing the setup time and effort.
I feel like you are fooling yourself here. Many small companies want to download and install not because they are difficult customers, but because they are either a) operating on a limited budget or b) risk adverse.
As you say yourself, getting them to sign up for SaaS is better for the developer because you can charge them more long term. By the same token, many customers just want a simple solution that works, without additional costs.
If I am a small business looking to invest in some software, why would I ever go with a SaaS from an unestablished company just starting out? As anyone here on yc should know, -most- companies fail. Odds are if I sign up for SaaS from an unestablished company (which the original poster's company would be if he is just starting up), they could very well go under in the future. Why would I open -my- small business to that risk if there is a competing program available I could download and install and no longer worry about whether the service provider will be around? That isn't being difficult, that's just being responsible.
As I say in another comment, I would probably still start with SaaS, just because it is easier on the developer starting out. Less worries re:installation process, fewer platforms to support, etc. But I just disagree completely with your characterization of customers who prefer a downloadable product.
You have to convince them that SaaS is the better deal - regular, automatic updates, tight security, redundant backups, and reliable access - and they won't want to consider the downloadable version.
You should prefer SaaS because you can charge them more in the long run if you get them dependent on you, in addition to reducing the setup time and effort.