You're right. Unless you have some predictable, ongoing medical expenses, putting aside money in an FSA is worse than gambling. I use all of mine every year because I see a provider who isn't covered under my insurance, but I don't think someone who's healthy should bother. If you have a physical and 2-3 routine doctor visits a year, it's not worth it.
The HSA is interesting in that the money you put into it does not go away. For those close to retiring, the funds stashed in one are still very usable after.
HSAs seem a little less problematic, but they still don't seem like a wonderful thing outside of certain circumstances.