Unless you provide sources, and, really, a viable build environment -- see RMS's discussions on this, and such counterexamples as Red Hat's rather difficult-to-reverse-engineer build environment (the real value-added of CentOS and other RHEL free forks), as well as Microsoft's long-standing source licenses to academics (build environment not included AFAIR) -- it's possible to hide either failures or backdoors in your products. True crypto is well-tested crypto. That doesn't mean "verified" (can't prove a negative), but it does mean very thoroughly vetted.
Even proprietary security companies have long practiced source provided (different from "open source") code for their key crypto engines. PGP comes to mind in this regard (the company, not the protocol).
But fully open source means you've got vastly more exposure of your crypto guts to examination.
Because availability of the source without conditions is the only way you can get effective independent audit. It's not a magic wand, but you're relying much more on trust without it. This, on top of many-eyes and the pressure not to rely on secrecy of the source for security (a bad practice) improve the situation substantially. Even so, nothing will be perfect, of course.
Open source is necessary (or nearly so) but not sufficient.