I think at least part of the solution is convincing upstreams that the world has changed. There are now many people and organizations who are actively working to subvert software - and some of them have a lot of resources and incentive. They're working to break into a variety of things (including repositories, build systems, and distribution processes) so that people run subverted software. Frankly, the world changed decades ago, but only recently have many developers started to realize it. I try to convince people by pointing out past attacks, for example. One piece that might help you is: https://blog.torproject.org/blog/deterministic-builds-part-o...
We can also make it easier. One great thing is that the Debian reproducible builds group has been modifying tools to make it easier to create reproducible builds. That doesn't mean there's nothing left to do, but making it easier makes it way more likely. The "containerization of everything" also has the potential to make life easier - it makes it easier to start from some fixed point, and repeat a sequence of instructions from there.
> Followup question: how do we find independent folk to help us check our work?
I don't have a very good answer for proprietary software. If a company is serious, I think they should pay people to independently review it. There's great evidence that software inspections detect a lot of defects, but in many circumstances detecting & fixing defects is simply not valued as much as the costs of reviewers. We need customers to demand independent analysis of important software.
For open source software, the situation is often better. I think you should work with the people who write/manage/run the relevant system or language package management tools so that the packages are reproducible.
As far as the broader question of "checking our work", there a lot of things you can do to make it easier for people to collaborate. I strongly encourage all OSS projects to try to get a CII best practices badge: https://bestpractices.coreinfrastructure.org/ That has a list of basic things you should do to encourage collaboration and be secure. (Full disclosure: I lead the CII best practices badge project. But you should do it anyway :-) ).
I'll send the link to my engineering director for a squiz. On a superficial scan I think we hit many of these criteria, but not all. Of interest, we're neighbours -- the Cloud Foundry Foundation is also managed by the Linux Foundation.
The one I'm happiest to exceed is the 60 day CVE-fix window. Our policy is to release updated versions of our buildpacks and rootfs within 48 hours of a high-severity CVE being patched upstream -- usually within the same day, actually. Only possible because we have very extensive testing and build automatic.
For internal reviews and teaching, one idea that one of my colleagues floated was having a red team with engineers rotated through that team. The idea being that it's easiest to think like an attacker if you have, for some time, been an attacker.
It would be difficult to find the right tempo, though. It'd take a few weeks to get a grip on common attack types and then start hunting for flaws, and we'd be struggling to find the balance between rotating as many engineers through as possible vs maintaining ongoing feature work and maintenance.
We already did the containerisation of everything (except we didn't lie to ourselves and just called them the processes they are) and discovered flaws along the way, fixing most of them.
Docker wants to find the flaws on its own, so is repeating the last ~30 years of software distribution development.
Every problem containerisation discovers has been discovered. Every flaw it fixes has been fixed. All that is achieved by the modern insistence that containerisation is in any way something new is a larger attack surface. The ability to return to "some fixed point" has always been present and simple. Judging by many years of building other people's software, for fun or profit, developers simply don't bother to do so. None of my work is ever release without returning the system to a known state in order to run the necessary barrage of tests (which, developers take note, includes TESTING THE GOD-DAMN INSTALLATION DOCUMENT) and I've achieved this without the need for chroot, whatever its name du juor.
The chance of an invisible magic snaphotting system not invisibly breaking the 'back-to-basics' assumptions is precisely zero, invalidating any tests performed within or by it.
Developing against a known state and always and only testing against that state is easy and has been since approximately always. Not doing it is lazy. Stop guessing, developers, programme!
Our ideal world would be fully reproducible builds with a complete chain of custody. We have some of it, but not the whole kit and kaboodle.
But we can't really do this so long as we rely on unreproducible upstream build configurations.