I don't think it's as rigorously defined as it's purported to be. I just envision the practical application of manipulating values to produce desired results, and then post-hoc rationalizing those value manipulations to obtain a higer-than-warranted level of confidence in the result, because, "I applied rigor!"
He's talking about the practical effects of fallible humans using the version of Bayes advocated in the original post (citing Yudkowsky and Muehlhauser).
Pulling numbers out of your backside and running them through a process makes you more confident than just pulling them out of your backside directly, but I've yet to see evidence it does more than increasing your confidence.
Oh yes, I wouldn't question the math -- it's well beyond my expertise, and I've heard of way too many implementations of the theory to refute it at a theoretical level.
