If C is binary or categorical, then confidence intervals don't make sense to talk about (although you could still talk about them if C is integer valued), and the equivalent thing is P(C|x). Unless I guess if you have a posterior distribution over distribution...
The definition of a Q (e.g Q = 0.95) confidence interval a < X < b is just that P(a < X < b) = Q. It's not even necessary that the interval contain the most probable value of X, but when you actually calculate them to want to tack on extra constraints to prevent stuff like that, like minimising the length of the interval.
If C is binary or categorical, it totally makes sense to talk about confidence intervals. Why wouldn't it? You are trying to figure out if your probability output for a class is reliable for that sample.
The definition of a Q (e.g Q = 0.95) confidence interval a < X < b is just that P(a < X < b) = Q. It's not even necessary that the interval contain the most probable value of X, but when you actually calculate them to want to tack on extra constraints to prevent stuff like that, like minimising the length of the interval.