I think your framing makes it seem like the negative impact of failure comes just from the judgement of others - I'd say it's much easier to build a positive self-image if you remove yourself from the environment in which you were found to be unfit.
I’m not sure what you mean. Is it that these people wouldn’t have a positive self image if they were in a role that’s a bad fit?
Sometimes I’ve thought of that and realized how it would be wrong to label them as incompetent when they’d be a star at something else. Other times I imagine they’d somehow still have high self esteem and handle this situation super well and I try to figure what that would look like.