Your viewpoint is a pretty standard neoliberal one. Personally I consider the existence of such massive inequality to in and of itself be a problem, separate from but related to the problem of poverty.
Which I must admit (to me) is a patently absurd viewpoint, and probably the entire source of our disagreement.
It is a moral issue when someone has great excess that could be redistributed to people who are not yet at the "comfortable life" standard (whatever has been agreed upon). However, if everyone is already at the comfortable life standard, what could possibly be your justification for saying that one guy having more than another is still immoral?
Earlier you said "I don't want everyone to have exactly the same level of wealth or anything like that", but honestly it sounds like you do. If your moral basis isn't "we need to fix poverty" but rather "delta bad", then your ideal must be for everyone to have the same, anything else is internally inconsistent.
And I will point out that despite the other half of your protestation ("I'm not a communist"), this is an unfortunate trait you share with most communist regimes of history – more interested in the "eat the rich" part than the "feed the poor" part.
