One of the very muddled thoughts I have in my head, along with Goodhart's Law and AIs which blissfully attempt to convert the universe into paperclips, is that having a single function maximized as a goal seems to give rise to these bizarre scenarios if you begin to scan for their existence.

I have started to think that you need at least two functions, in tension, to help forestall this kind of runaway behavior.

Even "two functions, in tension" still assumes that you can capture values as functions at all. But the reason ethics and morality are hard in the first place is that there are no such functions. We humans have multiple incommensurable, and sometimes incompatible, values that we can't capture with numbers. That means it's not even a matter of not being able to compute the "right" answer; it's that the very concept of there being a single "right" answer doesn't seem to work.

I think that's what it will approach in the limit, yes, if you are talking about humans. For AIs, I think it will be somewhat less so, and that it would be preferable for the sake of predictability.