It's pretty damn easy to pin down "parameters for loyalty, forgiveness, propensity towards warmongering, and more" (in fact, they already pinned them down) while making the AI smart enough to not leave lucrative resources unimproved, and not smash waves and waves of units into a walled city with +30 strength.
The problem is that not all combinations of those traits are equally valid, and the optimal distributions of those traits are ill defined. If warmongering is some number 0 to 1, with 0 being Gandhi and 1 being Genghis Khan, that's all fine and good. But what does the distribution curve of a randomly generated nation leader look like and how well does that curve conform to the expectations and perceptions of the player? You can't assume independence; players likely expect a high-warmonger/low-forgiveness leader to be more common than a high-warmonger/high-forgiveness leader, but the later combination shouldn't be strictly forbidden. Rather than pinning down all these curves, the Civilization series leans on a finite number of hand-crafted AI players, created by humans exercising intuition and instinct.
As for tactical AI (which is only one component of Civilization), an optimal play from the AI probably isn't what most players desire, nor is optimal play with randomly imposed inefficiencies. Real humans don't run countries optimally, so an AI that runs a Civilization nation optimally won't ring true. You want an AI that rarely makes mistakes that a human wouldn't make, but frequently makes mistakes that a human would make.
Making a good AI for a game like Civ is not nearly as straight forward as creating an AI for a game like Chess or Go because players have different expectations from the games.
> As for tactical AI (which is only one component of Civilization), an optimal play from the AI probably isn't what most players desire, nor is optimal play with randomly imposed inefficiencies.
No one's demanding optimal play from AI. Meanwhile I'm sure pretty much no one's happy with idiotic AI, which is what we have at the moment. I already gave an example: "smashing waves and waves of units into a walled city with +30 strength." Any human player familiar with the rules won't sink 100 city-turns worth of production over 20 turns into attacking an unconquerable city, losing everything while shaving 20% off the wall, then repeat that for another 40 turns; that's idiotic, and that's exactly how the AI behaves alarmingly often. Do you enjoy dealing with this kind of behavior? I bet you don't, unless you only derive pleasure from crushing AIs (which I do enjoy, but the satisfaction from the lack of challenge only lasts so long).
Currently game difficulty in Civ is basically defined by how much of a head start AI players have and how much they cheat (which in theory is a consistent advantage throughout the game but in reality matters less and less once the human player starts conquering), so once you overcome your early disadvantages the late game becomes boring.
> Making a good AI for a game like Civ is not nearly as straight forward...
I never said it's easy to create a great AI for Civ. However, it shouldn't be hard to outdo the existing, idiotic AI by a huge margin (while preserving all the traits like nuke-happiness) if, say, DeepMind decides to put some resources into it.
Making an AI that avoids engagements with unwinnable odds is fine and all, but I would expect that to be a relatively simple matter that doesn't flex the strengths of modern approaches to AI. In fact looking back on my extensive albeit dated experience playing Civ2, I'm pretty sure the AI does in fact prefer engagements in which the odds favor it. If anything, the biggest gripe I had with it is the AI would often employ sound but tedious tactics, moving several dozen units per turn which, although not forbidden by the rules, isn't a lot of fun for the human to wait through. That is a trickier matter, since some humans do in fact play like that, so perhaps the AI should when the human does, but shouldn't when the human doesn't.
I don't think difficulty in analyzing the odds of engagements like that is the reason good AI for Civilization is elusive.