Why not use an open source one? I saw a Google tech talk by Keith Fulton, one of the main architects of PlaneShift. He said that although the engine is GPLed, the characters, rules, scenery, etc. are all separate from the main project and can be scripted. So if someone wanted to, they could improve the GPL engine while creating proprietary art and storyline that could then be marketed.
Unfortunately, you can't use GPL or even LGPL licensed code on consoles, and try getting (L)GPL code past your publisher's/investors' lawyers. I had to go through an arduous approval process to be allowed to use zLib in a game.
