You also have to remember that from the beginning they knew they will also need * and # or some kind of extra keys for programming and admin tasks. With touch tone phones came many other innovations in the phone industry. So with 5x2 layout you would actually have to make it 6x2 or 5x2 +2 on the bottom. In both cases it would look and function poorly.

Also, remember that they had a lot of left over molds for rotary phone designs, and they probably intended to retrofit pay phones with rotary dials to touch tone. 3x4 layout fits a lot better into a circle than 5x2 or 6x2.