Well, others have been mentioning similar projects so let me plug in my own : Bitbox, Open hardware/software DIY 32bit console (you can buy it pre-made), 1MFlash/192k RAM/VGA/USB host/microSD , bare metal (no OS), 2d programming (some say racing the beam). Existing games, libraries, using gcc.

See http://bitboxconsole.tk / https://github.com/makapuf/bitbox

Not handheld but plugging a SPI TFT screen would be simple.