If you want to see the patches go into the Debian/Ubuntu packages, run apt-get source php5 and look at the php5-$version/debian/patches directory. The build process uses a quilt wrapper (dh_quilt_patch) to apply them in the correct order. There are patches for building (in general or for a specific architecture), segmentation faults, security issues, manual pages and very few configuration changes (mostly just FHS compliance).
That's pretty much exactly what I try to do. I grab the sources for the package and upgrade the underlying php version. I usually have to grab the PHP sources from a newer version of Ubuntu and modify it until it builds.
With a bit of experimenting you can easily do a network install of a headless Fedora/SL/CentOS in KVM using virt-install and a kickstart file. Use filesystem passthrough to access your source files, run php-fpm and configure its address as a backend in your web server. Keep SELinux enabled, but adjust it to your needs using setsebool if necessary.
That first link looks like exactly what I need, thank you!
I've tried Fedora in the past. I found it unstable and broken and I don't like RPM very much. It's undoubtedly improved since I last tried it (c. 2005-2006) but I don't miss RPM at all.
