You need a completely different userland, compiled for x32 instead of amd64. Just running the default amd64 installation with a x32 kernel doesn't make it x32. EDIT: It would also work for just running x32 apps on an amd64 system, but then you only get the x32 benefits for those applications.
You don't need a whole x32 userland in order to be exploited. Just a single x32 binary executable that happens to not depend on on any external libraries.
I did't mean for an exploit, I just meant that you don't have any use for a x32 kernel on your system if you don't have x32 apps and libraries (unless you build a statically-linked x32 app).
I realized after writing my post that you don't really need an entire system, just the x32 libraries (and not even those if you statically link your x32 application).