> if you are only using the NT native API then it should work just fine
I believe I had trouble executing any instructions in the new process at all. If you can make it work I'd like to see your code, otherwise I'm skeptical.
AFAIK this is how fork() is implemented in SUA, but I haven't tried it myself. You could load the SUA subsystem dll into IDA Pro and see how they actually do it.
I believe the POSIX subsystem has some kind of support from the native API that prevents you from writing your own random subsystem and expecting it to work; I don't remember what it was though (it's been a few years).
There's some special handling when loading the image where it does different things depending on the value of the subsystem field, but I don't know whether there's any special handling in the kernel that can't be duplicated by using the native api. But the NT kernel does very little when it comes to initializing new processes, most of the initialization is done in user space by ntdll, so it seems unlikely.
> had problems with getting it to work with the win32 subsystem
Keep in mind that the whole _point_ of Cygwin is to interact with the Windows world. If you want a POSIX sandbox, you can use a VM with fewer headaches and better performance.
I believe I had trouble executing any instructions in the new process at all. If you can make it work I'd like to see your code, otherwise I'm skeptical.