Exactly. In fact Mantle/Vulkan/Metal/DX12 is based on the realization that with full GPU MMUs user space can only crash its own context, so it's safe to give access to console like APIs on desktop systems, and only have the kernel driver mediate user space as much as the kernel mediates access to the CPU.