IMO all drivers need to be Free Software. The problem is that as soon as you introduce loadable modules into a kernel, there's really no way to prevent them because of the precedent set by Linux. One of the reasons the LGPL was developed was the linking issue, but Linux developers seem to be ok with letting binary blob modules as a "gray area": http://kerneltrap.org/node/1735
If Linux developers had from the start demanded Free Software drivers, they'd have had them (please spare me the arguments about "Linux wouldn't have grown if they didn't allow binary drivers"). Letting that precedent pass means that hardware developers don't have many incentives to provide sources for their drivers. This affects not just Linux, but all Free Software operating systems.
What incentive would hardware venders have to provide open source drivers for a tiny enthusiast operating system? No other OSs have had that requirement. It's hard enough to get working binary drivers.
I don't think that all the Linux driver issues would be solved with the release of source code. Driver code already makes up a large majority of the kernel. The sheer amount of code contained in binary drivers would dilute the already relatively small number of volunteers available. The best way to increase the quality of Linux drivers is to get venders to invest more in Linux driver development. The only way to do that is to vote with your wallets by supporting Linux friendly hardware, and increasing the size of the Linux market.
OpenBSD has always had a policy against binary drivers.
The reason it's "hard to get binary drivers" is because hardware manufacturers have no incentive to release documentation for their hardware, because they know they can get away with a binary driver for Linux. Then they call themselves "Linux friendly" and people like you get the wool pulled over your eyes (until you encounter a bug, or work on the kernel, or want to use FreeBSD, or suddenly find that your hardware isn't supported on Linux 2.6.xyz, etc, etc)
Even if they were to write Free Software drivers themselves, it's likely they wouldn't make it into Linux anyway because of quality issues (most driver code is shit). Community-developed drivers are frequently better quality.
Arguing that Free Software drivers would be "too much code" in Linux is just idiotic. Let's just stop writing Free Software altogether because it's "too much code."
I would guess that it's roughly in the same ballpark as non GPL software being able to run on Linux. Just happens to be ring0 rather than ring3.
I don't think it is realistic to expect all drivers to be Free Software especially specifically distributed under GPL. A huge number of Linux drivers are GPL anyway.
What makes you so confident that Linux hackers would have managed to reverse engineer all nVidia's stuff to the point of having a good stable performant driver? Especially considering nVidia release new cards with new capabilities (thus requiring driver modifications) very frequently.
If nVidia had decided that they would release no free driver and therefor no driver at all, I don't see the business case for somebody being paid to develop a free software driver for nVidia cards and it's certainly not a weekend project.
The problem is that all the ARM-system chipsets are shipping binary-only drivers for Android and OMAP devices, even though Linux is used almost exclusively on those devices.
That's the business case.
nVidia may not care about Linux because it's a tiny portion of their market, but chip manufacturers (like PowerVR, for example) would have had to release Free Software drivers if Linux hadn't set the binary precedent. This would also be true for a lot of other hardware now. But unless there's a radical change in policy, it's too late.
What's the point of having a Free Software OS if you can't run it on your hardware because the closed-source drivers won't work?
If Linux developers had from the start demanded Free Software drivers, they'd have had them (please spare me the arguments about "Linux wouldn't have grown if they didn't allow binary drivers"). Letting that precedent pass means that hardware developers don't have many incentives to provide sources for their drivers. This affects not just Linux, but all Free Software operating systems.