Eh, I'm not sure where you're getting your numbers. Here's a quick test using node 0.10.40 on my machine:

  mmalone$ node
  > var f = function() {
  ... s = Date.now()
  ... for (var i = 0; i < 1000000; i++) {Math.random()}
  ... return Date.now() - s
  ... }
  undefined
  > f()
  3
  > f()
  1
  > f()
  1
  >

So I'm getting a couple milliseconds for 1,000,000 numbers. I'm also toying around with a PRNG package in Go currently so I have some numbers there too:

  BenchmarkXorshift128p-8                  	500000000	         2.98 ns/op
  BenchmarkXorshift1024s-8                 	300000000	         4.62 ns/op
  BenchmarkXorshift4096s-8                 	300000000	         4.44 ns/op
  BenchmarkXorshift4096ss-8                	200000000	         6.29 ns/op
  BenchmarkUint64sFromCryptoRandIdiomatic-8	  200000	     10736 ns/op
  BenchmarkUint64sFromCryptoRandUnsafe-8   	  200000	      8081 ns/op

Go's crypto source pulls from urandom. This is more like what I'd expect in terms of performance. The difference is orders of magnitude, but we're still talking about microseconds so I think your argument is still valid. A proper CSPRNG is not "as fast" but it is probably "fast enough" for most use cases.