Nicely done. I wonder: what is the use case that prompted this investigation? 40-50 milliseconds is correct in terms of latency that musicians notice. If you're aiming at any realtime musical collaboration over the internet, the Big Problem is network latency. Unpredictable and just too long. And, if you're not working over the internet, why use a browser?

Edit: https://hello-magenta.glitch.me seems to be the use case. Cool and inevitable that this would be worked-on, but after spending years with both algorithmically-supported composition and all-human composition, I'm skeptical. There is a special sauce that machines will never grok. Or, I'm wrong.