Woah that would be cool. So the red would be invisible to the blue/green and visa versa? Seems like that could be easy to add....

Not OP here: Yes that would definitely work, but you have to be aware that color channels of the monitor and camera are not mapped 1:1, there will be some overlap.

Also be aware the light in the environment influences the light received by the camera (or emitted by the monitor surface, depending on the way you want to look at it). You may have to tweak the threshold if the light changes.

@OP: please do this, it will triple your data without additional costs. Don't ignore the color channels!

It shouldn't be so difficult, because you only have to recognise 8 distinct colours and thus separate out the bits accordingly:

   RGB
   000 - black
   001 - blue
   010 - green
   011 - cyan
   100 - red
   101 - magenta
   110 - yellow
   111 - white

I think that even using more than one bit per RGB channel isn't hard. For example, 0 -64-128-192-255 for each RGB would increase the speed for 15 times. The 1-2kbps would become 30kbps, a modem from few years ago.

When using the real webcam (and not just passing files around), just one frame for calibration and no auto-white balance should be enough.