I don't know how the light gun worked, but the vector display used was pretty unusual. The 'Typotron' (a.k.a. 'Charactron') could display 25,000 characters per second. It didn't draw them in the usual fashion of a vector display. Instead it displayed them by deflecting the electron beam at the right location in a stencil, then deflecting it again to the right spot on the screen!
The Typotron was like a Charactron, but could store data, too. Apparantly the light gun was used only to click on light points, not on dark bits of the tube, so it was 'trivial' to determine what was clicked on (the second page of the PDF talks about how the light gun was basically invented and prototyped in one day somewhere in '48 or '49).
It probably wasn't that complicated. The light gun was used as a quicker means of selecting an object being drawn on the display. It doesn't seem to have ever been used to select or draw things in blank space.
Since a vector display always knows what object it is drawing a particular time, the presence of light at the tip of the gun is all you need to identify the object.
