I do not know about the 3DO but I would think the start point of the display would be the top right corner of the screen as that is usually where the start of each video frame is drawn from on a display. With the controller port having a CLK, In and Out data lines communication is probably operating as an SPI bus or something very similar. You may not be able to find any publicly available information on it as I am sure that is protected IP but the most likely options are outlined below.
Let us assume that a read of the controller is performed once per frame and the read also resets the light gun timer.
When the trigger is pressed on the screen designed for it and a hit is registered the timer is stopped, and its value is outputted to the console. A 17 bit value seems like an odd one but I guess it may depend on the screen resolution, I suspect your value relates to the XY position in one of two ways...
The XY coordinates are calculated within the light gun and combined to create the returned value. You just have figure out the order and where the split is, as the X coordinates can greater than the Y if it is 17 bits it would make sense for Y to be the least significant byte and X to be the remainder as it may require two bytes to store it with it being a greater value.
It is quicker than method two as you can simple mask off the unwanted bits to obtain either value.
Lets say your screen is 320 x 200 pixels, maximum value (assuming one count per pixel) = 64000
You then enter a loop then that subtracts 320 from the count while it is greater than 319 (first x coordinate = 0) while recording the number of times that it has done so, that gives the Y coordinate in Lines.
The remining value then gives you the X coordinate along the line from the Left of the screen, a bit like this...
'Loop to get Y Coordinate in lines from top of screen
While Count > 319 then
Y = Y + 1 'Increment line count
Count = Count - 320 'Decrement line value from count
'Get X coordinate along the line from the remaining value
X = Count