I was on a pedal-only cycling forum before Yahoo Answers Cycling shut down; I used to think the same thing you're thinking. Once I started using Lane Control, mirrors, and being predictable. I find I'm in far less danger than I ever was riding to the far right in narrow lanes. Riding to the far right in narrow lanes encourages vehicles to a squeeze by on you. Using Lane Control causes vehicles to change lanes to pass like they'd have to do with any other slow-moving vehicle.
When I was a far-right only rider often vehicles would pull up beside me at STOP signs and red lights. Then cut me off with right turns. By using Lane Control my lane traffic is behind me and not beside me. Now, traffic stops behind me at STOP signs and red lights.
The incident you described is known in Cycling terms as a Right Hook. You probably wasn't seen right off or they thought you were a pedal-only bicycle and going much slower. Had you been using Lane Control the car would've been behind you instead of beside you at the intersection.