A permanent magnet DC motor can be treated as a voltage source in series with a resistor. The smaller the wire used in the motor, the higher the resistance of the wire, and the greater the resistance of that 'series' resistor.
When you draw current from the generator, the greater the current, the greater the voltage loss across the internal series resistance. Adding the tail light increases the current drawn from the generator, and when it did, the the voltage across the internal resistance increased as well. That, in turn reduces the available voltage to the load.
You can calculate the internal resistance with three simple measurements. First, at a fixed generator RPM, measure the voltage output from the generator, with no bulb or headlight connected. Then, connect the headlight, and take the same voltage measurement, across the headlight terminals. Finally, at the same RPM, measure the DC current through the headlamp. To calculate the internal resistance of the motor-generator, in ohms, take the the difference between the two voltages, and divide that by the lamp current in amps.
A 55W headlight SHOULD pull about 4.5 amps. If we assume that that was the actual measured current with your test, (and with the voltages you mention, above) the internal resistance would be
(12V-9.8V) / 4.5A = 0.489 Ohm
The power loss would be
4.5A * (12-9.8) = 9.9 W
The headlight power would be
9.8V * 4.5A = 44.1 W
The power lost divided by the total is 9.9 / 55, or 18%
You could increase the RPMs of the motor, so that it produces about 20% more voltage, unloaded, to compensate for the power lost to heat with the series resistance of the wire. You can also increase the RPM to compensate for increased current load. But, there comes a point where the internal power loss due to heat will slag the motor/generator, or the increased voltage would short out windings in the motor.