You'll find, too, that to achieve proper alignment, you'll probably need to twist the tensioner until the roller is parallel to the chain. It also needs to be well centred, so that it guides the chain onto the centre of the rear sprocket as closely as possible. Make sure the tensioner clamp bracket is very tight and cannot move and keep the chain well adjusted, with no more than 10-12mm slack at it's tightest point.
Once all of that's done, cross your fingers and wear that helmet. I never felt comfortable with the tensioner and rag-joint sprocket. An accident waiting to happen.
A 'Manic Mechanic' sprocket and adaptor is one answer, a shift kit is another, yet another is a tensioner mounted to two points on the frame, so it can't move inwards toward the spokes.
If the chain is climbing the sprocket teeth and coming off, it'll take more than a piece of plastic to stop it.
... Steve