The chain needs to be snug. I noticed my chain stretched quite a bit at first. If you can't take up the slack with the play in the drop outs I would leave the tensioner where it is.
I would hate to hear about something bad happening after taking that thing off.
Regards, Scottco
