The Skateboard roller as an idler is a good idea. They are usually much wider and built out of some pretty tough materials.. They also tend to come with double roller bearings to be able to support higher loads.
An Alternative option would be to ditch the cheap plastic wheels and go with an actual idler sprocket. You would have to build an appropriate spacer to set alignment, but this would allow better control over the chain, and thus allow for a looser tension to be run.
Examples:
This is the type often used on spring tensioners, but there is no reason it can't also be used in a fixed application.
Sprocket Idler 10 tooth 410H 415H 40 or 41 3/8" ID
www.staton-inc.com