A charity has been forced to clarify that it is not a biker gang after fears were raised by the public at a Welshpool council meeting.