I get called gay but I'm not?

Basically I've had a crappy life. I was abused physically and emotionally as a little kid. This made me sensitive and I was taught to be nice and well mannered. Then from jr high on, I was labeled gay, fag etc and of course I live in a small town so people here are naturally ignorant. I grew up in a house full of women. I dated in high school but nothing very serious. I was always the quiet type and everybody has called me gay even people I never talk to like my neighbors. It's starting to really piss me off. They have no proof of anything they say. I will admit I went through a bi curious stage but now that I'm in college I am not any longer. I want a girlfriend but between the abuse from parents and bullying, I have no self confidence at all. I get depressed. I am a virgin. I have never been with a dude. It just frustrates me.