Why do so many atheists seem to have a problem with Catholics?

Why is there so much hate against Catholics? I mean, everyone seems to hate us, Atheists, Protestants, Homosexuals. I am Catholic, and recently a very good homosexual friend of mine told me he hated Catholics, and I noticed on YA that everyone seems to agree with him, I mean, really, you can't say that you hate a group of millions of people because of their religion, it's racism! I understand that there has been a big scandal about Catholic Priests recently, but that was a minority of Priests, and their actions cannot define a whole group of other Priests and Catholic non-priests. Also, why can't Atheists just leave us alone, I don't go around telling the world that they are 'idiots' because they are not Catholic, because that's not true, so I find it very offensive when people say that about me, in fact, I recently read on a YA an answer stating that "all Catholics are uneducated, ignorant, sexist pigs' If any other group of people were treated like this because of their religion it would be called racism, so why do people feel it is okay to be so mean to Catholics?