Sometimes I get aroused randomly. Sometimes that happens with random people. It's annoying and it makes me feel confused.
I have almost the opposite problem. I rarely ever get aroused, so I make myself believe I'm attracted to others when really, I just like their outfit or am jealous of their hair. It's like most of the time I forget what attraction feels like, so I take what I'm experiencing and make it fit the mold. And that's confusing. It's as though I'm looking at all the freckles on my body thinking they're melanoma, except no one's going to tell me to chill about this one.
I have the same problem as you. A lot of times I look at a slightly attractive girl and a voice in my head tells me that I'm totally attracted to her. I don't feel arousal at first, but I might start getting false attraction. This can happen with guys as well. I rarely feel real attraction and I don't really remember what it feels like.