Or that they will leave them for another gender? A lesbian or gay man has exactly the same chance of leaving you for another man/woman as a bisexual, people also think that they get "straight passing privilege" which is complete BS because as long as they are not displaying PDA with their partner, I'd argue that nobody would even know they were lesbian/gay (unless it's at like, a gay pride parade or something)
It's largely a combination of many people believing that bisexuality is not real and is just something for "perverted people" to use to justify being perverted, and even many that do believe it's real think we're greedy for "not picking one" or think we have to have a partner of each gender at all times or some shit.
Most (if not all) prejudices, including this one, is a combination of ignorance and refusal to seek knowledge. People tend to think their opinions are absolute and unquestionable. Lots of people copy the prejudices from their parents/environment and then never question these concepts. In this case, people tend to think that bisexuals need to have relations with all genders to be satisfied, or that they are attracted to everyone in front of them. It simply doesn't make sense.