Pretty much stealth, now extremely ashamed of being trans?

Alright, so back when I started seriously transitioning- right around a year and a half ago- I was SO proud of being trans, and it was so important to me to claim that as a part of my identity. I was a real big activist, and all my friends were transgender too, and I felt no shame whatsoever. Now that I'm in the middle of my name change and on hormones and essentially stealth in my day to day life, I've found that I really don't want people to know that I'm trans anymore- I'm terrified that I'll be treated differently, and that people won't respect my identity as much (example- when I had to pay with a card instead of cash at a coffee shop I frequent and ended up outing myself to a barista I'm friendly with, he talked about how it was so interesting that I wasn't really male or female. Uh, no, I'm entirely a guy, you want someone nonbinary buddy). I'm far more interested in people knowing about my sexual orientation and seeing me as a guy who isn't straight rather than just a transgender guy. If anything, I want my gender identity to be on the backburner in people's perceptions of me, if not off the stove entirely so to speak. I feel bad for feeling so ashamed even though it makes a lot of sense but I can't help but feel like I'm abandoning where I came from.