I play a lot of casual games on my phone. Pretty much the casualest casual player that casualed. I never think it's worth it to call myself a gamer. Do you think it's worth it?
I would say its up to you... There is no "official"criteria for being a gamer, it just means that you enjoy playing video games regardless of the medium that you choose to play them on. (phone, consoles, computer, etc.)
People say "Pff... filthy casuals" as a joke, but you can probably call yourself that without really having people mock you. None of their business what you enjoy.