Going is your choice; but make sure you make it clear with him you are female. Sometimes men date us and they find it shocking, then lose all attraction or worse. As long as you are good terms about it, just make sure the number one rule........ Have fun! it's a date after all.
We need more information to answer this. Does he know that you are trans? Do you want to go on a date with him? Are you willing to tell him if he does not know?