Damn Jim stealing my thoughts again!
"You have his phone number so for one reason or another he'd like to hear from you." Yeah, you have his digits, call'em up the next day or something. If you call too soon it will seem clingy and somewhat emotional. Call him though, see if he'd like to go out again. If the sex was good on his end, undoubtedly it'll go further. If he made that mental connection with you, it'll go even further. You want to call to see if you're not just another booty call. Seriously.
Don't put too much weight into it without finding out the truth. Or thinking, "what if"'s because that will hurt only thrice as bad. We're boys, we do impulsive things and have physical desires that overrule our logic and morality.
Don't ever forget that, no matter what happens.