NEED DATING ADVICE (met a really nice guy) and self-esteem advice

I am 18 years old and haven't any real relationship experience. I've dated girls in the past and I haven't gone through any of the troubles I'm facing now because they were different circumstances. I wasn't sure of my sexuality then, it was high school, and sexual encounters were not something I had to deal with.

Since then, I had my first boyfriend I met at a track meet, first kiss, a complicated on and off relationship with him, and another guy where I didn't feel like we were a match. I'm in first year university now and I just met a guy while I was out on my own and we really hit it off. He's 22 (4 years older than me) but he looks really young and I thought we were maybe the same age when I met him. He is very sweet, kind, has similar interests, and seems like he genuinely likes me.

We went on our first date recently and I couldn't keep my eyes off him. After a long time out downtown, we went back to his place. It was not like, "Hey, do you wanna go back to my place and fool around?" When we were there we watched some Netflix and I snuggled up to him. Then after the first episode was over I looked over at him and kissed him. Then we made out and it was really nice. We got a little more intense with it but he said that if I felt uncomfortable at any time that I should say so. And he really meant it because he knew I wasn't that experienced and didn't want to take it to a place I was not comfortable with.

I won't go into too much detail but it went like that for a while and he made sure I was still comfortable. I'll get to the part where I need advice:

I ended up rubbing him through his underwear and I did see his stuff but I wasn't comfortable with him touching me. I felt like things were going too quickly and I was feeling self-conscious about my size.

He said several times throughout the day that I am cute, fit, and handsome. I'm hard on myself because I don't like hearing him say that because sometimes I don't feel like I am or that I deserve it. And so when it came to getting physical, I didn't feel adequate and I felt uncomfortable.

He was understanding and didn't want to push me so we just snuggled and I felt better after. Later on when we were walking to the bus stop which was maybe 10-20 minutes, he asked me how big I was. I felt uncomfortable again but he was saying that he didn't really care. He said there is "too big" and "too small" so somewhere around average. I don't think that I'm "too small" but I have had body image issues before (and still a bit now) so I have some anxiety over getting physical with him. I'd say I am on the lower side of average and it frustrates me (even though rationally I know that it shouldn't and that it does not matter but it really did frustrate me at the time and left me feeling very anxious).

The other thing is, how fast should I go with him? Especially because I haven't gone very far physically before. I have to admit that I was really horny but also really self-conscious and didn't want to go further because of that. I feel terrible right now. I know he'll be okay with it but I just feel very anxious, inadequate overall, and just unsure about it. I feel unsure about him but he is the best case scenario for dating so I know that there is clearly something wrong with my way of thinking.

I really need advice on this and any stories or experiences that you have had that could help. Thanks.