Confusion with an ex

I should probably start at the very beginning. I dated this guy a while back who I met through a mutual friend. After we met, we became friends on facebook and after a long messaging conversation he eventually asked me out. We messaged pretty constantly leading up to our first date, and in between the first and the second. I figured things were going pretty well when he mentioned he was having doubts about us. He decided he didn't want to date. I felt pretty crappy but it was amicable, and we decided to remain friends,and gave ourselves a few months without contact to deal with things outside of a few meet ups with other friends.

Flash forward a few months, and we get back in touch. We're starting to settle into a friendship, and him buying me a gazillion books for my birthday, when we end up treading the territory of our physical relationship ending up with him talking about he enjoyed touching me. This was when I asked what we're doing, and we ended up having a painful conversation about why we broke up. My feelings had come to the fore again, but he told me he just felt that our relationship just wasn't crazy enough for him. Again, it hurt but I was just glad it hadn't ruined our friendship. We sweeped it under the rug and moved on.

Then a few weeks later, we arrange to meet up, having not seen each other in a while. To my surprise it ends up that he's inviting me alone, which I thought was a bit odd considering how recent our last awkward moment was but I took it as a sign that I hadn't damaged our friendship and that he was comfortable with me. Then it turned that he was housesitting, which meant we would be alone in an apartment. Anyways, we met up, spent some time wandering around in the rain, then we got to the apartment, and he was busy for a half an hour setting up his new phone. Then we played some games and then watched some movies. During the second one he ended up leaning on my shoulder, and before long I was resting my head on his head, and we spent half an hour brushing our fingers against each other before we made firm contact. We ended up making out and he ended up stripping me down until I was very nearly naked. I had to go because I had to catch a bus, but while he was walking me down he suggested I stay over but I couldn't.

I messaged him afterwards and he said he was confused about what happened but said he doesn't think he wanted to do what he did and that he lost control. And he couldn't do physical things with me because he felt bad afterwards, that he was afraid of how he acted around me. I asked if he wanted us to forget about it and go back to normal. He saw the message but didn't answer. I waited until the next day and when he still hadn't answered, I explained how I felt confused and how the whole thing had sent me very mixed messages about us, with his change of heart hours after trying to get me to stay. He saw that message too and didn't reply. I've spent the the past few days waiting for him to respond but he hasn't, and I'm wondering what to do. How long should I leave it before trying to contact him again? If there's a point where I should try to take charge of the situation, and do I try to figure things out with him in person? Any advice on how to approach this would be much appreciated.