Should I just end things?

There's this guy I've been seeing for about the past 9 months. We started off just as fuck buddies and then things began to develop more towards a relationship. During this time though he broke things off once by saying that he didn't want a relationship with me because he just couldn't picture himself being with me. While it hurt I accepted it and we continued doing our normal thing of having sex and being intimate such as dates and the such without actually being boyfriends. Well he decided to break things off once again later and then we ended up going back to where we were again and finally around 3 weeks ago he ended things once again after telling me that he didn't feel comfortable continuing since he would never be able to bring himself to introduce me to his family because of the fact they would judge me for my background which they knew about (mom is an alcoholic, dad was never around) even though I came out the exact opposite of what would normally be expected of a child in that kind of situation seeing as how I've moved out, have my own car and job, and have cut ties with my family as they were doing nothing but trying to impede my progress with my education and life in general.
So said guy has since told me that during time we were together he didn't always enjoy me as a person but rather just the feeling of being with me sexually and the feeling of being able to be intimate and cuddle and go on dates and say sweet things etc. After being told the week before that I basically am his ditry little secret and who's never be introduced to his parents even as just a friend and now being told this I lashed out and called him out on everything. After which I was told I was bring manipulative and that I only focused on the bad things he said (which I did as I saw nothing good come from that conversation) and that I have everything wrong. With that in mind we just got over having a huge fight because he gave me the option of trying a friendship or just keeping sex without the friendship and nothing more. I chose the sex because I figured there was no way I could be friends with someone who until recently only viewed me as a sex toy. After asking him if he could be friend with someone after they told him something like that he blew up and we basically fought until we decided that it would be best if we meet to talk things out and then take a month break or so apart from each other before attempting a friendship. I honestly don't see what taking a break from each other is going to solve but I agreed to it just to be able to see him again finally.

I'm now confused on what to do when I do see him. One part of me really just wants to give up and remove him from my life entirely but another part wants to try the friendship since maybe, just maybe since we started things out with sex and then tried to move towards a relationship maybe if we were able to become friends it could lead to it. I can picture us being together happily if he weren't so against the fact that he isn't going to introduce me to his family (they play a huge role in his life as his parents are rich and he still depends on them) and I know that if they were able to accept me that it would be a big step forward. Other than the recent events between us things have been perfect. We both enjoy each other's company and we both share the same general life goals not to mention both agreeing that we doubt we'd ever meet someone else we're as physically compatible with as we are with each other.

So what do you guys think? Should I try the friendship or would it be best to just break things off here?

 

So I'm not crazy as he's making me out to be that he's treating me poorly?