Breaking with your family is something really heavy and mostly indefinite, so don't break if it is not necessary. but if associating with your family brings pain and discomfort you can try to take some distance first and if that doesn't help then you can break with them