It sounds like you have a conflict with your family. Just because your family is famous doesn’t mean you can’t be with someone from a non-famous family. But in your case, you say that that is a problem. And even your boyfriend didn’t want you to tell anyone the two of you were dating. Before you get into another relationship with anyone, you should be clear in your own head whether you are willing to be honest with your family and date whoever you want as long as they are a good person, or whether you prefer to abide by your family’s wishes and date someone who fits in with your family. If you don’t get clear on this issue, you’re going to bring conflict to whatever relationship you have with anyone you date from a “regular” family.
So figure that out first.
Next, your boyfriend was hurt because you broke up with him due to class reasons. He wasn’t from the right class, and you succumbed to family pressure to break up with him. He didn’t do anything wrong. He was just “from the wrong side of the tracks.” He has every right to feel hurt and wonder if this will come up again with you if he goes back with you.
It sounds like you are lonely since the break up, and when he called, he stirred up old memories. But getting back together with him will lead you right back to family problems again. So don’t get back together with this guy now. Figure out what part your family plays in your personal life, and then be true to your decision.