Just because your ex-boyfriend is over you, married with children and wants to be friends does not mean it’s right for you to do so, too. In fact, my advice is to forget the friendship with this guy because it’s upsetting you and interfering in you marriage. You need to prioritize, and make your husband the most important man in your life. Just cut it off with your ex-boyfriend. You don’t owe him an explanation. Just be too busy to respond to his calls and internet communication. He’ll get the message.
The more contact you have with him, the more you’ll fantasize, so let go of your fantasies that are interfering with your marriage and focus on what you do have and what is real. Don’t have any more contact with your ex. It’s not worth your emotional pain.