This is bothering you because your boyfriend is not treating you like you’re the most important woman in his life — one that he would do anything (rational) to make happy. It doesn’t matter if he won’t cheat on you, sexually, with his ex. The reality is that he’s giving her a spot as a woman in his life, that makes you feel less than important.
This isn’t about cheating, sexually. It’s about his making you feel like number one. And after a year and a half of dating, you’re not wrong to want and expect that from someone you’re investing in.
Your instincts are correct that this may end up being a deal breaker. Picture a future together where you’re feeling put upon by children and work and the ups and downs of marriage — and he’s lunching with his ex, putting his mother ahead of you, and taking his sister’s calls before yours. I know that’s just a possible scenario where he puts other women ahead of you in his life, but if he doesn’t understand your feelings and accept your needs — which I think are reasonable — then this will become a deal breaker down the line and you’re not wrong to deal with it now to protect yourself from wasting your time with someone who isn’t Mr. Right, as much as you want to believe he is.
If he wants to see his ex-girlfriend as a foursome with you and the ex-girlfriend’s new boyfriend, every few months, then that might be a compromise. But if he’s really looking to keep her as a confidante and special friend, you have an incompatibility in your relationship socially and emotionally.