Well, first of all, since you are partners, you have the right to ask and know where he is going. And if he truly has nothing to hide, he would let you touch or see his phone.
I’m not saying for sure that there is something going on, but we also can’t be certain that there isn’t. It would be better to have a serious, honest conversation with him about this issue so things can become clearer for you and so your worries can be reduced.