It’s not silly at all. When someone you love keeps pulling away, especially on a day that matters to you, it hits deeper than just “no sex.” It feels like rejection, even if he doesn’t mean it that way.
But here’s the thing wanting him doesn’t fix whatever’s going on in his head. And pushing, even gently, only makes him shut down more. You’ve already seen that.
The part that worries me is how lonely you sound inside your own relationship. Four years in, you shouldn’t feel like you’re begging for scraps of affection.
You’re not wrong for feeling hurt. You’re human.
Just don’t turn it into a fight. Tell him the truth not about the sex, but about how distant it’s making you feel. If he loves you like he says, he needs to meet you halfway.