You’re definitely not overreacting. Your neighbor is making a play for your husband, and your husband appears to be crossing appropriate boundaries. I’m not sure if he’s having sex with her, but he’s not being loyal to you.
My suggestion is that the two of you stop seeing these neighbors altogether. It doesn’t matter what he says to end it, he just has to end it.