Your lover’s behavior is hot and cold because your relationship is hot and cold. Your flirting is hot and your acknowledgement that you’re both married to other people makes it cold. That’s the nature of what you have, have always had, and will probably continue to have.
He’s stopped e-mailing you because the two of you agreed not to rendezvous any more. He’s kicking your chair and staring at you because he misses not having those rendezvous’.
It’s really pretty simple.
The ball is in your court and you have to decide if you want to continue the status quo, break off the office flirtation and affair as it is, or dive further into what you have together, risking your marriage, and maybe his, too. Your call.