If in your state or part of the world sex with a minor includes anyone under the age of 21, then understand that he could get in huge trouble if the two of you start to have a relationship — or his behavior is considered sexual in nature, and yes, touching can be misconstrued. So if what he’s intending to do — which is to have a physical relationship and maybe more — is illegal in your state, stop doing it, and get support in stopping.
If it is legal, then you have every right to pursue a relationship with him. I assume you’re both single and he’s not your boss at work (which complicates things). The age difference you describe is only a problem when there are cultural differences or huge gaps in sophistication or experience or intelligence. But many relationships with a big age gap work out fine — and even great because the older man gets something genuine from a younger woman and vice versa.
I hope that answers your question for you! Please follow me @AskAprilcom on Twitter.