I think you should walk away for now. This, from a man who grew up under a single mother who put social life before kids. What did that get me? A string of jack a$$ men who were physically abusive to the both of us, a mom who turned to drugs and alcohol, and few fond memories of childhood.
Because you have a child, EVERY decision you make needs to be about what is in the best interest of that child. It didn’t ask to be born to a young mother. If there is even a sliver of a chance that the Ex could do anything to cause drama in the life of your child, then forget about it.
Spend your time working on making your life and the life of your child a better one, and good things and good guys will come!
Best wishes!