I think he cares. Otherwise he would not have been concerned about your text. The next time you hang out, leave without staying the night. Make up an excuse like you aren’t feeling well. If he calls the next day to see how you are feeling, then it’s all good.