Ex with no boundries

You have to set the boundaries. Matter of opinion but don't you want to move on without having an ex keeping tabs on you. Divorce happens and we are supposed to be able to move on without them showing up hanging out as if they still have that right.  
My husband walked out on me leaving our home and getting a temporary rental place. I was/am devastated.
I told him the day he was moving some of his stuff out of the house that he does not have the right to enter this home whenever he chooses. That would be unfair to me. 
He has hidden his location from me and I let him know that since I am not allowed to enter his abode at any time - he's not allowed to do that to me. It's only fair in love and war.
He agreed. So far I am not aware of him entering while I am away. He calls first before stopping by and he knocks on the door when he arrives. I have not been to his location yet and won't.
I was very straight forward about setting the boundaries right away. At first because I was so hurt but later appreciate knowing he won't pop in at anytime without me knowing (even though I've wanted him back home with me).
So what if he gets upset. You are divorced. Let him know you have a right to privacy. You are divorced.
Good luck and I do hope you become strong enough to set the rules for your home. You deserve your privacy and to move on.

Sadly to say, you just have to not let him.  You let him come over when ever he wants and so therefore, he will keep doing it.  I'm assuming you already talked with him about it.  I'd just stand firm with him and tell him why you need him to give you notice or plan ahead of time.