Keeping Cats Out of neighbours garden?

PLEASE HELP! I really need to stop My cats going into my neighbours garden. the man has got rather nasty recently and had a go at me. I have spent quite a bit of money on those crystals you shake about the area you do not want the Cats to go. They do not work. My Cats still go over to the neighbours garden! I have thrown water at them when they go in there but this only deters them in that moment, not long term. I know that Cats find a place they like to go to the toilet, and keep going there. I have looked up many websites and read lots of things about this issue, but nobody can suggest anything. Please help me out!