How do you handle people that are clearly disturbed?

We have neighbor's that seem to think that it is our obligation to let their children swim in our pool. We can rarely sit around our pool without these kids comming over and begging to swim. We are very uncomfortable sitting in our own backyard. Unfortunately, the parents exhibit a total lack of personal boundaries. There is just no consideration from their end especially since they ENCOURAGE their children to ask if they can swim! When they ask to swim we politely tell them that it isn't a good time etc. but they just don't get it. They have even called our house and asked if the kids can swim. They are clearly not all there and they don't understand the word NO or the concept that it's impolite to invite yourself ANYWHERE. We can continue to say no but it doesn't stop them from allowing their children to stand there and watch us trying to relax! Now that Summer is around the corner I cringe at the thought of what is to come. We have never seen them socially but they just assume that what is ours should be shared with them. We try to ignore them but it's impossible. The children ring our doorbell every sunny Summer day. What else can we do but pray for rain?