Neighbors taking oranges off of my tree

I have an orange tree in the corner of my backyard. My wife said that she saw the neighbor who lives behind us use one of those fruit grabber poles to basically reach over the wall into our side and grab a bunch of them off the tree. We like to leave them on late into the season because they're sweeter when you pick them. I don't mind if they grab whatever is on their side or facing their yard. Of course it'd be nice if they asked. But am I right that reaching into our side isn't cool?

There were a few left on my side that's closer to the neighbor to the side, and he must've grabbed them all because they disappeared. I don't get why people think that's okay.