AITA for allowing my ex-wife into my home?

Early yesterday evening my ex-wife came to pick up our son from my home. At the time my wife and I had a few visitors over for dinner. The visitors included her sister and husband, but was mostly friends.

Usually my ex phones our son when she's downstairs (we live in an apartment building) and he goes down to meet her. Yesterday she called our son as usual, but asked him to ask me if she could come up because one of her younger kids needed to use the bathroom.

I obviously said yes. She came up with her 3 younger kids. She didn't know we had company, so apologized for interrupting, before taking the kids to the bathroom. On her way out a couple of the guests stopped her for a quick chat (friends I've had since she and I were married) but it was only a few minutes. My son gave his little siblings some of the snacks that were out for guests.

After they left my wife's sister was quick to call my ex rude for barging in. One of my friends replied to her saying something about small kids bladders and nothing more was said about it.

However, after everyone but my wife's sister had left, my wife and her sister started talking about it and both agreed that it was rude and tacky. They couldn't believed she started talking to some of our friends. When I objected they said it was wrong of me to have let her in without asking my wife first, I said that's ridiculous.

Later, my wife and I discussed it again and she said she found it embarrassing for me to just let her in like that. I asked why, but she said I wouldn't understand.

Am I really the asshole here? I was trying to do a normal nice thing but now I feel bad about upsetting my wife.