Husband upset I didn’t invite his family to my birthday celebration

My husband and I just got married last year, a month later after our wedding it was my birthday. We invited my family and my in laws to celebrate my birthday at our place. This year for my birthday my parents suggested celebrating my birthday, just us (parents, siblings, brother in law and husband) as it would be something very small and intimate. I didn’t argue against it because I was okay with it. (Either way I was stuck between wanting everyone (extended family and in-laws) to go or if just us. We have never celebrated birthdays just us (or maybe it’s been years since we last did). A couple days before my birthday I told my husband it would just be us, he was okay with it. The day came, my mom made the food of my choice and we had a small cake.

Now, today is my sisters birthday. My husband asked if he should invite my inlaws, and I said no because it’s just us (as it was a last minute thing and my brother in law and I would make my sister a cake. I don’t even think anyone is cooking a meal). Plus, it’s my sisters birthday, why would she want MY IN-LAWS in her birthday when she doesn’t even want her own in-laws and again, it’s something super small. Well my husband got upset and told me that I never want to invite his family for stuff. Keep in mind that throughout the years of us dating and being engaged he never once thought of inviting my family either and I never questioned it. Now he is upset that didn’t invite them to my birthday celebration last month. He said “they are your family too. You never invite them for anything.” I reminded him that we had them over last year for my birthday and that I suggested we invite them to come to our place on weekends. But he said he means for birthdays or events, I told him how he never invited my family either like ever, and now he says I’m bring that up as a sort of revenge and that it was different cause we weren’t married.

Do you guys invite your in-laws to every single event/celebration/ or to eat birthday of yours? Am in the wrong for not having invited my in laws for my birthday celebration?

EDIT: I’d like to mention that before we were married, my in-laws sometimes celebrated my birthday and did something small. Usually just cake.

It was my dad’s idea to celebrate my birthday, my mom hosted it, and it took place at my sister’s apartment.

This year we celebrated my husband’s birthday at our place, and we invited both our families.

Yes I get along with my in-laws but have had a rocky past especially with his mom and brother.

Christmas is celebrated separately, Christmas Eve with my family (if you’re Hispanic you know many Hispanics celebrate it on the 24th) and Christmas Day with his family as they celebrate it on the 25th. We have agreed to celebrate New Years Eve each year alternating families, last year it was with his family and this year will be with my family.

My family usually sticks to themselves as his family does too. Our families do get along btw.

We try to see my in-laws every weekend, they live an hour away from us.