Start lagging when playing with friends

Normally my lag is fine but as soon as I join a friend’s game I keep freezing and being forced into first person and the game is unplayable. I’m on xbox series s does anyone know why this happens and how to fix it?