Airpods Pro Not Working Well While Only 1 Plugged

Hey everyone, I hope everyone is well.

I have Airpods Pro and when one of the airpods is in my ear and the other is out, phone always tries to connect one which is not in my ear. When I change rooms in my home, I can’t use the Airpods in my right ear. I have to go out of limit of Airpods or set the Airpods inside to their box.

Does anyone have this problem too?