Baby cries at all strangers

My baby is 5.5 months old, from the time he was around 3 months he cries at all strangers who try to hold him, he can watch others from me and my husband laps but that is it, he wont tolerate any other people, thing is since he started to understand his surroundings weather got very bad where we are and we remained at home mostly and did not socialize much, but I am very concerned that something is wrong with him