Baby smells like someone else

Is it just me, or do other parents hate when their baby comes back to you smelling like someone else? I hate smelling other peoples perfume, cologne, etc on my baby boy. I’m fine with my husbands cologne or obviously my perfume, but it really bugs me to have him smelling like other people. I think i get why in nature, they say not to touch a baby fawn or else the mother will come back and abandon the baby. I obviously won’t abandon my son, but the mother deer probably doesn’t like the different smell on her baby lol. Am I the only parent who feels this way?