What are your thoughts on non muslims claim: "If religion makes you a better person then you aren't actually a good person"?

Ive seen this claim thrown around quite a lot over the years. The main focus of the people who claim this is that they think we're only "good" towards others because if we aren't we're scared of God punishing us, which in turn means we're actually only doing everything as to benefit ourselves, and it has noting to do with the other person so in the end we don't actually have a good morality towards other people, nor any empathy for them and are just self absorbed and only care about ourselves. What do you think about this?