Is eating food at table while wearing a hat being rude?

I’ve always been taught to remove my hat while at the table for meals but often see lots of people still wear them.