The only time Louis reacted to someone's criticism