Has the best ROI always been the S&P 500

Over my 40 years of investing, I would often hear mutual funds, real estate or other investments compare to the S&P 500.

If that's considered the "gold standard" for long term investments, is there a reason to choose anything else? I understand diversification but it seems semi built in when we're talking 20+ years or more.