I think I’ve made a mistake speeding through the storyline

I am currently in Year 4 and suddenly am realizing that the TLSQ may be something I need to pay more attention to. I ignored almost all side quests and quidditch and only played the main storyline up until I recently. Will quests missed in the first 3 years come back? Is there an advantage to ignoring the storyline and working on side quests?