Can I Semi-Retire

Hey guys! I'm a 35-year-old male with ~$400,000 in VFV.TO in an RRSP. I currently work a FIFO job at a mine with an expected lifespan of 3 years remaining. I also have around 5 years left on my mortgage which is my only debt. I should receive roughly $100,000 in severance when this place closes. My plan is to do something chill like drive a school bus when I'm done here, or some other casual job where I can golf all summer and be off for holidays, etc.

My questions are: should my portfolio be more diversified, and is $500,000 enough at age 38 to stop saving for retirement? My wife makes around $90,000 a year, and we live a modest lifestyle, which I would continue in retirement.

Cheers!

Edit: Thanks for all the great info guys! I have lots of homework to do! I have a couple of points to add:
I have medical coverage for life through the military. I'm a licensed Electrician so if need be I could always do side jobs for extra cash! Also I have a unique skill set that would allow me to return to the mining sector very easily at a high pay rate. Additional, I spent 10 years in a "hard sea trade" in the CF and the last 6 years working a mile underground, driving a bus seems chill! Thanks again everyone!