Tonight, we turn it ALL off
It all starts at 10pm Saturday night. They want ALL servers, and I do mean ALL turned off in our datacenter.
Apparently, this extremely forward-thinking company who's entire job is helping protect in the cyber arena didn't have the foresight to make our datacenter unable to move to some alternative power source.
So when we were told by the building team we lease from they have to turn off the power to make a change to the building, we were told to turn off all the servers.
40+ system admins/dba's/app devs will all be here shortly to start this.
How will it turn out? Who even knows. My guess is the shutdown will be just fine, its the startup on Sunday that will be the interesting part.
Am I venting? Kinda.
Am I commiserating? Kinda.
Am I just telling this story starting before it starts happening? Yeah that mostly. More I am just telling the story before it happens.
Should be fun, and maybe flawless execution will happen tonight and tomorrow, and I can laugh at this post when I stumble across it again sometime in the future.
EDIT 1(Sat 11PM): We are seeing weird issues on shutdown of esxi hosted VMs where the guest shutdown isn't working correctly, and the host hangs in a weird state. Or we are finding the VM is already shutdown but none of us (the ones who should shut it down) did it.
EDIT 2(Sun 3AM): I left at 3AM, a few more were still back, but they were thinking 10 more mins and they would leave too. But the shutdown was strange enough, we shall see how startup goes.
EDIT 3(Sun 8AM): Up and ready for when I get the phone call to come on in and get things running again. While I enjoy these espresso shots at my local Starbies, a few answers for a lot of the common things in the comments:
Thank you everyone for your support, I figured this would be intresting to post, I didn't expect this much support, you all are very kind
We do have UPS and even a diesel generator onsite, but we were told from much higher up "Not an option, turn it all off". This job is actually very good, but also has plenty of bureaucracy and red tape. So at some point, even if you disagree that is how it has to be handled, you show up Saturday night to shut it down anyway.
40+ is very likely too many people, but again, bureaucracy and red tape.
I will provide more updates as I get them. But first we have to get the internet up in the office...
EDIT 4(Sun 10:30AM): Apparently the power up procedures are not going very well in the datacenter, my equipment is unplugged thankfully and we are still standing by for the green light to come in.
EDIT 5(Sun 1:15PM): Greenlight to begin the startup process (I am posting this around 12:15pm as once I go in, no internet for a while). What is also crazy is I was told our datacenter AC stayed on the whole time. Meaning, we have things setup to keep all of that powered, but not the actual equipment, which begs a lot of questions I feel.
EDIT 6 (Sun 7:00PM): Most everyone is still here, there have been hiccups as expected. Even with some of my gear, but not because the procedures are wrong, but things just aren't quite "right" lots of T/S trying to find and fix root causes, its feeling like a long night.
EDIT 7 (Sun 8:30PM): This is looking wrapped up. I am still here for a little longer, last guy on the team in case some "oh crap" is found, but that looks unlikely. I think we made it. A few network gremlins for sure, and it was almost the fault of DNS, but thankfully it worked eventually, so I can't check "It was always DNS" off my bingo card. Spinning drives all came up without issue, and all my stuff took a little bit more massaging to work around the network problems, but came up and has been great since. The great news is I am off tommorow, living that Tue-Fri 10 hours a workday life, so Mondays are a treat. Hopefully the rest of my team feels the same way about their Monday.
EDIT 8 (Tue 11:45AM): Monday was a great day. I was off and got no phone calls, nor did I come in to a bunch of emails that stuff was broken. We are fixing a few things to make the process more bullet proof with our stuff, and then on a much wider scale, tell the bosses, in After Action Reports what should be fixed. I do appreciate all of the help, and my favorite comment and has been passed to my bosses is
"You all don't have a datacenter, you have a server room"
That comment is exactly right. There is no reason we should not be able to do a lot of the suggestions here, A/B power, run the generator, have UPS who's batteries can be pulled out but power stays up, and even more to make this a real data center.
Lastly, I sincerely thank all of you who were in here supporting and critiquing things. It was very encouraging, and I can't wait to look back at this post sometime in the future and realize the internet isn't always just a toxic waste dump. Keep fighting the good fight out there y'all!