This has been a great community to me since I showed up around Sept 2012. I've done good by my customers ensuring a quality service and I've been proud about that. This is not a letter about me closing up shop though the obstacles I'm facing are pretty steep.
On 3/7 around 2:30PM MST Data102 was doing some preventive maintenance on their UPS systems. While a official RFO will come in the morning from them to me, but the gist of it was an internal piece of the UPS failed and when they tried to put the load on the UPS it shut off completely. This was after they had the system in by pass for preventive maintenance to replace a logic board that had nothing to do with the other part.
So far the damage is the following nodes have died (Failed Raid Cards)
Pike - KVM - Failed Raid Card
Breck - KVM - Failed Raid Card
Kenobi - OpenVZ - might be fixable just not sure.
Webserver - the boot partition appears to be corrupted, we hope to recover the database off of it and rebuild WHMCS sometime tomorrow.
We still have 1 KVM and 2 OpenVZ nodes running, the VPS Control Panel is still running so you can manage your VPS's but right now I have no way to handle trouble tickets.
While we have a spare server handy, it's not going to replace 3 server's worth of customers and the 3 Nodes above and this one were all ordered at the same time, I'm loathe to even put it into service right now.
I will work to make this right, just give me a few hours to collect my thoughts, talk to my insurance company on what they can help with, yell at my system builder for a bit, and a couple hours of sleep please. My first priority is restoring the website and the customer portal while getting the bits and pieces together to fix the other servers.
Thanks for everyone's patience with this and I'm terribly sorry. I'll know more later on today.
Corey
CEO Front Range Hosting, LLC