Server Outage - 24th-26th Feb

[Jon]

The dedicated server that runs the autoitscript.com / autoitconsulting.com site had a severe outage for the last couple of days.

The main problem was that one of the disk drives failed.

The server has two drives in RAID 1 so normally this isn't a problem, in fact there have been two disk outages over the past few years that many people won't have even noticed. Usually the server runs as normal with one drive down and there is only a brief 5 minute outage required when the disk is replaced and the server is rebooted. However in this case the server hard crashed and wouldn't boot into the OS on the remaining disk. I'm guessing that the drive was actually failing...working...failing and causing issues because it wouldn't just die.

I logged a support call and asked to get the disk replaced, but some of the communications were lost and it took 12 hours for them to acknowledge that the drive was dead and that they had the authorisation to go ahead and open the box and replace the drive (they only touch the hardware, I'm responsible for all the software/OS config). During this time the server was sitting in a recovery/linux boot disk mode used for troubleshooting. When the disk was finally replaced (8pm UK time 25th of Feb) the the technician only emailed my hostmaster / autoitscript email address to say that it was complete but I didn't receive it because of course that email address is hosted by the server...which is in recovery mode - they forgot to email the temporary gmail address I was using during the outage. So I only found out the work had actually been completed when I chased them up this morning on the phone.

I've started the RAID rebuild onto the new drive which will take a few hours to complete so the site may be sluggish until then. I also had to repair a few mysql tables that had crashed - but it didn't look to be a problem.

Edited February 26, 2014 by Jon