You Suffered A Catastrophic Failure, Please Surrender Your Server
Posted: Thu May 30, 2019 11:10 am
Sorry for the outage.
Yesterday when I came back (late) from my Norwegian lessons, I found out the server was dead. It was actually cold, so I suppose it had been dead for a few hours at least.
Anyway, as far as I can see, we suffered from a number of things:
- The PSU died
- That somewhat corrupted the filesystem on the two primary drives (the SSD with the OS and the HDD with all the files)
- After finally finding a compatible power supply (the joy of mini-itx cases) the BIOS told me it was corrupted, but that thankfully it was a "Dual BIOS" motherboard so it was able to restore itself (phew).
- After rebooting again, I found out that the defence-force.org and osdk.org sites worked, but the phpbb claimed I was using some ancient version of PHP, which was surprising since it had been updated to the latest just a couple months before.
- At that point I realized that GRUB failing to boot on the main boot drive decided to boot on the old Intel X25 SSD which had the original Ubuntu 10.04 install, which yes, was old and everything... but at least that gave me a working shell... Unfortunately for some reason, the fscheck tools were signaling errors
- I tried booting the main OS drive, and got a litany of filesystem errors, missing inodes, ...
- I decided to ask for help on #ubuntu (on freenode) where using a Live CD was suggested.
- Downloaded the ISO, burnt it... booted it... did not boot... yeahhhh, first failed DVD burn in a decade.
- Tried another DVD with the original Windows ISO burner, and this time that worked fine.
- A bit of checking, mounting, backuping (in case of), and after a clean reboot... it decided to stop on a blinking cursor
- Found out that the Bios had decided to put the non bootable HDD as the primary drive
- After fixing that, it booted just fine, and as far as I can see is still running.
- I installed the latest updates, rebooted again, and here we are.
So now the question is: Did anything actually got corrupted, are there some broken pages, is SVN broken, is PHPBB somewhat corrupted, etc...?
If you find anything, please tell, in the meantime I'll have to order a new PSU, and investigate if I can somewhat find some way to get less hardware problems leading to downtime, without having to use a shitty hosted server that makes my life miserable and the latency intolerable.
Thanks for your patience!
Yesterday when I came back (late) from my Norwegian lessons, I found out the server was dead. It was actually cold, so I suppose it had been dead for a few hours at least.
Anyway, as far as I can see, we suffered from a number of things:
- The PSU died
- That somewhat corrupted the filesystem on the two primary drives (the SSD with the OS and the HDD with all the files)
- After finally finding a compatible power supply (the joy of mini-itx cases) the BIOS told me it was corrupted, but that thankfully it was a "Dual BIOS" motherboard so it was able to restore itself (phew).
- After rebooting again, I found out that the defence-force.org and osdk.org sites worked, but the phpbb claimed I was using some ancient version of PHP, which was surprising since it had been updated to the latest just a couple months before.
- At that point I realized that GRUB failing to boot on the main boot drive decided to boot on the old Intel X25 SSD which had the original Ubuntu 10.04 install, which yes, was old and everything... but at least that gave me a working shell... Unfortunately for some reason, the fscheck tools were signaling errors
- I tried booting the main OS drive, and got a litany of filesystem errors, missing inodes, ...
- I decided to ask for help on #ubuntu (on freenode) where using a Live CD was suggested.
- Downloaded the ISO, burnt it... booted it... did not boot... yeahhhh, first failed DVD burn in a decade.
- Tried another DVD with the original Windows ISO burner, and this time that worked fine.
- A bit of checking, mounting, backuping (in case of), and after a clean reboot... it decided to stop on a blinking cursor
- Found out that the Bios had decided to put the non bootable HDD as the primary drive
- After fixing that, it booted just fine, and as far as I can see is still running.
- I installed the latest updates, rebooted again, and here we are.
So now the question is: Did anything actually got corrupted, are there some broken pages, is SVN broken, is PHPBB somewhat corrupted, etc...?
If you find anything, please tell, in the meantime I'll have to order a new PSU, and investigate if I can somewhat find some way to get less hardware problems leading to downtime, without having to use a shitty hosted server that makes my life miserable and the latency intolerable.
Thanks for your patience!