Wow, what a crazy series of events! Sunday June 8th, 2003 around 10:30 pm the server that runs our community had a hard drive failure. The next 2 days were spent restoring all the sites on the server so the 300+ email accounts and 80 other business that use this server could be back up and running.
Organic Hosting is a side business I started in 1999. I have donated the server resources for our community so my number one priority was to restore the "paying" client's websites and then get the forums working again. Of course my "real" job was also requiring 100% of my attention so it was hectic for me this week to say the least.
To tell you the truth, it was probably our community that crashed the server because we are getting so much traffic now. I plan on moving the community to a higher performance server at some point in the next couple of months.
When the hard drive failed the database became corrupted. In the beginning I was backing up the database on a daily basis. Due to the growth of our community I changed that to a weekly because the backups are so large.
I spent a couple of days trying to resolve the corrupt database so no information would be lost. I had to learn what to do, contact database administrators and try and fix it myself and I am not a programmer. It was very tough and frustrating. I tried everything possible and finally decided not to spend more time trying to fix it and just move forward restoring to a backup. It is one thing to make backups, but you need to know what to do with them in case of emergency. So back to the manuals and MySQL documentation. I sure have learned a lot!
Our backups are so big that it takes 3 hours to upload them using a cable modem. On top of that, storms rolled through Dallas this week that knocked out power at my apartment building twice. So just getting the backups to the server was a chore! Finally I was able to try and restore from the uploaded backup. That did not work! The backup was corrupt! I spent some time trying to fix the corrupt backup and finally gave up. So then I had to upload the other backups I had and restore to the latest one that was not corrupt. Once again, uploads took forever! Finally the May 20th, 2003 backup restored without any problems. So it was approximately 18 days before the meltdown.
So if you registered after May 20th, 2003 you will need to reregister. All posts, threads, etc. posted after May 20th were lost. I apologize for this and will revise our disaster recovery plan to include some quality controls to make sure backups are not corrupt moving forward (I will test them after each backup).
Once again, sorry for the downtime, I learned very much from it and I will post our disaster plan when I am done writing it. I am off to the Farmerís Market to get some fruit then go to the park and relax. After the week I had, it is going to feel soooooo GOOD!