The Lion's Den

Eminent Drive Failure

Warning! Warning! Eminent Drive Failure!

Those are three words you really don’t want to see come up on your server. While trying to recover from a mangled dselect configuration I was forced to reboot the server that hosts GeelRoar.com and PWASOH.com after 212 days of uptime. During boot the server complained about hard drive damage and directed me to conduct an fsck. The data drive seems to be healthy but the system drive complained of a lot of unrecoverable errors and on reboot gave me that sickening little sentence 'the system has detected eminent drive failure'. What to do? As I see it I have four options:

1. Ignore it: The data drive is healthy and whatever damage happened to the system drive seems to be affecting parts of the OS I am not using because the system continues to work with no damage reports. The system drive could fail anytime between the next minute and next decade. Chances are it will go down within a few months but that could be a few months to find some other solution.
Pros: No cost, no downtime
Cons: The drive could fail at any time. Once it fails my domains will go down and getting data off the data drives will not be trivial.

2. Replace the System Drive: I can buy a new 40GB drive pretty cheap and swap it out. I would have to shut the system down while I move the data and get the server to boot from the new drive.
Pros: Cheap. The rest of the system seems to be ok so this will probably give the server another few years of life.
Cons: The servers will be offline for however long it takes me to get the new drive set up. I could try to boot the system with the new drive mounted and scp the files directly over but I will then be inheriting some setup glitches and I seriously doubt the old drive will let all the data be moved over. At this point I might as well re-install LINUX and set the whole server up from scratch using a newer distribution. Which brings us to:

3. Replace the whole server: The machine that serves my domains is pretty old by now as is the distribution of Debian I have installed. The system is plenty fast to serve the sites out at the speeds my home DSL can handle but I have the feeling this hard drive may be the first of many components to give up and die. I could buy a new inexpensive generic Wintel box or assemble one from parts, put a new OS on it and set everything up from scratch. Last time I did this it took me a few weeks to get everything configured properly; I am guessing I can do it in half the time now which is still a few days worth of work.
Pros: Very little downtime since I can set up and configure the new machine while the current one handles the load. The only downtime will be when I switch over and do a final config on the new machine.
Cons: I think I can get a decent machine for under $700 but that is still money I would rather not spend. It would still take me some time to get the new machine up and running.

4. Switch from home hosting to a hosting site: This is the big switch I have been contemplating for some while now. There is something very appealing about knowing my sites (my email) wont go down every time my router decides to lock up or a breaker goes off at home. Last time the server failed I was in France and had to ask someone to enter my house and reboot the network switch/access point. The sites will be faster, have updated software and be backed up and looked after. I have looked around and researched different sites and like the offering made by DreamHost the best. For $20 a month (no setup fee if I pay for a year up front) I get everything I need to run my domains (shells, SSH, PHP, PERL, Apache, CGI, POP, SpamAssassin, SSI, .htaccess).
Pros: No more worries about running my own server. Backups. More uptime. Faster sites. Announcement lists (I have tried to set them up a home twice now and failed both times).
Cons: $20 a month for basically services I can offer at home. Bandwidth restrictions that can get very expensive if they are exceeded. The $20/month package comes with 192GB/month. I have calculated my cumulative usage (excluding POP and SCP, basically only HTTP) to be 50GB/month with a peak of 94GB in the last year. I have been slash-dotted a couple of times (not by Slashdot per se but by other sites linking to articles on GeekRoar.com) but my puny home DSL has each time acted as an automatic choke only allowing a few simultaneous connections and thus keeping the usage below spec. I don't know what the total monthly usage during these times would have been if it had not been restricted and fear they could have gone up to if not beyond 192GB. At $0.50 per additional GB per month an eightfold increase of my average monthly traffic (very plausible if a page suddenly becomes overly popular) would cost me an additional $106 for the month. Not exactly a bank-breaker but more than the $20 I had allocated. In this model I not only have to monitor my use closely but I can no longer indiscriminately post movies or other large media files.

What to do? I am honestly not sure. I am leaning towards options 3 (replace the whole system) or 4 (go to external hosting) with 3 feeling better right now. I would love your thoughts and opinions.