Nightmare Weekend

It wasn't the worst weekend possible, but let's just say it wasn't the best either. It all started Friday when I found out my mother's Windows ME based PC was no longer able to boot (“Windows Protection Fault” yada yada yada). It appeared that a virus had exploited the IE security holes and attacked her system. Considering that this was the second Windows ME reinstall the computer appeared to need in as many months, I suggested that we switch her system over to GNU/Linux.

So, I set about working on the system, first trying to repair Windows and then installing Fedora. I got done about 2:00 a.m. Yesterday morning, I was tired, but mostly refreshed.

Here's the catch: she likes to use MSN Groups and enjoys the Internet Explorer-only features that MSN makes available. She also likes JASC Paint Shop Pro. So, I do the logical thing: I get Win4Lin 5.1 for the system. Only problem is, Win4Lin doesn't support the distribution I had picked out for her system, Fedora Core 2. So I start playing around with generic kernel patches from NeTraverse (Win4Lin's developer) to see if I can get it to work. Finally I do, by switching to an entirely generic kernel without any of Red Hat's optimizations, but unfortunately that made it so that I couldn't get the nVidia drivers to install.

While I wanted her to have GNOME 2.6, I decided it wasn't worth fighting that. So I pulled out Mandrakelinux PowerPack+, which has Win4Lin enabled kernels and automatically installs support for the latest nVidia and ATI graphics cards. It goes in smoothly and I set about updating the system, migrating data and so forth. I worked until about midnight and decided to call it a night. I had hoped to give Mom's system back to her today (Sunday), but I was was a bit pooped from the late night the night before.

12:00 a.m.Before jumping into bed, I flipped on my laptop to check my e-mail. Only I couldn't connect to my e-mail. Asisaid and all of my other sites were crawling. I jumped over to serverforest.com only to see the server's load averages being reported at about 80 for each. That isn't good, if you're wondering. It is actually really, really bad unless there is some specific reason it should be that high. I tried logging into my administrative web interface or SSH but both were unavailable thanks to the high processor load.

12:15 a.m. I had no choice but to e-mail my data center and ask them to reboot the system. I can't do anything until the system is in a state other than a deadly resource tailspin. I start to panic, I'm suppose to be up within about seven hours to go on a short trip and a solution is nowhere in sight.

12:30 a.m. The data center technician starts trying different was of coaxing the system to reboot. Nothing works for a very long time and he eventually gives up on logging into the system. I begin to wonder if cPanel blasted apart my mail server again during a nightly update.

12:50 a.m. The server comes back on after being forced to reboot. The resource usage starts to climb again. Local bandwidth and load monitoring shows 100x increase over a two hour period — something is really wrong. It looks like it is going to be a long night. The amount of time I will have to sleep is shrinking.

1:15 a.m. I realize that SAFARI Count+Stat Remote, my referrer tracking tool is being hit hundreds of times a minute by one site, forcing dozens of copies of MySQL and Apache to load. I disable the script and place a message about it being temporarily out of order in its place.

1:40 a.m. Everything finally returns to normal. I go to bed. I can't sleep wondering if the onslaught might be a DoS rather than just CSR being used by a busy site. I get up a few hours later.

Today I'm pooped — two late nights and then getting up early this morning have really taken their toll. But, I did get my mother's computer almost ready to go back to her. I'm hoping to give it back to her tomorrow. The trip to see the Cahokia Mounds in Illinois that I went on with some family turned out very enjoyable, even despite lack of sleep.

Concerning CSR, I re-enabled it today and all seemed well, but it was starting to act up again, so its down for the moment. It is being used on a financial scam site in Australia that is, I think, spamming people and thus causing a massive amount of traffic every night (U.S. time, daytime down under). CSR doesn't presently have a way to block sites from using it, so I'm developing one to block this site that has caused so much trouble. sigh

So that is what I've been up to, how about all of you?