We had some more issues.

To add insult to injury after our bumpy move to a new server, we had some more issues in the past couple days.

When we got fired up on the new server, we had a software RAID array. We had two harddrives, one mirroring the data on the other; sort of an "always backed up" setup. Unfortunately software RAID setups often have issues and we got kicked hard. The mirror failed and we ended up with some corrupted data. However, we got lucky and found out that only a small number of files got corrupted, and if they were corrupted on one of the disks, the other disk probably had a good copy. So we were able to recover almost all the data. We decided to knock one of the drives down to "emergency use only" since the file system was more corrupted than the other one.

Today, however, the worst happened. It turns out the cause of our software RAID setup working wasn't a software glitch or anything relating to the RAID itself, but was a harddrive failure in the making. It started out as a series of filesystem errors and then things got really flaky. (If any of you use Linux, take a look at the dmesg command and imagine <i>all</i> that output replaced with harddrive errors.) Luckily, we were able to recover data. Unfortunately this time we were unable to get an almost "fresh" copy of it, instead everything has rolled back to April 13th. Unfortunately this is bad news for a board such as this, since there were alot of interesting posts in the past few days.

In the mean time we've moved all of our customer sites to another server on the network until we can get that drive replaced. We will not move back to the regular server until we are assured that it is solid and reliable and we have a good backup solution in place. We plan on having a backup image updated every hour on the hour when we put it in place.

<b>Also</b>, and probably most importantly, due to the problems we've been experiencing lately, ketnar.org is now having a billing freeze. All outstanding balances will be reset to zero owed when billing resumes, and 2 months of service will be credited to all current accounts, in addition to the time that our billing freeze is in effect. We apologize for the problems we've been having lately, and are trying to get things back in order so that we can continue to offer the outstanding service and support our customers deserve.

<b>*</b> For the curious, the drives that failed were both Western Digitals. They're pretty good disks for light or medium work loads, but obviously not suited for a server environment.

Again, we apologize deeply for the mess that's been happening lately. Hopefully the road from here on out will be much smoother.

Sincerely,
Damon Canine
-- ketnar.org Systems Administrator

lol, i had a WD hd go bad on me too... sure, and they claim to have great safe storage systems... well, from here on i will no longer buy their drives ;)

also, silver/fluke/mods.... should we try to re-post stuff we lost, or just leave things as are and continue on with life? either works i guess ^^

(Edited by thelonewolf 27 at 3:07 pm on April 18, 2003)

Oh well! I am only a bit upset, because in the past week, I have posted like 2 or 3 new topics! Craps! But then again, shit happens, ya haveta git used 2 it! I don't really feel like reposting em' all, but I will try mi best! Good thing I didn't like post THOUSANDZ o'
newtopics, cuz that woulda' really sucked! It's just a very odd coincidence that all this horrible stuff is happening this week! Sorry about my complainin', but like hecka stuff in my life is going wrong rite now!

All well! U must "roll with the punches" as many people often say. So long.

(Edited by jewelswulfe at 4:31 am on April 19, 2003)

So that's where my post went! Glad that's solved. Anyways,lucky nothing serious happened. And like Nex said,I feel like I've gone back in time or something.

All the best!

Thanks again for all of the support Damon.  Small glitches or not, you're still doing a great job.

Ah, repost if you want to.  It's also partially my fault.  I should make it a point to back up the forums everyweek so we don't lose too much data.

And we're in a midst of cleaning out old forums threads, so it won't hurt to start a few new ones. ;)

Oooo... Like the new AV, Silver. It has a very artistic antique feel to it, superb and memory-inducing. It sort of reminds me of the cover on my copy of "Stone Fox." That was such a sad book.

~ Jay ~
KSC FL

Thanks! �It's scan of Balto from tne book, Balto and the Great Race. �I just messed with the hues in photoshop.

yeah, i like it too silver =)

but even if you were backing it up it every week, we still would have lost it all, this was only six days of posts...

(Hi guys, this is hudson from ketnar.org)

Just a little update on damon's post:
The backups would be active backups. every hour..
IE: user uploads new files, posts on a forum, or whatever. the server checks for 'changed, new, updated' files, every hour, and zaps them over to the backup server.

saves server load, instead of letting a weeks worth of changes pile up and then try to cram into the backup server.

We will also be backing up mail spools, system config files, user/pass/group files, and other things that are required to get our system up and running in short order in case we have some sort of freak system failure and we have to rebuild.

This backup will also include proper database dumps and backups.

bottom line: all bases coverd, we WONT be doing this again.

What doesn't kill you makes you stronger.

A very good and true saying indeed. I myself follow that saying.

Well, i got suckerpunched by the "Free Scratch Cards"
Nag-ad lol the good news is that after modifying the msconfig start up and removing a few reg files, it doesnt pop up anymore, so i guess im safe LOL

(Edited by LongBow986 at 2:04 pm on April 20, 2003)

Never know when it might come back. ;)

That's great to hear!  Thanks Hudson, you guys at ketnar are the best. ^_^