Site Status

[Theo]

No rest for the wicked. I had to replace a server on Christmas night.

[Yodama Man]

booooooo, that's no fun, ya should have spent christmas with your family, not like anyone was on christmas night anyway.

[Theo]

Site might have been running slow today. We did some database replication to a backup server.

[Theo]

One of the nodes ran out of memory. For some reason the swap file was only set to 200mb. I should have been 2000mb. It's been fixed but required a reboot.

[Theo]

If you are reading this then you are not affected by the Comcast->ATT router hiccup that is preventing traffic to this web site. Our provider is awaiting hardware replacment by ATT for our ISP.

It's taken me a while to get this email out as I am one of these affected users and have had to reroute a lot of VPN tunnels just to get access to this site.

[Yodama Man]

what if you are reading this?

no wait...i mean arn't...how will i know what i am not reading?

i'm scared

[Theo]

Yeah yeah yeah, the clock's off by an hour. I know... Sheeeesh, you figure since I was patching servers all day you guys might let me slide.

Good news is that I patched all our servers last week. Bad news is that the patch didn't work and had to be re-applied today.

So, for those that are interested, I will be patching the AA servers (including databases) on Tuesday morning at 9:am PST. It will take a painful 20 minutes.

[Theo]

Todays slowness and outage is still being investigated. While looking into the cuase we found that one of the drives on the database server is having problems. It will be replaced. I'm not sure about the schedule though. In the mean time, I will postpone the time update until Thursday.

If anyone happens to have any 1u dual opteron servers with 2gb of ram laying around ready to donate, we'd love to hear from you.

[Theo]

Today we had another outage. There were a few problems. I think we have tracked part of them down to some recent changes. Not sure what the others are at this time. I just haven't had the time to put into diagnosing it. The NFS server running under XEN didn't seem to help with the recent change. I've moved it to a server with a little more horse power.

[Mai]

(Theo, my man. It's been a while. Why don't you let me annoy you anymore? )

Oh, oh, oh!..

"1u dual opteron servers with 2gb of ram "
English?

(You psycho. )

[Theo]

Translation:

"1u dual opteron servers with 2gb of ram" = Computer hardware or donations

[Mai]

Oh, you mean smart Theo stuff.

[Theo]

I'm currently replacing the drives on the server. There might be a couple interuptions during this time. The shouldn't last more than 5 minutes each, if that. There has only been a single outage thus far tonight (though I have failed the nodes over a dozen times already today).

[Theo]

Maintenance is done for the night. Actually I missed night by 2 minutes. Anyway, the last reboot took about 3 minutes. Everything appears to be back up and running just fine. I guess I will find out what's broken in 5 hours when the clients start calling...

[Theo]

I had to reboot the database/file server for AA. It was done for < 60 seconds. Everything appears to be working now.

[Commander Sparx]

There seems to be allot of maintenance to the site to keep it running.