08/29/13 Outage

Toruk Makto · August 29, 2013, 05:45:41 PM

The colocation facility had another power failure that took us down today. This time apparently it was a 100A 3-phase breaker that suffered a seal failure allowing all the oil in it to drain out. When it finally tripped, the NOC tech tried resetting it. He actually did that a couple of times (which is bad, because it can turn a server's filesystems into mince pie). The failed breaker was replaced and we are back with apparently no lingering effects other than the filesystem consistency checks that will have the LN and Dothraki sites a bit slow for a few hours. Good news is that our server survived what is sometimes a fatal sequence of events. Our Kelutral is strong!

ta Markì

Tìtstewan · August 29, 2013, 05:49:55 PM

Quote from: Toruk Makto on August 29, 2013, 05:45:41 PM
Our Kelutral is strong!

FÌ'U!

Well, I don't know, what I would do with me, if happend a data crash...
I was correcting the pronoun guide today...*whuaaa not again* in my thoughts...

Toruk Makto · August 30, 2013, 11:41:54 AM

Kelutral is strong, but Toruk Makto is a bit of a nawma skxawng (true to the film, right?). I missed a critical step in the filesystem checks yesterday evening that kept us down all night. Oeru txoa livu!

ta Markì

Tìtstewan · August 30, 2013, 12:00:44 PM

Don't worry! Everything is OK now.

Clarke · August 30, 2013, 12:04:10 PM

None of the images are loading for me, even with a hard refresh.

`Eylan Ayfalulukanä · August 30, 2013, 12:23:57 PM

An oil leak in an 100A 3 phase breaker? I've never heard of breaker like that. We have 400A 3 phase breakers here that have no oil in them.

This also sounds like a 'single point of failure' type device, and are a problem unless you have a server with dual power supplies. (Many of our broadcast devices have dual power supplies, designed to be connected to two different power sources.)

In any case, thing are working again, ulte tsatìfkeytok set lu zo!

Irayo nìtxan nìwotx!

Toruk Makto · August 30, 2013, 12:31:01 PM

I am looking into a redundant PDU power drop to our cabinet. The server is a single PS type, but I can get a small auto transfer switch that will do pri/sec switching in ~15ms, which would keep us up in case of a single feed failure again.

Clarke: I forced a refresh of the CDN cache at CloudFlare. Please let me know if you continue to have trouble loading images.

Taronyu Leleioae · August 30, 2013, 12:57:40 PM

Quote from: `Eylan Ayfalulukanä on August 30, 2013, 12:23:57 PM
An oil leak in an 100A 3 phase breaker? I've never heard of breaker like that. We have 400A 3 phase breakers here that have no oil in them.

Oil filled circuit breakers are an OLD design and were mostly for power plants and biiig applications to dissipate heat. And even then, there are new materials. Usually breakers of medium duty are filled with air or some inert gas.

The facility should offer a connection. If it's ridiculous pricing, then maybe find a clean older APC 3000 rack mount and replace the batteries.

Markì, on behalf of all of us, "irayo" for putting up and dealing with this...