Author Topic: 08/29/13 Outage  (Read 1624 times)

0 Members and 1 Guest are viewing this topic.

Offline Toruk Makto

  • LearnNavi Admin
  • Toruk Makto
  • Palulukan Makto
  • *****
  • *
  • Posts: 6215
  • nv Eywa'eveng
  • Karma: 215
  • . Txepsiyu Markì .
    • Learn Na'vi
08/29/13 Outage
« on: August 29, 2013, 05:45:41 pm »
  The colocation facility had another power failure that took us down today. This time apparently it was a 100A 3-phase breaker that suffered a seal failure allowing all the oil in it to drain out. When it finally tripped, the NOC tech tried resetting it. He actually did that a couple of times (which is bad, because it can turn a server's filesystems into mince pie). The failed breaker was replaced and we are back with apparently no lingering effects other than the filesystem consistency checks that will have the LN and Dothraki sites a bit slow for a few hours. Good news is that our server survived what is sometimes a fatal sequence of events. Our Kelutral is strong!

 ta Markì

Lì’fyari leNa’vi ’Rrtamì, vay set ’almong a fra’u zera’u ta ngrrpongu
Na'vi Dictionary: http://files.learnnavi.org/dicts/NaviDictionary.pdf

Offline Tìtstewan

  • LearnNavi Zeykoyu
  • Toruk Makto
  • Eywatsyìp
  • *****
  • *
  • *
  • Posts: 10003
  • de Germany
  • Karma: 324
  • Ke lu oeru kea krr krrtalun!
    • My YouTube Channel
Re: 08/29/13 Outage
« Reply #1 on: August 29, 2013, 05:49:55 pm »
Our Kelutral is strong!
FÌ'U!

Well, I don't know, what I would do with me, if happend a data crash...
I was correcting the pronoun guide today...*whuaaa not again* in my thoughts...

-| Dict-Na'vi.com | Na'viteri Files | FAQ | LM | Puk Pxaw 'Rrta | Kem si fu kem rä'ä si, ke lu tìfmi. |-

Offline Toruk Makto

  • LearnNavi Admin
  • Toruk Makto
  • Palulukan Makto
  • *****
  • *
  • Posts: 6215
  • nv Eywa'eveng
  • Karma: 215
  • . Txepsiyu Markì .
    • Learn Na'vi
Re: 08/29/13 Outage
« Reply #2 on: August 30, 2013, 11:41:54 am »
 Kelutral is strong, but Toruk Makto is a bit of a nawma skxawng (true to the film, right?). I missed a critical step in the filesystem checks yesterday evening that kept us down all night. Oeru txoa livu!

 ta Markì

Lì’fyari leNa’vi ’Rrtamì, vay set ’almong a fra’u zera’u ta ngrrpongu
Na'vi Dictionary: http://files.learnnavi.org/dicts/NaviDictionary.pdf

Offline Tìtstewan

  • LearnNavi Zeykoyu
  • Toruk Makto
  • Eywatsyìp
  • *****
  • *
  • *
  • Posts: 10003
  • de Germany
  • Karma: 324
  • Ke lu oeru kea krr krrtalun!
    • My YouTube Channel
Re: 08/29/13 Outage
« Reply #3 on: August 30, 2013, 12:00:44 pm »
Don't worry! Everything is OK now. ;) :D :D

-| Dict-Na'vi.com | Na'viteri Files | FAQ | LM | Puk Pxaw 'Rrta | Kem si fu kem rä'ä si, ke lu tìfmi. |-

Offline Clarke

  • Taronyu
  • ****
  • Posts: 505
  • Karma: 8
  • This is gonna be great
Re: 08/29/13 Outage
« Reply #4 on: August 30, 2013, 12:04:10 pm »
None of the images are loading for me, even with a hard refresh.

Offline `Eylan Ayfalulukanä

  • Palulukan Makto
  • *****
  • *
  • *
  • Posts: 4767
  • us United States
  • Karma: 44
  • Palulukan alu Kenya 06/23/1996 - 01/15/2017
    • The Lionlamb website
Re: 08/29/13 Outage
« Reply #5 on: August 30, 2013, 12:23:57 pm »
An oil leak in an 100A 3 phase breaker? I've never heard of breaker like that. We have 400A 3 phase breakers here that have no oil in them.

This also sounds like a 'single point of failure' type device, and are a problem unless you have a server with dual power supplies. (Many of our broadcast devices have dual power supplies, designed to be connected to two different power sources.)

In any case, thing are working again, ulte tsatìfkeytok set lu zo!

Irayo nìtxan nìwotx! :)

Yawey ngahu!
pamrel si ro [email protected]

Offline Toruk Makto

  • LearnNavi Admin
  • Toruk Makto
  • Palulukan Makto
  • *****
  • *
  • Posts: 6215
  • nv Eywa'eveng
  • Karma: 215
  • . Txepsiyu Markì .
    • Learn Na'vi
Re: 08/29/13 Outage
« Reply #6 on: August 30, 2013, 12:31:01 pm »
  I am looking into a redundant PDU power drop to our cabinet. The server is a single PS type, but I can get a small auto transfer switch that will do pri/sec switching in ~15ms, which would keep us up in case of a single feed failure again.

Clarke: I forced a refresh of the CDN cache at CloudFlare. Please let me know if you continue to have trouble loading images.

Lì’fyari leNa’vi ’Rrtamì, vay set ’almong a fra’u zera’u ta ngrrpongu
Na'vi Dictionary: http://files.learnnavi.org/dicts/NaviDictionary.pdf

Offline Taronyu Leleioae

  • Taronyu
  • ****
  • Posts: 521
  • Karma: 16
Re: 08/29/13 Outage
« Reply #7 on: August 30, 2013, 12:57:40 pm »
An oil leak in an 100A 3 phase breaker? I've never heard of breaker like that. We have 400A 3 phase breakers here that have no oil in them.
Oil filled circuit breakers are an OLD design and were mostly for power plants and biiig applications to dissipate heat.  And even then, there are new materials.  Usually breakers of medium duty are filled with air or some inert gas.

The facility should offer a connection.  If it's ridiculous pricing, then maybe find a clean older APC 3000 rack mount and replace the batteries.


Markì, on behalf of all of us, "irayo" for putting up and dealing with this...

 

Become LearnNavi's friend on Facebook Follow LearnNavi on Twitter! Watch LearnNavi's videos on YouTube

SMF 2.0.18 | SMF © 2021, Simple Machines | XHTML | RSS | WAP2 | Site Rules

LearnNavi is not affiliated with the official Avatar website,
James Cameron, LightStorm Entertainment or The Walt Disney Company.
All trademarks and servicemarks are the properties of their respective owners.
Images in the LearnNavi.org Forums and Gallery may not be used without permission.

LearnNavi Affiliates:
ToS

LearnNavi is the community to learn Na'vi, the Avatar Language
"A place where real friendships are made." -Paul Frommer

AvatarMeet | Learn Na'vi Forum | Learn Na'vi Wiki | Na'viteri

LearnNavi