RFO for outage Monday, August 14th

Started by Toruk Makto, August 14, 2017, 11:31:04 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Toruk Makto

 At approximately 00:15CDT on August 14, all services on the primary LearnNavi server failed with filesystem permission errors. SSH access was also disrupted.

The reason for the outage was a typo in a chmod command inadvertently resetting the server root directory permissions incorrectly. A local console session was required to correct the permissions which further required an on-site presence at the Houston Internap data center.  After traveling to the site and obtaining local access, resolution was accomplished within 10 minutes. All apps and services were restored at 18:45CDT. Total outage time was 18.5 hours.

-Markì

Lì'fyari leNa'vi 'Rrtamì, vay set 'almong a fra'u zera'u ta ngrrpongu
Na'vi Dictionary: http://files.learnnavi.org/dicts/NaviDictionary.pdf

Ertew

Great to know that it's just access problem and no data are lost.
BTW I notice that IRC still working while forum and main page are down. What's the reason for that?
Any errors in spelling, tact or fact are transmission errors.

Hi! I'm a signature virus. Copy me into your signature to help me spread.

Tìlu

Quote from: Ertew on August 16, 2017, 04:51:33 AM
Great to know that it's just access problem and no data are lost.
BTW I notice that IRC still working while forum and main page are down. What's the reason for that?
Perhaps not hosted all on the same server/s that went down.  :-\ At least my thought until otherwise said.
Glad to hear it wasn't anything major though!

Toliman


Tanri

That's a nasty example of kxeyey ayzekwäyä ;)
Tätxawyu akì'ong.

Toruk Makto

Quote from: Ertew on August 16, 2017, 04:51:33 AM
Great to know that it's just access problem and no data are lost.
BTW I notice that IRC still working while forum and main page are down. What's the reason for that?

Ma Ertew, the IRC service is provided by multiple servers, so even if the LN node is down, there are a couple of others. Also, Teamspeak, Discord, Webex and the social media pages are diverse as well.  -M.

Lì'fyari leNa'vi 'Rrtamì, vay set 'almong a fra'u zera'u ta ngrrpongu
Na'vi Dictionary: http://files.learnnavi.org/dicts/NaviDictionary.pdf

Toruk Makto

Quote from: Tìlu on August 16, 2017, 06:57:33 AM
Quote from: Ertew on August 16, 2017, 04:51:33 AM
Great to know that it's just access problem and no data are lost.
BTW I notice that IRC still working while forum and main page are down. What's the reason for that?
Perhaps not hosted all on the same server/s that went down.  :-\ At least my thought until otherwise said.
Glad to hear it wasn't anything major though!

Srane, you are precisely correct!   :D

Lì'fyari leNa'vi 'Rrtamì, vay set 'almong a fra'u zera'u ta ngrrpongu
Na'vi Dictionary: http://files.learnnavi.org/dicts/NaviDictionary.pdf

`Eylan Ayfalulukanä

I see I missed some excitement while I was out of town. Glad to hear it was easily fixed.

Ma Tanri, sunu oeru lu san kxeyey ayzekwäyä ;)

Yawey ngahu!
pamrel si ro [email protected]

lea89

Thank you for the information.
I was already afraid that the domain has expired.
I recently found this forum and it would be devastating information
Jaki preparat usuwa toksyny i czyści organizm? Jeśli jesteś ciekawy zapraszam.