Get yourself a @skxawng.lu email!

[hawnuyuna'viyä]

Update:
There were problems with
a) Physical memory usage - often resulting in issues with spamd actually. Which was the single largest process on the system.
b) Kernel memory usage - mostly resulting in Postgres failing
c) Socket limits set by the host - at one point the limit was hit, rendering me unable to ssh in to fix the problem
d) Bad design on my part (the monitoring system is on the same server, so since postfix went down, it was unable to send me a mail to tell me that there was a problem with postfix). <- non-recursive solutions to this problem are welcome
e) A compromised user account over the weekend, resulting in these problems being amplified
f) My presence at the London minimeet (see that thread on ToS), meaning that I lacked much time to deal with any of these problems
g) Issues with my personal computers as well, which broke my network connection (thanks Systemd!)
h) The lack of rate limiting currently on the mail accounts - I hadn't got around to implementing it yet. (it was on the to do list).

So, as you can see from the graph, a mailqueue of ~40k mails isn't going to make this small vps particularly happy. As you can see, I managed to ssh in (after having the socket limit of my server increased), to let me kill postfix and wipe the queue of the random spam that had built up in it from the compromised account.
Postfix is still offline. I hope to bring it back later today.

In the meantime, I should probably go sleep (it is now past 0300 am here, and I need to be up again by 0800).

Ngaytxoa.

[hawnuyuna'viyä]

Some progress was made today:
There is now a secondary MX server (mx2.skxawng.lu) which will keep track of any incoming messages in the event mx1.skxawng.lu is down. (This prevents any getting lost, but does not mean you can access your account when mx1 is down).
Mail rate limiting quotas have now been put into place: For any internal (@skxawng.lu<->@skxawng.lu) rate limit is ignored. For any external emails, you may receive up to 50 mails/h, and send 15 mails/h.

This should prevent a repeat of the compromised account, which sent out slightly more than 100,000 before Postfix fell over stopping it.
Interestingly, it was the work of a bot-net, because the logs show slightly over 1000 unique IPs connected to the account before it was stopped.

This has slightly lowered the ranking of the @skxawng.lu domain temporarily (I know that Google is rejecting some mail currently, Yahoo was yesterday, but doesn't seem to be any more). But the reputation should improve automatically again over the next few days, since the problem has been dealt with.

HN

PS. If there is anyone following this who doesn't have an account, but would like one. Or who did have an account, but has forgotten the password, PM me to sort it out for you.

[Swoka Ikran]

Ouch. At least postfix died. As for the limits...the sending limit has an obvious purpose (although I'd say it's a bit on the low side), but why a receiving limit at all?

Yes it can keep the server from falling over, but such a limit seems like it'd create a new problem: it gives someone an easy way to deny a person receiving capabilities by just sending junk mails to them. After that, their mail gets dropped/backlogged for an hour. While normal accounts likely won't hit this limit, I feel it's begging to cause its own problems.

Spammers seeking to abuse an account would want to abuse the limited outbound side.

d) Bad design on my part (the monitoring system is on the same server, so since postfix went down, it was unable to send me a mail to tell me that there was a problem with postfix).

If you have a second server, put the monitor there and have it connect periodically to the various services to see if it gets expected results. Put another postfix install there, and use it only for the alerts (i.e. don't allow people to connect from the outside world).

If the server has to 'self-monitor', use another mail server for the alerts. If not possible, I've seen alerting done by having the monitor make HTTP requests to a PHP script sitting on a remote web server. The web server had program on it that logged the errors and sent email if appropriate. Web hosting capable of running this is available for free.

Only thing this won't handle is the network connection itself failing, and if that happens, all bets are off anyway since no form of self-monitoring would work.

Also two questions: Are you monitoring the monitor? And is there a monitor for the monitor's monitor?

[hawnuyuna'viyä]

[...] but why a receiving limit at all? Yes it can keep the server from falling over [...]

Which is  precisely why it is there. I admit that it could cause its own problems, but based on the current log data, 50 messages/h is far above where the accounts seem to be currently.
The numbers for sending/receiving will need to be tweaked, I picked them pseudo-randomly based on a quick glance at the current server statistics.

If you have a second server, put the monitor there and have it connect periodically to the various services to see if it gets expected results. Put another postfix install there, and use it only for the alerts (i.e. don't allow people to connect from the outside world).

Which is what this server used to be for some others that I control. But it is a bit over-powered for a simple monitoring master, so I put this email handling for this domain on it...

I have another 2 small vps's which I will be bringing up over the next few weeks anyway, so I may look at changing some of the roles I was going to put on them.

Also two questions: Are you monitoring the monitor? And is there a monitor for the monitor's monitor?

The monitor's monitor is me (manually checking the graphs and automated emails when the system is unhappy <- except for when the email system is unhappy).
The monitor's monitor's monitor is currently non-existent. (Perhaps if I setup a Google Calendar recurring event to ensure I check the monitor regularly... )

[hemmond]

Just question... I noticed that skxawng.lu changed owner and running again. What setup do I have to use in order to access my old skxawng.lu email adress? (ofc. that without e-mails, which I suppose was lost somewhere on the old server )... What server, port, encryption, etc... is in use?

EDIT: seems that I found that by trial&error method, but it'd be still cool to add the specs for others who weren't for some reason able to keep an eye on this topic.

[hawnuyuna'viyä]

Just question... I noticed that skxawng.lu changed owner and running again. What setup do I have to use in order to access my old skxawng.lu email adress? (ofc. that without e-mails, which I suppose was lost somewhere on the old server )... What server, port, encryption, etc... is in use?

Yep, I now control it. I have the old user/password list from Sir.Haxalot so your old login should work ok. I couldn't get any of the old emails though, so unless you backed up those yourself, they are lost.

Details can be found in msg578268.
(I will put them on the website once I make user registration work, exams ATM. though, so I haven't had much time to do anything).

[hemmond]

I knew, you've the passwords list. But I couldn't find in which post the infos was. But as I said, my Thunderbird is happy again. Has all of it's e-mail adresses working under his wings properly.

[Human No More]

Out of interest, how are you tracking reputation? That's a part of my job, managing email systems
As for the compromised account, you might want to use fail2ban to temporarily block IPs with authentication failures if it was just brute forced - it comes with premade regex sets for dovecot and courier. Depending on your script-fu, perhaps some kind of monitor for excessive senders as well.

Also, just had a read of this thread with the issues with spamassassin - how do you have it configured, via spamc? You should also look into sa-compile to optimise the standard (non-bayes) ruleset performance.

[hawnuyuna'viyä]

Out of interest, how are you tracking reputation? That's a part of my job, managing email systems
As for the compromised account, you might want to use fail2ban to temporarily block IPs with authentication failures if it was just brute forced - it comes with premade regex sets for dovecot and courier. Depending on your script-fu, perhaps some kind of monitor for excessive senders as well.
Also, just had a read of this thread with the issues with spamassassin - how do you have it configured, via spamc? You should also look into sa-compile to optimise the standard (non-bayes) ruleset performance.

Currently, the system has no concept of reputation, it assumes that everybody will play nicely, though it now has some limits to prevent major abuse (Max send/recpt rates, limited numbers of messages/size-of inbox).
I have Policyd configured, though that is currently tracking only connections (so I know that X last sent to Y at Z), and quotas (so I know X is sending/receiving at most Y messages/hour).

Fail2ban is already on the server, and is set to block postfix+dovecot failures. I see ~2 hosts/day blacklisted (though mostly from failed ssh attempts).

Spamassasin is currently in its 'dumb' mode using the bayesian classifier (I haven't yet gotten around to configuring it so it can 'learn'), with postfix passing messages through it upon receipt, then telling dovecot whether to mark as SPAM or INBOX. The ruleset is updated nightly via cron.

PS. Only 1 week of exams left, then I can finally finishing making the site so users can register/configure the forwards etc. themselves.

[hawnuyuna'viyä]

Finally got a site up with user registration and configuration.

Check it out at https://mail.skxawng.lu.

It could do with being prettified, and being pen-tested, but seems to work ok. (Anybody got some rails pen-testing experience?)

@mods: Probably time to get the first post updated with some useful information now.

HN

[Tirea Aean]

Finally got a site up with user registration and configuration.

Check it out at https://mail.skxawng.lu.

Sweet!

It could do with being prettified, and being pen-tested, but seems to work ok. (Anybody got some rails pen-testing experience?)

Nah man, no rails exp at all. Wish I could help.

@mods: Probably time to get the first post updated with some useful information now.

HN

Done.

[hawnuyuna'viyä]

@mods: Probably time to get the first post updated with some useful information now.

Done.

Irayo ma Tirea.

[Tirea Aean]

@mods: Probably time to get the first post updated with some useful information now.

Done.

Irayo ma Tirea.

Nìprrte'. Did I miss anything?

[hawnuyuna'viyä]

@mods: Probably time to get the first post updated with some useful information now.

Done.

Irayo ma Tirea.

Nìprrte'. Did I miss anything?

Nothing major. The line that states there are no backup MX servers is now out-of-date (since there are currently 2 active secondary mx servers [and a third will be added at some point soon, when I finish some other tasks]), but that is about it.

[Tirea Aean]

@mods: Probably time to get the first post updated with some useful information now.

Done.

Irayo ma Tirea.

Nìprrte'. Did I miss anything?

Nothing major. The line that states there are no backup MX servers is now out-of-date (since there are currently 2 active secondary mx servers [and a third will be added at some point soon, when I finish some other tasks]), but that is about it.

edited. Cool. Thanks for all the work you're putting into this.

[Mesireatu]

Kaltxì ma frapo!
Oel kameie ayngati. Anyone that can help a computer idiot with this? Irayo on beforehand! Kìyevame ulte Eywa ayngahu

[Swoka Ikran]

Kaltxì ma frapo!
Oel kameie ayngati. Anyone that can help a computer idiot with this? Irayo on beforehand! Kìyevame ulte Eywa ayngahu

What do you need help with?

You can register for an account here: https://mail.skxawng.lu/ Once you have an account, you need to set up some form of email software (Outlook, Thunderbird, etc.) so you can use it. Thunderbird is free. If you have a mobile device, the mail app that comes with it will usually work.

My Thunderbird configuration for skxawng.lu looks like this: http://i.imgur.com/j0aaIjJ.png (copy everything and substitute your own email and password). If it gives an error about the password being incorrect, click "Advanced config", then "Server settings" on the left, then change "Connection security" to "SSL/TLS" (it may be on this already) and "Authentication method" to "Normal password".

I did also make a webmail interface for my private use so I can check mail from friends' PCs (didn't have a smartphone until last week). If there's interest and hawnuyuna'viyä is OK with me posting it, I can change a few settings and post a link.

[Mesireatu]

Kaltxì ma frapo!
Oel kameie ayngati! Yes, I have a cell phone, but not a new one, so I can't use the app stuff.. I will take a peek on my work computer tomorrow. Ayirayo Il Sogno Viandante Kìyevame ulte Eywa ayngahu

[hawnuyuna'viyä]

If it gives an error about the password being incorrect, click "Advanced config", then "Server settings" on the left, then change "Connection security" to "SSL/TLS" (it may be on this already) and "Authentication method" to "Normal password".

This sounds like a mistake on my part, since I think SMTP auth should be happy with 'encrypted' passwords. (Though it is over STARTTLS anyway, so it is not a security problem). I will look into this at some point.

I did also make a webmail interface for my private use so I can check mail from friends' PCs (didn't have a smartphone until last week). If there's interest and hawnuyuna'viyä is OK with me posting it, I can change a few settings and post a link.

I have no problems with you posting your webmail interface for others to use. (Though of-course you will need to take reasonable steps to ensure it is not misused, but being on your own hardware I am sure you already do this.)
[It may prove useful for me to link to it from the domain anyway...]

[Swoka Ikran]

This sounds like a mistake on my part, since I think SMTP auth should be happy with 'encrypted' passwords. (Though it is over STARTTLS anyway, so it is not a security problem). I will look into this at some point.

Was wondering about this. Also, SMTP doesn't seem to accept an initially-encrypted connection. STARTTLS only, but then there's no real difference other than sending one extra command and enabling encryption after it.

I have no problems with you posting your webmail interface for others to use. (Though of-course you will need to take reasonable steps to ensure it is not misused, but being on your own hardware I am sure you already do this.)
[It may prove useful for me to link to it from the domain anyway...]

I'm running it on the same host that all my sigbar stuff is on. I wish I could run on my own hardware, but I don't have a decent connection to host on. Biggest concern will be space...if this gets significant use I'll may need to find somewhere else to host it.

I'm gonna offer the client I've been using personally (since its already there), and Roundcube (since there were requests for it a while back). They should be up tomorrow some time.

Also, is there a reason my account is not receiving any mail right now in any client (tried iPhone, T-bird, web)? I can see what's in my account, and can send fine, but I sent 2 test emails from Gmail, and 1 from my university account and didn't get any of them. And what is the max mailbox size for these accounts?