warthog9: Warthog9 (Default)

For the better part of a decade, I've been working on kernel.org. About half that time, I was a completely uncompensated volunteer, as that's how kernel.org was run until 2008. The remainder of that time I did the same work as a Fellow under the auspices of the Linux Foundation. When I say work, I mean much more than just holding a 9-5 job; I mean pouring blood, sweat, and tears into the inner workings keeping kernel.org running.

Thanks to the Linux Foundation, and the individuals and companies that support them, I've been able to work on kernel.org full time for the past 4 years. However, I have been doing some soul searching lately and I've realized the time has come to move on.

I'm sad to be leaving kernel.org - I've met a lot of people and made friends within the many communities I've dealt with over the years. (For the record, all of you are all still invited over for bad movies and dinner anytime!)

kernel.org has always been more than just work for me. It’s been a passion and it will always hold a special place for me. It was never so much "work", as work has the wrong connotations. It’s been the very essence of the phrase “labor of love” - something I did because I loved doing it. The users and community are amazing; I will always remember when I asked for folks to gc their git trees and the kernel developers independently turned it into a contest to see whose tree shrank the most.

That being said, I'm going to take the rest of August off and relax, catch up on some open source stuff I haven't been able to get to (PXE Knife, maybe working on the stateful rsync stuff I've been pondering for years), and start considering what comes next. I will be down for a couple of days at LinuxCon, and then I'm off to the Penny Arcade Expo (PAX) up in Seattle late on the 30th (which also happens to be my birthday).

Beyond that, it's time to see what the future holds.

- John 'Warthog9' Hawley

warthog9: Warthog9 (Default)
So on Friday of last week I got a slightly frantic phone call from our US upstream data provider, ISC.  I completely missed the calls, but when I checked my voicemail I was a little surprised to hear:

"Hey John, so it looks like your the subject of a DDoS attack, we just wanted to let you know and we are going to start blocking some traffic at our switches for you, give us a call back."

Errr, what?!  As it turned out someone with a botnet decided to point it's impressive abilities at kernel.org by trying to flood it with completely random UDP traffic on any arbitrary port.  According to ISC they were seeing nearly 3gbps (yes that's giga-bits per second) of incoming bandwidth being directly targeted at the two machines that service www, git, android.git, mirrors.git and a number of other sites.  This could have gone very badly, but..

Strangely enough, no one reported any inability to get to the sites or problems getting data or anything.  Those two boxes were seeing their entire incoming bandwidth full of a lot of garbage and they just kept trucking along.  Loads didn't spike, memory usage stayed fairly consistent and we just kept going.

So my hat goes off to HP for donating us some dead rock solid hardware, those DL380 G5's we got a couple of years ago now are happily humming along being awesome.  I will also heave a sigh of relief knowing this could have gone a lot worse.  They could have targeted all of the machines, both US and Europe and both the www and mirrors boxes.  They could have targeted some of the equipment we have at Oregon State.

Thankfully what they targeted was capable of keeping up with the onslaught, and our upstream providers were able to handle the sudden jump in traffic!  For the record, I can't say enough good and awesome things about ISC being one of our upstream bandwidth providers and they handled the whole thing spectacularly.

Things thankfully quieted down over the weekend and stuff seems to be back to normal.  We are keeping an eye on the bandwidth graphs right now, but suffice it to say we survived!

warthog9: Warthog9 (Default)
Computer Science has only 3 numbers that matter.
  • 0 - There is nothing to worry about, this is easy!
  • 1 - Nearly as easy as 0, things are simple, doing things once isn't so hard & honestly this is how most people think.
  • Many / Infinite - This is where things get a little tricky. The step from 0 to 1 isn't a huge jump, but the step from 1 to Many is a doozy.
I've got maybe 60-70% of the infrastructure in place now to deal with parallel mediawiki installs on korg now, and a lot of grunt work done with respect to php.
warthog9: Warthog9 (warthog9)
UPDATE July 6th, 2011: Updated to include a *POSSIBLE* fix from Microsoft about the problem.  I have no way of testing or confirming it, but it sounds like it has potential.
UPDATE March 2, 2010:
Updated to include, now known, actual cause of the problem -JH

Recently I've had the exciting job of debugging a massive e-mail failure between kernel.org and a large hardware manufacturer (LHM). The LHM has been particularly nice about this, understand the problem and has been kind enough to do my evil bidding while I try to debug this whole problem with only one side of the entire equation to work with.

Here's the problem: The LHM has contracted all of their e-mail out to Microsoft's Exchange Hosted Service (EHS). Generally speaking this is fine, it saves LHM a lot of money in their IT budget, the mail servers are more up to date, generally more secure, etc. Really I don't have a lot to complain about this choice - except - when it fails.

So the long version of this story is, back in December LHM's couple of employees who need to e-mail kernel.org on a regular basis started noticing that their e-mails weren't getting anywhere, and in fact where getting bounced back at them with some strange errors. They didn't think *TOO* much about it at the time, but the problem persisted and around mid-January they came to kernel.org and asked what we were seeing. I figured it was just a problem with our greylisting, we are a little more aggressive about it than others and it catches up some mail servers. So I did #include <STD_EMAIL_GREYLIST.h> response and expected the problem to go away. Sadly it didn't, so I opened up a proper look into what was going on.

LHM's employees were quite forthcoming with what they knew about how their e-mail worked, mainly that it was a giant black box that no one at LHM could see into because it was a Microsoft run service (known as Bigfish). I asked kindly if they could get me the logs from their box, or at least all of the error messages and I started delving into my logs.

That's when I noticed something very weird.

Feb XX YY:58:03 hera sendmail[32520]: STARTTLS=server, error: accept failed=-1, SSL_error=5, errno=104, retry=-1
Feb XX YY:58:03 hera sendmail[32520]: o1JKw24C032520: va3ehsobe005.messaging.microsoft.com [] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA

Hmmmmm ok, that's odd. No really that's very odd... This had been happening since Decemberish or so. Great. But I was having the employees from LHM e-mail both kernel.org, and my personal domains and the e-mails to my personal domains were going through - why?

I started checking the certs, I started monitoring more things then I paid slightly closer attention to my personal server's mail logs, my primary mail server was exhibiting the same problem that kernel.org was - ok at least I'm not insane, but why was I getting the e-mail? I looked at my secondary mail servers, and one of them - my longest running box

# uptime
13:02:45 up 1003 days, 15:27, 1 user, load average: 0.06, 0.11, 0.09

Was actually receiving the mail, and actually accepting it. It would spool on that machine and then push to my primary without issue. Hmmmm......

So I turned up the debugging levels on kernel.org and had LHM's employees send me more e-mails. This is what I found:
Feb  4 07:17:37 hera sendmail[22509]: NOQUEUE: connect from
va3ehsobe005.messaging.microsoft.com []
Feb  4 07:18:13 hera sendmail[22509]: AUTH: available mech=NTLM GSSAPI
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: Milter (clamav):
init success to negotiate
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: Milter
(spamassassin): init success to negotiate
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: Milter (greylist):
init success to negotiate
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: Milter: connect to
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: milter=clamav,
action=connect, continue
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509:
milter=spamassassin, action=connect, continue
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: milter=greylist,
action=connect, continue
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 220
hera.kernel.org ESMTP Sendmail 8.14.3/8.14.3; Thu, 4 Feb 2010 07:18:13 GMT
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: <-- EHLO
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509:
milter=spamassassin, action=helo, continue
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: milter=greylist,
action=helo, continue
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: ---
250-hera.kernel.org Hello va3ehsobe005.messaging.microsoft.com
[], pleased to meet you
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: ---
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250-PIPELINING
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250-8BITMIME
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250-SIZE
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250-DSN
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250-ETRN
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250-AUTH
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250-STARTTLS
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250-DELIVERBY
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250 HELP
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: <-- STARTTLS
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 220 2.0.0
Ready to start TLS
Feb  4 07:18:14 hera sendmail[22509]: STARTTLS=server, info: fds=6/4, err=5
Feb  4 07:18:14 hera sendmail[22509]: STARTTLS=server, error: accept
failed=-1, SSL_error=5, errno=104, retry=-1
Feb  4 07:18:14 hera sendmail[22509]: o147Hb7C022509:
va3ehsobe005.messaging.microsoft.com [] did not issue
MAIL/EXPN/VRFY/ETRN during connection to MTA
and doing some quick deciphering on the error returned by sendmail's
attempted TLS connection I get the following:
	for more on what SSL_ERROR_SYSCALL entails.  Short
	version is that it's an error reported by the under
	lying I/O layer, and to check what errono was returned
errno=104 | ECONNRESET
	#define ECONNRESET      104     /* Connection reset by peer */
	/usr/include/asm-generic/errno.h which is provided by the Linux
	kernel itself (as it's handling the TCP/IP stack here)

Long story short, this looks like bigfish opens up a connection to my mail server, attempts to start TLS (secure connection) and then bigfish sends a reset and sendmail goes "Ergh, well I guess we didn't want to chat afterall".

I've sent all of that to LHM, and as weird as this is going to sound - I'm now working through Microsoft's support side of things (I think I'm in Layer-2 with this, the Layer-1 guy was keen enough to realize we were way outside the depth of his script and bumped us up) to get this resolved. It's kind of weird to be the Chief Administrator for kernel.org and be debuging Microsoft's Exchange mail server for them - brings a smile to my face in a lot of ways.

Some additional details should others be seeing this or want to know more:

  • Openssl versions 0.9.8g and before seem to work fine, this would include my ancient Fedora Core 5 secondary mail server
    • Redhat Enterprise Linux (RHEL) / Centos 5 ship with Openssl 0.9.8e (this seems to work)
    • Debian Lenny is at 0.9.8g (which was confirmed by an employee from LHM to work)
    • Fedora <= 10 ships with 0.9.8g or earlier (This is known to work)
    • Fedora >= 11 ships with 0.9.8k (This is known to NOT work)
    • Ubuntu 9.10 may have 0.9.8k in it - this is known NOT to work
    • Opensuse has 0.9.8k in it - this is known NOT to work
  • Openssl has not been updated on my Fedora 11 or Fedora 12 boxes, this is not a change on my server side as there have been no updates to those packages that have been pushed down through Fedora's update process.
  • Disabling TLS negotiation in sendmail alleviates the problem, but I'm not keen to disable it universally if I can.  I generally think that TLS is a good thing, and that MTA <-> MTA communications should at least have some modicum of security (even if it's not perfect)
  • There are Microsoft domains that can send e-mail to these systems fine, they however do not attempt TLS/SSL negotiation
I'll update things here if/when I found out more information.  I'm honestly unsure that anything will come of all of this, I've provided Microsoft with packet dumps of the connection attempts and I've given them all of the information above.  I'm not entirely sure how much more debugging I can do for them (Microsoft) on this issue just because I'm not sure there's anywhere else for me to go at this point.

I'm going to post this in the hopes that if someone Googles for some of this they have at least some explanation of what's going on.  I would also argue that the only reason this isn't more wide spread is Systems Administrators have a tendency to be stodgy and slow moving with their mail servers, and I doubt there are really that many out there running something as new as Fedora 11, I would guess the vast majority are on something like RHEL/Centos or Debian which aren't affected - yet.


Ok so I got lucky on this one and I can now point out the exact and specific issue with the whole thing.  I've struck out a chunk of data above as it's not relevant any more (and actually wrong now that I know the exact cause).

So after going through all of the above, the LHM helping push Microsoft on the issue and for the record, to date, I still have not seen a *SINGLE* line of log information from Microsoft, I've seen no good analysis from them and to be perfectly honest I ended up in a Copy/Paste war with them while they were trying to analyse the logs I had sent them - which I thoroughly walked through the whole thing with way more detail and insight and Microsoft latched onto something completely unrelated and blamed my whole SSL negotiation issue on Sendmail's inability to write out statistical data.  Needless to say I was unamused and very very frustrated with the whole situation.

However, I got lucky.  One of the other kernel.org administrators (Kees Cook specifically) happened to pipe up on the problem, and comment that it looked exactly like a problem he had only recently been debugging with OpenSSL and GnuTLS.  After some quick checking I can confirm that the problem *IS* a bug on Microsoft's side, but it's one that's caused by a specific setting on the end that Microsoft is talking to.

The problem, as excruciatingly detailed above, comes into play during the SSL/TLS negotiation, specifically in the Certificate Authority section.  When you send the certificate you also send the Certificate Authority certificate, both for verification of the CA but for verification of the cert itself.  The default on most systems (particularly Fedora) is to send the entire CA bundle that the OS ships with.  This is a sledgehammer approach that should work for most everyone, and it's understandable why it ships this way: Most CA's are included in the bundle and it just works.  The problem comes in when the size of the bundle gets too large.

On my older Fedora secondary mail server (the one with the obscene uptime) has about a 418K in size, on my F11 boxes it is 654K.  Not a huge change, but it's enough to tickle this problem: when the CA cert grows too large the remote side wigs out and can't handle it.  This was seen by Kees with GnuTLS, when it got the bundle it would more or less exhaust the buffer and just die.  Thankfully he had access to both sides of this problem and could reasonably debug it.  I did a couple of small checks and low and behold - that was the problem.  So what I ended up doing was changing our CA cert that sendmail is passing back to be our same certificate.  Why?  It's a self signed cert, and we don't really have a CA that's signed it so I'm not terribly worried about it and *VERY* few (probably no) mail servers try to verify the cert before sending the mail, mainly because mail is handled at a central server, there's no one to verify the cert manually.

So there you have it, the bug is present in EHS, it's trivially fixable by the end that is Microsoft is talking to (in my case kernel.org) but for this problem to really go away it needs to be fixed in Exchange, I.E. Microsoft needs to fix this.  I've handed all of this information to Microsoft and the LHM, MS assures me there is a bug filed with the Exchange team on it but there's no ETA on when this will be fixed - my guess is somewhere around 2020, but I might be being pessimistic.

This now serves as a one stop source for this problem, hopefully others will find it should they run into this problem - I can only hope.

Was just contacted by the company that had the problem originally as a potential "fix" from Microsoft may have some impact on this:


What's interesting about it is that it talks about TLS/SSL fragmentation.  Now that said they do talk about the negotiation " TLS/SSL handshake messages become too large to be contained in a single packet " which sounds a little odd, and I garauntee you the stage at which we are getting the CA cert is being sent over many packets (CA cert that worked being 418K and the max MTU is 1.5K).  Also TLS/SSL for most uses is used via TCP/IP where you aren't particularly worried about the size of a single packet.  But I begin to wander and digress.

If you find this, and you are having this problem give the KnowledgeBase article a read, possibly even give it a shot.  If it works let me know, I would actually be fascinated to know.

Thanks to renormalist for giving me the heads up on the KB article!
warthog9: Warthog9 (Default)
So kernel.org has gotten up to the point where it's 10 some machines in size and scope with general plans to grow that number of machines as needed. Now at 10 machines copying around configuration files, directories, etc is a little unwieldy but it isn't so cumbersome as to really make me want to change things. Generally speaking our configurations are very static, we don't add new machines very often and our configurations don't change much over time.

That said we recently started work on deploying an 11th machine for a specific project, we are still feeling out the project so I can't say much about it yet, but I figured it was time to start trying to centralize the configuration for kernel.org into something a bit more centrally managed and controlled.

Why? Well designating certain machines as 'masters' of a configuration is fine, but it still means machines could diverge in ways I'm not expecting. It's also hard to explain to everyone else who has to work on these machines, and ultimately it's not really a good thing for long term sustainability. Plus I wanted to play with puppet on a larger scale.

So as of today I've got now 2 machines hooked up to puppet, our backend machine that deals with our dynamic web content (demeter) and our mystery project box (nott*). I've got the following things ported over:
  • smtp / mail
    • sendmail
    • greylisting
    • clamav
  • ntp
  • puppet (itself)
  • rsync
  • sudo
  • yum-updatesd
While I recognize this is not a particularly impressive list, this is what I've gotten through in only a couple of days worth of work, and our mail setup - is *NOT* simple (there's over 1M of configuration data to deal with that alone). I'm at the stage where I'm just taking our existing configuration files and copying them around, in a lot of cases this is likely how it will just be, but I can see where there would be some much more interesting setups using their templates and such. The example that immediately floats to mind is our configuration files for vsftpd - which don't allow us to store all the possible ports to listen on in the configuration file. It sucks, most everything else doesn't care, but vsftpd does.

That said I have been pondering something that I would genuinely love: the ability to include reported information from the clients in templates.

Lets say I have a list of machines that are connected to puppet:
  • machineA
  • machineB
  • machineC
  • machineD
  • machineE
Lets also say that I've applied the class 'greylist' to all of these machines. Now in a puppet template I can include information about that specific machine in the configuration file, say the IP address of the machine. This is very useful indeed, I can trivially tell them which interface to listen on and they are good to go. However what if I want to have all of the machines (nodes) that belong to class 'greylist' (or that have imported it) listed in the configuration file?

So far I can't find a way to do this, but it would be extremely useful and it might be something that I implement outside of puppet and do some more basic queries on my own, it's just annoying that this doesn't already exist inside puppet. That said I'll happily admit to being a n00b when it comes to puppet, and I fully expect to re-write the rules that I've already written in the near future (in fact I think one of the rules has already gone through 3 revisions). I'll likely post some more commentary on this as I get further, come up with something novel, etc.

* For those interested this is the Goddess of Night from Norse Mythology http://www.godchecker.com/pantheon/norse-mythology.php?deity=NOTT


warthog9: Warthog9 (Default)

December 2014

141516 17181920


RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Apr. 27th, 2017 03:17 am
Powered by Dreamwidth Studios