Feb. 19th, 2010

warthog9: Warthog9 (Default)
So I've had a problem adapting to the Nexus One these past couple of weeks. It works great, it's super fast, the screen is bright, clear and astounding. But I've run into two problems that are continuously causing me to scratch my head and go 'ERGH?'
  1. There's not enough room for apps.
    So I moved over from my hacked G1 to my hacked N1, and immediately found I couldn't port over all of the apps I had installed. This isn't a huge deal in the grand scope of things, it just means that I don't have the bazillion tower defence games on my device - now I only have a million or so ;-)
  2. Where's the goram frelling TAB key?!?!?!?!?!?!?!
    So apparently I use the TAB key a lot.  Like daily, in nearly everything I do
    1. When I want to write a grocery list I will most often open up a note of some sort add one item per line and when I get it at the store I tab it over so that it's not completely left justified anymore
    2. When I want to align text, in notes, in my calendar, anywhere
    3. When I'm using ssh - *ESPECIALLY* when I'm using ssh
So obviously 1 there isn't worrying me too much, Google has already said they will support running apps off the SD Card at some point in the future, so I'm not terribly worried.  As for 2 however - I'm astounded.  I can't be the only person who wants a keyboard with a tab key.  My G1 has one on the physical keyboard, so does the Motorola Droid/Milestone.  Yet there is no tab on the virtual keyboard, not as a shift character, not in the symbols, and not as an alternative in the symbols.  I mean seriously I get a character that looks like the new Netapp logo, but I don't get a tab key?!

So this started my adventure into the Android Market.  I've generally had good luck rummaging around in there (except when I'm in Canada and I want to buy an app, come on Google let people North of the Border buy apps!  It's either that or I use my proxy / vpn).  So I figured, hey the keyboard on Android is replaceable lets see what they've got.

One search for 'keyboard' later I noticed there are something like 600 (I counted) applications for keyboards!  I mean there's what, 30,000 apps on the Market and 2% of them are keyboards - that's pretty impressive... but wait a second, I thought to myself looking just a smidgen closer, some thing is amiss here.....

Most of these "Apps" are themes, or skins, or whatever they want to call them for a single application: Better keyboard.  To find what I was really after I had to wade through page after page, after page, after page of "Applications" to find the maybe 10 or so keyboards in the market to try and find one with a tab key ( For the record AnySoftKeyboard is what I finally found that had a tab key and was, more or less, usable).

This is when it dawned on me: the market idea, as it currently stands, is doomed.

I'm not being a naysayer here, in fact I really like the idea of the Market, or the Apple App store (for definitions of like and closed minded we can bounce your app if we don't like it for any reason, like it was sunny in California today).  The ideas are quite sound: give the user a one stop shopping place to find apps to make their device more awesome, slower and in need of being upgraded.  Great!  But here's the catch, for things like Better Keyboard they offer themes or skins or whatever they are, and each of these ends up taking up a slot in the market and thus becomes the problem: There's so much random stuff, I really don't care about, that I can't find what I'm looking for.

Why are there apps that show you pictures of attractive individuals?  Isn't that what the web is for?  Why isn't there a way for people to say "I'm a theme of Better Keyboard" and have all of those "Apps" collapse under the general heading of better keyboard?  Why must the Market be such a disastrous mess I can't even find what I'm looking for?

What we really need is better organization, for things like themes they should be more directly attached to the application, heck even a little sub-category for them all would work.  Click on Better Keyboard and you can install it and after the comments section there's a listing of all the themes for the application, which take you to their individual pages, etc.  But the entire thing only searches as a single entity when I search the market.

Dunno, I'm obviously frustrated at the market for making me wade through more "Apps" purporting to be keyboards than I have tower defence games, and I'm doubly frustrated 'cause all I really wanted was a tab key.  Are those really so much to ask for?

warthog9: Warthog9 (warthog9)
UPDATE July 6th, 2011: Updated to include a *POSSIBLE* fix from Microsoft about the problem.  I have no way of testing or confirming it, but it sounds like it has potential.
UPDATE March 2, 2010:
Updated to include, now known, actual cause of the problem -JH

Recently I've had the exciting job of debugging a massive e-mail failure between kernel.org and a large hardware manufacturer (LHM). The LHM has been particularly nice about this, understand the problem and has been kind enough to do my evil bidding while I try to debug this whole problem with only one side of the entire equation to work with.

Here's the problem: The LHM has contracted all of their e-mail out to Microsoft's Exchange Hosted Service (EHS). Generally speaking this is fine, it saves LHM a lot of money in their IT budget, the mail servers are more up to date, generally more secure, etc. Really I don't have a lot to complain about this choice - except - when it fails.

So the long version of this story is, back in December LHM's couple of employees who need to e-mail kernel.org on a regular basis started noticing that their e-mails weren't getting anywhere, and in fact where getting bounced back at them with some strange errors. They didn't think *TOO* much about it at the time, but the problem persisted and around mid-January they came to kernel.org and asked what we were seeing. I figured it was just a problem with our greylisting, we are a little more aggressive about it than others and it catches up some mail servers. So I did #include <STD_EMAIL_GREYLIST.h> response and expected the problem to go away. Sadly it didn't, so I opened up a proper look into what was going on.

LHM's employees were quite forthcoming with what they knew about how their e-mail worked, mainly that it was a giant black box that no one at LHM could see into because it was a Microsoft run service (known as Bigfish). I asked kindly if they could get me the logs from their box, or at least all of the error messages and I started delving into my logs.

That's when I noticed something very weird.

Feb XX YY:58:03 hera sendmail[32520]: STARTTLS=server, error: accept failed=-1, SSL_error=5, errno=104, retry=-1
Feb XX YY:58:03 hera sendmail[32520]: o1JKw24C032520: va3ehsobe005.messaging.microsoft.com [] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA

Hmmmmm ok, that's odd. No really that's very odd... This had been happening since Decemberish or so. Great. But I was having the employees from LHM e-mail both kernel.org, and my personal domains and the e-mails to my personal domains were going through - why?

I started checking the certs, I started monitoring more things then I paid slightly closer attention to my personal server's mail logs, my primary mail server was exhibiting the same problem that kernel.org was - ok at least I'm not insane, but why was I getting the e-mail? I looked at my secondary mail servers, and one of them - my longest running box

# uptime
13:02:45 up 1003 days, 15:27, 1 user, load average: 0.06, 0.11, 0.09

Was actually receiving the mail, and actually accepting it. It would spool on that machine and then push to my primary without issue. Hmmmm......

So I turned up the debugging levels on kernel.org and had LHM's employees send me more e-mails. This is what I found:
Feb  4 07:17:37 hera sendmail[22509]: NOQUEUE: connect from
va3ehsobe005.messaging.microsoft.com []
Feb  4 07:18:13 hera sendmail[22509]: AUTH: available mech=NTLM GSSAPI
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: Milter (clamav):
init success to negotiate
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: Milter
(spamassassin): init success to negotiate
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: Milter (greylist):
init success to negotiate
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: Milter: connect to
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: milter=clamav,
action=connect, continue
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509:
milter=spamassassin, action=connect, continue
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: milter=greylist,
action=connect, continue
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 220
hera.kernel.org ESMTP Sendmail 8.14.3/8.14.3; Thu, 4 Feb 2010 07:18:13 GMT
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: <-- EHLO
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509:
milter=spamassassin, action=helo, continue
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: milter=greylist,
action=helo, continue
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: ---
250-hera.kernel.org Hello va3ehsobe005.messaging.microsoft.com
[], pleased to meet you
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: ---
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250-PIPELINING
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250-8BITMIME
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250-SIZE
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250-DSN
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250-ETRN
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250-AUTH
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250-STARTTLS
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250-DELIVERBY
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 250 HELP
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: <-- STARTTLS
Feb  4 07:18:13 hera sendmail[22509]: o147Hb7C022509: --- 220 2.0.0
Ready to start TLS
Feb  4 07:18:14 hera sendmail[22509]: STARTTLS=server, info: fds=6/4, err=5
Feb  4 07:18:14 hera sendmail[22509]: STARTTLS=server, error: accept
failed=-1, SSL_error=5, errno=104, retry=-1
Feb  4 07:18:14 hera sendmail[22509]: o147Hb7C022509:
va3ehsobe005.messaging.microsoft.com [] did not issue
MAIL/EXPN/VRFY/ETRN during connection to MTA
and doing some quick deciphering on the error returned by sendmail's
attempted TLS connection I get the following:
	for more on what SSL_ERROR_SYSCALL entails.  Short
	version is that it's an error reported by the under
	lying I/O layer, and to check what errono was returned
errno=104 | ECONNRESET
	#define ECONNRESET      104     /* Connection reset by peer */
	/usr/include/asm-generic/errno.h which is provided by the Linux
	kernel itself (as it's handling the TCP/IP stack here)

Long story short, this looks like bigfish opens up a connection to my mail server, attempts to start TLS (secure connection) and then bigfish sends a reset and sendmail goes "Ergh, well I guess we didn't want to chat afterall".

I've sent all of that to LHM, and as weird as this is going to sound - I'm now working through Microsoft's support side of things (I think I'm in Layer-2 with this, the Layer-1 guy was keen enough to realize we were way outside the depth of his script and bumped us up) to get this resolved. It's kind of weird to be the Chief Administrator for kernel.org and be debuging Microsoft's Exchange mail server for them - brings a smile to my face in a lot of ways.

Some additional details should others be seeing this or want to know more:

  • Openssl versions 0.9.8g and before seem to work fine, this would include my ancient Fedora Core 5 secondary mail server
    • Redhat Enterprise Linux (RHEL) / Centos 5 ship with Openssl 0.9.8e (this seems to work)
    • Debian Lenny is at 0.9.8g (which was confirmed by an employee from LHM to work)
    • Fedora <= 10 ships with 0.9.8g or earlier (This is known to work)
    • Fedora >= 11 ships with 0.9.8k (This is known to NOT work)
    • Ubuntu 9.10 may have 0.9.8k in it - this is known NOT to work
    • Opensuse has 0.9.8k in it - this is known NOT to work
  • Openssl has not been updated on my Fedora 11 or Fedora 12 boxes, this is not a change on my server side as there have been no updates to those packages that have been pushed down through Fedora's update process.
  • Disabling TLS negotiation in sendmail alleviates the problem, but I'm not keen to disable it universally if I can.  I generally think that TLS is a good thing, and that MTA <-> MTA communications should at least have some modicum of security (even if it's not perfect)
  • There are Microsoft domains that can send e-mail to these systems fine, they however do not attempt TLS/SSL negotiation
I'll update things here if/when I found out more information.  I'm honestly unsure that anything will come of all of this, I've provided Microsoft with packet dumps of the connection attempts and I've given them all of the information above.  I'm not entirely sure how much more debugging I can do for them (Microsoft) on this issue just because I'm not sure there's anywhere else for me to go at this point.

I'm going to post this in the hopes that if someone Googles for some of this they have at least some explanation of what's going on.  I would also argue that the only reason this isn't more wide spread is Systems Administrators have a tendency to be stodgy and slow moving with their mail servers, and I doubt there are really that many out there running something as new as Fedora 11, I would guess the vast majority are on something like RHEL/Centos or Debian which aren't affected - yet.


Ok so I got lucky on this one and I can now point out the exact and specific issue with the whole thing.  I've struck out a chunk of data above as it's not relevant any more (and actually wrong now that I know the exact cause).

So after going through all of the above, the LHM helping push Microsoft on the issue and for the record, to date, I still have not seen a *SINGLE* line of log information from Microsoft, I've seen no good analysis from them and to be perfectly honest I ended up in a Copy/Paste war with them while they were trying to analyse the logs I had sent them - which I thoroughly walked through the whole thing with way more detail and insight and Microsoft latched onto something completely unrelated and blamed my whole SSL negotiation issue on Sendmail's inability to write out statistical data.  Needless to say I was unamused and very very frustrated with the whole situation.

However, I got lucky.  One of the other kernel.org administrators (Kees Cook specifically) happened to pipe up on the problem, and comment that it looked exactly like a problem he had only recently been debugging with OpenSSL and GnuTLS.  After some quick checking I can confirm that the problem *IS* a bug on Microsoft's side, but it's one that's caused by a specific setting on the end that Microsoft is talking to.

The problem, as excruciatingly detailed above, comes into play during the SSL/TLS negotiation, specifically in the Certificate Authority section.  When you send the certificate you also send the Certificate Authority certificate, both for verification of the CA but for verification of the cert itself.  The default on most systems (particularly Fedora) is to send the entire CA bundle that the OS ships with.  This is a sledgehammer approach that should work for most everyone, and it's understandable why it ships this way: Most CA's are included in the bundle and it just works.  The problem comes in when the size of the bundle gets too large.

On my older Fedora secondary mail server (the one with the obscene uptime) has about a 418K in size, on my F11 boxes it is 654K.  Not a huge change, but it's enough to tickle this problem: when the CA cert grows too large the remote side wigs out and can't handle it.  This was seen by Kees with GnuTLS, when it got the bundle it would more or less exhaust the buffer and just die.  Thankfully he had access to both sides of this problem and could reasonably debug it.  I did a couple of small checks and low and behold - that was the problem.  So what I ended up doing was changing our CA cert that sendmail is passing back to be our same certificate.  Why?  It's a self signed cert, and we don't really have a CA that's signed it so I'm not terribly worried about it and *VERY* few (probably no) mail servers try to verify the cert before sending the mail, mainly because mail is handled at a central server, there's no one to verify the cert manually.

So there you have it, the bug is present in EHS, it's trivially fixable by the end that is Microsoft is talking to (in my case kernel.org) but for this problem to really go away it needs to be fixed in Exchange, I.E. Microsoft needs to fix this.  I've handed all of this information to Microsoft and the LHM, MS assures me there is a bug filed with the Exchange team on it but there's no ETA on when this will be fixed - my guess is somewhere around 2020, but I might be being pessimistic.

This now serves as a one stop source for this problem, hopefully others will find it should they run into this problem - I can only hope.

Was just contacted by the company that had the problem originally as a potential "fix" from Microsoft may have some impact on this:


What's interesting about it is that it talks about TLS/SSL fragmentation.  Now that said they do talk about the negotiation " TLS/SSL handshake messages become too large to be contained in a single packet " which sounds a little odd, and I garauntee you the stage at which we are getting the CA cert is being sent over many packets (CA cert that worked being 418K and the max MTU is 1.5K).  Also TLS/SSL for most uses is used via TCP/IP where you aren't particularly worried about the size of a single packet.  But I begin to wander and digress.

If you find this, and you are having this problem give the KnowledgeBase article a read, possibly even give it a shot.  If it works let me know, I would actually be fascinated to know.

Thanks to renormalist for giving me the heads up on the KB article!


warthog9: Warthog9 (Default)

December 2014

141516 17181920

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Apr. 27th, 2017 03:20 am
Powered by Dreamwidth Studios