Kernel.org and the DDoS we nearly missed
Sep. 13th, 2010 05:50 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
warthog9So on Friday of last week I got a slightly frantic phone call from our US upstream data provider, ISC. I completely missed the calls, but when I checked my voicemail I was a little surprised to hear:
Errr, what?! As it turned out someone with a botnet decided to point it's impressive abilities at kernel.org by trying to flood it with completely random UDP traffic on any arbitrary port. According to ISC they were seeing nearly 3gbps (yes that's giga-bits per second) of incoming bandwidth being directly targeted at the two machines that service www, git, android.git, mirrors.git and a number of other sites. This could have gone very badly, but..
Strangely enough, no one reported any inability to get to the sites or problems getting data or anything. Those two boxes were seeing their entire incoming bandwidth full of a lot of garbage and they just kept trucking along. Loads didn't spike, memory usage stayed fairly consistent and we just kept going.
So my hat goes off to HP for donating us some dead rock solid hardware, those DL380 G5's we got a couple of years ago now are happily humming along being awesome. I will also heave a sigh of relief knowing this could have gone a lot worse. They could have targeted all of the machines, both US and Europe and both the www and mirrors boxes. They could have targeted some of the equipment we have at Oregon State.
Thankfully what they targeted was capable of keeping up with the onslaught, and our upstream providers were able to handle the sudden jump in traffic! For the record, I can't say enough good and awesome things about ISC being one of our upstream bandwidth providers and they handled the whole thing spectacularly.
Things thankfully quieted down over the weekend and stuff seems to be back to normal. We are keeping an eye on the bandwidth graphs right now, but suffice it to say we survived!
"Hey John, so it looks like your the subject of a DDoS attack, we just wanted to let you know and we are going to start blocking some traffic at our switches for you, give us a call back."
Errr, what?! As it turned out someone with a botnet decided to point it's impressive abilities at kernel.org by trying to flood it with completely random UDP traffic on any arbitrary port. According to ISC they were seeing nearly 3gbps (yes that's giga-bits per second) of incoming bandwidth being directly targeted at the two machines that service www, git, android.git, mirrors.git and a number of other sites. This could have gone very badly, but..
Strangely enough, no one reported any inability to get to the sites or problems getting data or anything. Those two boxes were seeing their entire incoming bandwidth full of a lot of garbage and they just kept trucking along. Loads didn't spike, memory usage stayed fairly consistent and we just kept going.
So my hat goes off to HP for donating us some dead rock solid hardware, those DL380 G5's we got a couple of years ago now are happily humming along being awesome. I will also heave a sigh of relief knowing this could have gone a lot worse. They could have targeted all of the machines, both US and Europe and both the www and mirrors boxes. They could have targeted some of the equipment we have at Oregon State.
Thankfully what they targeted was capable of keeping up with the onslaught, and our upstream providers were able to handle the sudden jump in traffic! For the record, I can't say enough good and awesome things about ISC being one of our upstream bandwidth providers and they handled the whole thing spectacularly.
Things thankfully quieted down over the weekend and stuff seems to be back to normal. We are keeping an eye on the bandwidth graphs right now, but suffice it to say we survived!