CONNECTION PROBLEMS CAUSE BLIP ON THURSDAY
by Hazed
If you found yourself unable to connect to Fed2 for a while on Thursday, we're sorry for the inconvenience, although it wasn't our fault. But finding out exactly what had caused the problem took a bit of detective work.
At first, we thought it was caused by a power cut in Manchester. That's where Fed2 lives now, since we moved to a data center called Melbourne there a few months back. On Thursday evening there were two major powercuts in the city. We knew about them because they affected another entertainment product. Rock band Oasis, who come from Manchester, were playing a gig to 70,000 in the city when the power outage hit them. The show had to be delayed for half an hour. As a result, the band's frontman Noel Gallagher promised that fans could claim their money back, so it became in effect a free gig. If all the fans had taken up that offer, it would have cost the band over three million pounds. Well, when you've made as much money as Oasis have over the years you can afford to be that generous!
So naturally, since the times matched, we thought the power outage in the city might have affected internet connections in and out. But no, it wasn't that. Here's the explanation, as discovered by Freya:
Outage report - 05.06.09
Unfortunately it’s not possible to adequately explain what went on without going into technical detail so I'll try to keep this simple while giving you a general explanation of what happened. (Network/IT professionals can email me at: barb at ibgames.com for specific details if desired.)
First of all, in a complex system like the internet, problems are generally caused by a cascade of smaller issues, each affecting the next and increasing the magnitude. This problem took place in the backbone of the internet, the core that everything runs off. It never touched the ibgames server itself, which actually was up and running the whole time although none of us could reach it. Also, I can only tell you what happened on our end of the problem, not who started it or how or how many other services were affected elsewhere in the US and the rest of the world.
Part of the structure of the internet comprises network machines called routers advertising routes (called BGP sessions) to guide data between ISPs. In this case, our problem was triggered by someone in the US putting a "rogue" (malformed) route onto the global routing table. When this incorrect information was received by our Host ISP's border routers (they are the ones facing the internet) it caused instability making the connection go up and down repeatedly (this is called "flapping"). The software the border routers was running was affected by the rogue route. Our Host ISP restored full connectivity by upgrading some of their border routers to a later version of the routing software, instantly fixing the problem and stabilizing their connectivity. This fix restored everyone's connectivity to the ibgames server which provides not only the game Federation 2 but also our website and ibgames email.
So there you have it. Nothing to do with Manchester power cuts - it was just a strange coincidence that Manchester blacked out at the same time as something going wrong on the other side of the Atlantic messed up internet routes.
|
