There’s been a bit of discussion about how much this DDOS actually slowed down the Internet globally. Fact is that the Internet didn’t come to a halt but the large amount of new traffic that had to be handled by some of the carriers did result in congestion and significant packet loss by some of the Tier1 carriers last weekend. In this blog post we’ll look at this event from the routing perspective, what effects did this have on the Internet Exchanges and we’ll also look at some BGP hijacks related to this attack.

BGP hijack affecting Spamhause
The majority of the attack towards SpamHaus and cloudflare was a brute-force DDOS of attack. But in an attempt to affect spamhause services different techniques were used, one of them was a BGP hijack by the alleged initiator of the attack. Greenhost.nl has a great description on their blog about how AS34109 Cyberbunker/CB3Rob (the alleged organizer of the spamhause attack), announced a more specific route for one of the spamhaus servers: 0.ns.spamhaus.org with IP address 204.16.254.40/32.

As this was a more specific route everyone receiving this route would have routed to the Cyberbunker server where each spamhaus query was then reported as a spammer.

Cyberbunker/CB3Rob announced this prefix on the NL-IX an Internet Exchange in the Netherlands. This means that networks that peer with 34109 on the NLIX would have received this prefix and most likely used this “fake” spamhaus server.
It’s obvious that an attack like this takes very little resources (as compared to the large scale DDOS) and can be very effective.

When we look at the history of AS34109 and AS51787, both assigned to Cyberbunker, we see that there were previous BGP incidents. We recorded two recent incidents during which AS34109 announced prefixes that are not assigned to them (Hijack). On January 9 AS34109 briefly announced 205.189.74.0/24 and between March 7 and March 21 the prefix 205.19.72.0/23 was announced by AS34109. This prefix is assigned to the US department of Defense and normally not announced to the Internet. It’s currently listed  in spamhaus so it’s likely that this prefix was used to send spam.

Cyberbunker prefixes have also been on the other side of the story, on February 6th some of their IP addresses in the 84.22.106.0/24 range were announced as /32’s inside the TATA communications (AS6453) network most likely because these were being null routed by TATA.

Internet Exchange point route announcements
One of the things that made this attack unique as compared to previous DDOS attacks was that the attackers decided to attack critical Internet infrastructure such as Internet Exchange points that are used by cloudflare and its upstream providers. This allowed the attacker to bypass the Anycast infrastructure of Cloudflare and focus the attack towards a limited set of devices resulting in a lot of  (new) traffic towards a very specific point on the Internet.

Normally these addresses used on the Exhange points should not be globally reachable or at least do it in such a way that it can easily be controlled and withdrawn. This is done specifically to prevent attacks towards the Internet Exchanges. Unfortunately this was not the case for two of the major Internet Exchanges in London, which was one of the reasons why these attacks towards the cloudflare routers was so intrusive.
Looking at the BGP data for the last few days we can see that both London Internet exchanges have now de-aggregated their announcements so that it’s easier to control announcements for the peering networks in London. Now all that remains is stopping members from announcing this prefix.

This week we’ve seen that DDOS are a big problem on the Internet and it’s unlikely this will go away any time soon, in fact the volume just keeps growing. This event also showed that DDOS can sometimes hide other attacks, as DDOS are very visible and intrusive it will typically get the attention of network and security engineers with the risk that it can be used to hide other forms of attacks such as intrusion attempt and in this case BGP hijacks.

Certificate authorities and SSL
Just as the DigiNotar storm seemed to have calmed down, Google announced they discovered, yet another Certificate Authority that was involved in a similar incident. TURKTRUST, a certificate authority, mis-issued two intermediate certificates that were later used to intercept SSL traffic to Gmail. In cases like this the attacker is interested in intercepting communication between Gmail users and the Gmail servers. In order to successfully execute such an attack the attacker will need to insert his fake Gmail impersonating webserver between the user and the actual Gmail servers, this is what we call a Man in the Middle Attack, sometimes referred to as MITM.
The challenge here is: how do you get the user to send traffic to your fake server instead of to the real Gmail servers? One common way of achieving this is to have some kind of interception appliance on the edge of the network. But what if you don’t have control of the network where the user resides? Perhaps the users the attacker is interested in are in a different country or even a different continent.

DNS
If you could just somehow change the DNS response for Gmail.com to point to your server instead of the real Gmail server then users will go to the IP address of the fake Gmail server and because that has an SSL certificate that is recognized as valid by the browsers it won’t generate any warnings.
Over the last few years we’ve seen viruses such as DNSCHANGER change the DNS server settings to DNS servers that are controlled by an attacker. Other ways to achieve this are DNS cache poisoning attacks.
Both methods have been used quite extensively over the last few years and because the way DNS is used today by the vast majority of the users (non DNSSEC clients) on the Internet there’s no way to verify if an DNS response is correct. This makes it relatively easy to insert fake DNS responses.

BGP MITM
One other way to redirect traffic from the user to the attacker is to go lower in the network stack and try to fool the routing system. BGP is the routing protocol of choice on the Internet today and since it’s largely based on trust, it’s relatively easy to fool the system. For example, if the attacker is able to announce a more specific route for the Gmail address range (IP prefix) and it’s picked up by the major transit providers, Internet users will be redirected to the attackers. It’s obvious that the potential impact of this is much bigger than let’s say DNS cache poisoning as that tends to be more localized.  Incidents like this happen relatively often by mistake, just take a look at our blog were we’ve published several high profile prefix hijack incidents over the last few years. It’s important to note that most of these cases are the result of configuration mistakes and most of the times it’s relatively easy to determine who hijacked the route. But what if an attacker could just somehow launch a man in the middle attack using BGP that’s much harder to detect and still allows the attacker to redirect traffic globally…?

Stealing the Internet.
This exact use case was presented a few years back at Defcon16 in a presentation titled “Stealing The Internet, An Internet-Scale Man In The Middle Attack”.
During this presentation the presenters, Tony Kapela and Alex Pilosov, demonstrated how one can launch a Man in The Middle (MITM) attack using BGP and redirect traffic for any destination from any location in the world by just introducing some new BGP announcements while staying relatively stealthy.

A recent Real Life BGP Man in the middle Event
Earlier in this article we stated that (accidental) BGP hijacks are relatively common, BGP MITM attacks however are rare and harder to detect. Having said that, our software does have logic to detect this kind of attack and last week we noticed a sudden increase in BGP MITM alerts being triggered for many prefixes including those of large networks such as France Telecom, Facebook and Microsoft. Let’s dig a bit deeper into this specific case and try to find out what exactly happened.

The following is an example alert and provides us with a starting point.

 

Finger printing Man in The middle Attacks
A BGP MITM attack as demonstrated at Defcon16 has the following properties: the BGP announcement is for a new more specific prefix and when looking at the ASpath closely we see AS relations that are typically not correct (fake) and most likely partially spoofed. Obviously this type of attack can be tuned, so the fingerprint may vary depending on the intent and creativity of the attacker.

Finding the root cause of this BGP MITM event.
With the above finger print in mind and numerous alerts helping us focusing in on the rather large data set, we started to dig deeper and tried to determine what exactly had happened here.
One of the clues we have when troubleshooting BGP is the ASpath. By looking at the ASpath we can say who originated the prefixes, which network provides transit to the originator and how does the path to the eventual receiver of the BGP update look like.

Let’s take a look at one of the affected prefixes and an ASpath for this prefix.
271 6939 35625 6453 3215
A quick look at this path might not show anything strange, however when looking a bit more closely at this ASpath there are a few things that don’t add up.

In this case we know that originator of the prefix, AS3215 France Telecom, does not have a direct peering/transit relationship with Tata (AS6453). This relationship does however show up in the ASpath. The other thing to note is that the ASpath shows that AS35625 (Avenir Telematique) is receiving the route from Tata (6453)(originated by France Telecom, 3215) and then announcing it to HE AS6939 (a peer of 35625) which then announces it to its customers. This means that 35625 (Avenir Telematique) is providing transit for 6453 3215 towards 6939 (Hurricane Electric). Given the size of both Tata (AS6453) and Hurricane Electric (AS6939), AS35625 should never be in the middle of these two.

So to summarize, the reason this update was marked as suspicious and eventually as a possible man in the middle attack is because it was a new more specific, the ASpath is suspicious as it contains non-existing relationships and one AS is leaking between two large providers. Our software has a few other checks and balances in place to prevent false alerts, but this pretty much sums up why it was flagged as suspicious.

Putting the pieces together
When looking closer at the ASpaths for all the events that were flagged as possible MITM we found that all ASpaths had one Autonomous System in common, AS35625 (Avenir Telematique), the same AS that appeared to have leaked the announcement to HE. At that point we focused our attention on this Autonomous System and we presumed that AS35625 was the one introducing these new announcements including the fake ASpaths.

After contacting the team responsible for AS35625 our suspicions were confirmed. As it turned out AS35625 has a “route optimizer” appliance that changes and introduces new BGP announcements, by breaking up prefixes in more specifics and altering the ASpath. All this is done in order to improve reachability and latency. Obviously these announcements are supposed to stay within the boundaries of the autonomous system, but in this case they were leaked to many of its peers.

Impact
If we look at the impact and affected networks we see that the number of prefixes that match the fingerprint was 418 unique prefixes of 133 unique Autonomous systems, including Facebook, Microsoft, Cogent, Bell Canada, Verizon, Level3, Shaw, Tata, Comcast, Yahoo, Verisign and many more, see full list here. The total event lasted for about 30 minutes, although it should be noted that the impact varied per prefix and peering partner.

The new more specific prefixes were announced to numerous peers of AS35625, we detected it via approximately 50 direct peers of AS35625, most notably via AS6327 (Shaw Cablesystems) and AS6939, Hurricane Electric.
As these are more specific prefixes it’s fair to assume that networks that received the BGP update for the affected prefixes, including the large customer base of both Hurricane Electric and Shaw would have rerouted traffic for some of the 400 prefixes towards AS35625 in France for several minutes.
In this case the only thing that limited the impact and prevented more prefixes to be affected were “max prefix filters” on the peering connections. In the case of Hurricane Electric the impact was limited to ~80 prefixes.

This event demonstrates how easy it is to accidentally steal parts of the Internet, and it make you wonder what could be done if an attacker would carefully plan and execute such an attack (would it be detected?).
It’s obvious that once an attacker has access to a Certificate Authority and can issue seemingly valid SSL certificates at will, there are numerous options for redirecting traffic. The event described in this blog show how BGP can help attackers redirect traffic for any network in the world, while staying relatively stealthy.
All this demonstrates the fragility of the current routing, CA and DNS system. The good news is that new technologies are currently underway to make the Internet more secure, DNSSEC, DANE and RPKI & BGPSEC are all technologies to make these the Internet infrastructure more secure, the bad news is that most of these technologies lack significant deployment or are still in the standardization phase.

This morning between 10:26 and 10:27 all routes originated by AS29256 (The Syrian Telecommunications Establishment) were withdrawn and became unreachable.
The only Syrian prefixes left in the routing table are 5 prefixes originated by TATA, AS6453. These are the prefixes that are still reachable via TATA:
216.6.0.0/23, 63.243.163.0/24, 116.0.72.0/22, 66.198.39.0/24, 66.198.41.0/24

What happened?
We have no official confirmation about what happened, but similar events in the past [Syria, Egypt] were all government ordered. Because the primary telecom provider is state controlled in Syria, an outage like this is relatively easy to implement by ordering the primary telecom provider to shutdown the external links or BGP sessions with the external providers. External providers that provide services to Syria are:
AS9121 Turk Telecom
AS6762 telecom Italia
AS3491 PCCW Global
AS6453 Tata
Not the first outage

Brief country wide outage earlier this week

Our historical data shows that Syria had an outage of similar scale earlier this week on November 25th at 08:46 when it lost the same number of prefixes for ~10minutes.
This might have been a test run, or perhaps more likely indicate that the outage is the result of damage to the physical infrastructure( optical fiber, radio links, routers, data centres, power) that makes the Internet work in Syria .

Update Fri 30 Nov 2012 02:15 UTC
The last 5 remaining Syrian prefixes originated by TATA are down as of 23:45 UTC. Syria is now 100% offline!

Brief sign of life
There was a brief sign of life from the Syrian prefixes announced by TATA (AS6453). BGP updates for these prefixes were briefly seen between 15:50 and 15:52 as well as between 17:39 and 17:43, UTC times on November 30th.

A brief restoration of power could be one reason for this short sign of life.

Back Online (Dec 1st, 14:32)
Internet connectivity to Syria has been returned and all Syrian prefixes are now back online. Connectivity was restored at 14:32 UTC (Dec 1st) via AS3491 PCCW Global, AS6453 Tata and AS6762 telecom Italia.
About 1,5 hours later at 15:58 Syria’s connection with Turk Telecom was also re-established.

So I’m happy to announce that as of today BGPmon.net services will be available in two flavors: a free ‘entry level’ service and a full-featured premium commercial service.

With these changes, BGPmon.net will become more sustainable and provide better support, and allow us to continue improving services while adding new features.

What to expect
Our base services remain free, but with a limited feature set and up to 5 prefixes per account.

The premium commercial service allows you to monitor as many prefixes as needed and provides the full-feature set on a new powerful platform. The routing report, SOAP API and additional email address features are now part of the premium service. Pricing details can be found in the new client portal.

Please note that your current account has been converted to a basic account. Login into our new portal to see what this means for you and to make changes to the list of prefixes.

We included many changes in this new release. Recent upgrades include our new website, PeerMon, the improved API, the new portal and full RPKI support.

I’d like to thank all of you for your continued support and encouragement over the last few years. I hope our new service offering will meet your needs and will help strengthen BGPMon.net for the years ahead.

Introduction into the new BGPmon client portal

Today many network operators saw their BGP session flap, RTT’s increase and CPU usage on routers spike.  While looking at our BGP data we determined the root cause to be a large BGP leak in Canada that quickly affected networks worldwide.

Dery Telecom
Based on our analysis it seems that Canadian ISP Dery Telecom Inc (AS46618) is the cause of what we observed today. AS46618 is dual homed to both VIDEOTRON and Bell. What seems to have happened is that AS46618 leaked routes learned from VIDEOTRON to Bell. This in itself is not unique and happens relatively often. However normally transit ISP’s like Bell have strict filters applied on these BGP sessions, limiting the number of prefixes they accept from their customers. In this case the filter failed to work or simply wasn’t (correctly) applied by both Bell and Dery Telecom.

Sequence of events
At 17:27 UTC  AS46618 ( Dery Telecom Inc) started to leak a ‘full table’, or at least a significant chunk of it to its provider Bell AS577. Bell selected 107,409 of these routes as best routes. Even though many of the ASpaths were much longer than other alternatives it was preferred because many ISP’s localpref customers higher than other peers and transit providers, so as a result customer routes are always preferred even when the ASpath is longer.

Bell then propagated the learned prefixes to its peers. Tata was one of the ones that accepted and used the bulk of these prefixes and re-announce these to its peers and customers.

Who was affected?
Interested if your prefixes were affected? We made a list of all prefixes and ASn’s that were leaked, feel free to see if your prefixes was one of them here: http://www.bgpmon.net/bell-leak.txt

BGP update storm
BGPmon routesevers saw a significant increase in BGP updates. A number of routers on the Internet were not able to keep up and experienced pegged cpu’s, some even had flapping BGP sessions.  Many Tata and Bell customers also reported performance and reachability problems.

BGP leaks
BGP leaks are relatively common, though the impact varies.  Earlier this year we reported about another large leak involving the Australian incumbent Telstra, causing most of the Internet in Australia to be affected.  The solution to the problem is simple, filter, filter, filter your BGP peers.

We normally see about 860 prefixes for IP addresses in Lebanon. The diagram below shows that this was reduced by about 90% at 16:16 (UTC), leaving only about 85 prefixes reachable. Networks relying on Liban Teleccom (AS42020) , the largest provider in Lebanon, are most severely affected.
In the hours after some of these networks were restored as they were rerouted via Satellite ISP AS30721 SATGATE.

At the time of writing about 60% if the Lebanese prefixes are reachable, which leaves 40% unreachable. The prefixes that are reachable will likely experience slower connections then normally, as they now rely on lower capacity connections. This report indicates that the current Internet capacity for Lebanon is reduced to 3Gb/s only.

For Lebanon this is the second country wide outage this week. On Monday July 2nd planned maintenance work on the same submarine cable resulted in a 2,5 hour outage for Lebanon. Luckily large-scale country wide outages are rare. We reported earlier about large outage in Egypt and Australia. In the case of Lebanon limiting the reliance on this single point of failure should improve the uptime of the networks significantly.

Let’s hope for everyone in Lebanon that this outage will be resolved soon!

Telstra is one of Australia’s major Internet providers. It normally originates approximately 500 Ipv4 prefixes and 3 Ipv6 prefixes. Telstra also provides Transit for many ISP’s and enterprises such as for example AS38285 ‘Dodo’ an Australian ISP and AS10235 ‘National Australia Bank’. So how could such a large provider go down, surely it has lots of redundant hardware and multiple connections in and out of the country?

As it turns out Wednesday’s outage was caused by a routing error many network engineers have first hand experience with, a simple routing leak. A routing leak can happen when small ISP X buys transit from ISP A and also from ISP B. ISP X receives a full bgp routing table from A and because of incorrect filtering  relays these messages to ISP B. As a result ISP B now learns all Internet routes via ISP X to ISP B and ISP X (the customers) now became an upstream provider for ISP B.

The above is likely what happened last Wednesday between Telstra en Dodo (AS38285). Dodo a Telstra customer, re-announced all Internet routes to Telstra, which because it prefers customer routes now thinks the best way to the Internet is through Dodo. This post on the Ausnog mailings list shows how Telstra was using Dodo (a customer) as transit to reach a network in India.

This is not a new zero day attack scenario or anything like it. Instead it’s probably the number one mistake when configuring BGP routing. I remember when I was just learning about BGP my mentor always used to tell me.. Filter, Filter, Filter, filter!! Which is exactly what didn’t happen here.
Because it is so easy to accidentally leak routes in BGP you have to explicitly define filters that prevent this. In this case Dodo should have had filters to make sure they would only announce their prefixes and Telstra should have had these filters as well to prevent hijacks but more importantly to protect its own infrastructure. In this case these filters did not seem to be in place, which allowed this leak to happen.

However, this alone should not have brought down all of Telstra’s International connections. So what happened? It’s likely that Telstra now tagged all routes learned from Dodo (all 400,000 of them) as customer routes and faithfully announced this to all of its peers and upstream providers.

As keeping large filters up to date can be tedious we often see large providers use a mechanism known as max prefix limits. Instead of explicitly defining which prefixes to allow the number of prefixes expected plus some extra is set as the maximum number of prefixes allowed. This is useful to prevent a sudden spike in announcements, often caused by leaks. In case the limit is reached the BGP session is brought down to prevent the leak from spreading.

This is what likely happened with Telstra’s upstream providers. Telstra announced a full Internet routing table to its upstream providers, triggering the max prefix limit causing the BGP sessions to go down and as a result isolating Telstra and its customers from the rest of the Internet.

This outage is clearly visible when we look at the Telstra routes in the Internet BGP routing tables. The graph on the left clearly shows how all of Telstra’s prefixes suddenly disappeared at 2:40 UTC and came back  approximately 30 minutes later when the leak problem was resolved.

Our analysis shows that this affected approximately  1400 prefixes. This includes Telstra and Telstra customer networks and accounts for about 10% of all the Australian IPv4 prefixes. Worth pointing out is that non of the IPv6 prefixes were affected. This is because it’s common to use different BGP sessions for IPv4 and IPv6. In this case only the IPv4 session went down leaving the IPv6 networks unaffected.

And so it could happen that one simple customer leak can bring down a complete carrier network. So today’s lesson is Filter Filter Filter!

Often these anycast nodes are deployed to only serve specific network and/or regional areas, making them only reachable from the networks where they are deployed. In the case of the F-root server, operated by ISC, there are quite a few local servers. See this page for a detailed list: https://www.isc.org/community/f-root/sites One of these servers is the F-root located in China.

This weekend for about 25 hours this particular root server was reachable not only from China, but from several places in the world when using IPv6. Normally this shouldn’t be a real problem, as the root servers return the same DNS response regardless of their geographical location. However in the case of China, all traffic typically passes through the great firewall where DNS responses for requests such as facebook.com are often rewritten to different false IP addresses. Please note that I am not aware of any data to confirm if this is what happened in this particular case!

Technical details
This weekend between 01 Oct 2011 17:56:12 and 02 Oct 2011 19:01:09 GMT a number of providers thought that the best IPv6 path to the F-root server (AS3557), was to the F-root in China.

The prefix for the F-root server is 2001:500:2f::/48 and is originated by the following AS numbers:
3557 ISC-F-ROOT Internet Systems Consortium, Inc
1280 ISC-AS1280 Internet Systems Consortium, Inc.
30132 ISC-AMS1 Internet Systems Consortium, Inc

By looking at the next hop AS number we can get a good feeling of where the server for this particular announcement resides. When analyzing the announcement for 2001:500:2f::/48 we saw one new AS we had not seen before outside of China, AS55439.

When looking at all announcements for the F-root in China, we saw the following constant in the ASpath:
6939 23911 18344 37944 24151 55439 3557

In this case AS55439 is the AS for ISC-PEK2 Internet Systems Consortium, (Beijing, China). Next the announcement was passed along to AS24151 (China Internet Network Information Center), AS37944 (CHINA SCIENCE AND TECHNOLOGY NETWORK), AS18344 (CNCGROUP-BACKBONE-NORTH), it then ended up at the AS23911, the China Next Generation Internet Beijing IX. That’s where it was learned by AS6939 Hurricane Electric, the world’s largest IPv6 provider. Hurricane electric then advertised it to its customers.

The dataset I used showed that numerous Hurricane electric customers in countries worldwide, including for example Germany, South Africa, Brazil and the US selected this path to China. Some of the large providers include AT&T, Telia and Interoute.

What does all this mean
It’s likely that DNS queries over IPv6 to the F-root server from clients worldwide were answered by the F-root in China. Although this is in itself is not necessarily a problem, it does expose this traffic to the Great Chinese Firewall with the risk of altering the responses.

This is not the first time this happens, last year Renesys reported about a similar event involving the I-root server and China.

The only way to make sure that DNS responses are not rewritten is to use DNSSEC, this will allow the resolver to verify the authenticity of the response.
With regards to BGP, in this case all that seems to have happened was a BGP leak. Things like this are typically the result of an operator mistake. Best practice is to monitor for these kinds of things, so you’ll be alerted as soon as it happens and as a result allowing operators to limit the effect of such an event. BGPmon.net provides users with several flexible ways to monitor BGP announcement and to check if they match your BGP policies.

The Internet in Syria is dominated by “The Syrian Telecommunications Establishment”, which routes its networks from AS29256 and AS29386. Besides these providers there are AS6453 – Tata communications which routes 6 Syrian prefixes and AS39154 The Syrian Higher Education Network.

As of this morning only 19 of the normally 56 Syrian prefixes are routed.  Interestingly the prefixes that are no longer routed are normally  originated by AS29256 and AS29386, “The Syrian Telecommunications Establishment”.  The 6 Tata communication prefixes as well as the Syrian Higher Education Network have not been affected.

The table below show how many prefixes are normally routed by these providers, and how it has changed over the last hours.

The pie charts below show how the Syrian network are distributed over the different providers. Clearly visible are the reduced number of prefixes announced by “The Syrian Telecommunications Establishment”, AS29256 and AS29386.

Prefix Distribution June 1

Prefix Distribution June 3

Update June 4th, 2011
As of this morning (approximately 08:00 UTC) all Syrian networks routed by “The Syrian Telecommunications Establishment” are back online! Some prefixes came back earlier this morning. Also see Google’s Transparency report here.

Barrett Lyon reported on his blog that he noticed that AT&T was routing traffic to Facebook through a Chinese network (China telecom AS4134). In this blog post we will take a closer look at what exactly happened.

The raw data
By analyzing the raw data we can indeed see BGP announcements for several of Facebook prefixes with rather long and odd looking ASpaths. We also see several US based providers, such as AT&T, that selected a path through SK Telecom (Hanaro Telecom South Korea) and China Telecom.

It seems to all have started on Tuesday March 22, at 07:15:02 GMT. That’s when we see the first announcement for one of Facebook’s networks, routed through Korean provider SK Telecom. The last announcement via SK Telecom was at Tuesday, 22 Mar 2011 16:11:02 GMT, so about 9 hours in total.

The impact of this incident varied per provider. Some networks didn’t use the path through Korea, some only for a few of Facebook’s prefixes as some for (almost) all of Facebook’s prefixes.

Two Japanese providers, who peer directly with SK Telecom (Korea) routed the following Facebook networks trough AS9318, SK Telecom (Hanaro Telecom South Korea)

• 204.15.20.0/22
• 66.220.144.0/21
• 66.220.152.0/21
• 69.171.224.0/20
• 69.171.239.0/24
• 69.171.240.0/20
• 69.171.255.0/24
• 69.63.176.0/21
• 69.63.184.0/21
• 74.119.76.0/22

The providers affected by this were:
Internet Initiative Japan Inc. (IIJ)   ASpath:  2497 9318 32934 32934 32934
KDDI   ASpath: 7660 2516 9318 32934 32934 32934

Several other large providers such as Savis, AT&T, Tinet, KPN, Telia, Qwest, AboveNet and Telecom Italia (note this list is not inclusive, there are several more) were also affected, however in these cases only two prefixes were affected.

69.171.224.0/20
69.171.255.0/24

All of these providers learned this announcement via AS4134, which is China telecom. In all cases the Facebook prefixes were prepended three times. Different peers on different locations saw the announcements, the common part in the ASpath was:

4134 9318 32934 32934 32934

This proves that the announcements were not spoofed by China telecom, as others learned it directly from SK Telecom as well.

What exactly did AT&T see at the time of the incident?

This is a snapshot of the AT&T routing table at March 22^nd,  8AM, showing Facebook networks only. Note that only 2 networks were routed through Korea and China.

As can be seen from the snapshot above, the majority of the prefixes (as seen from AT&T) are routed through Level3 (3356). So what happened to the other 2 prefixes? Why was the path to those networks not through Level3. To answer that question we need to know the path to these networks from Level3′s perspective. When looking at the same datasource we see the following for 69.171.224.0/20:

This is interesting; it shows that Level3 learns the Facebook prefix with a rather long ASpath, Facebook’s AS is prepended 5 times. This explains why AT&T chose the path via China and Korea. Even though it’s long, it was shorter than the heavily prepended path via Level3. As a result providers who peer with China Telecom and had this shorter alternative, would have chosen the path via China.
Global Crossing, another large global Transit provider showed the same behavior as Level3, an ASpath with 5 times Facebook’s AS32934.

What could be the reason?

For some reason the path via providers such as Level3 or Global Crossing, Transit networks that are commonly used by the affected providers, were advertised with a very long ASpath. And as a result the providers that have a peering agreement with China Telecom (such as AT&T) started to use the shorter path via China and Korea.
SK Telecom normally is not one of Facebook’s transit providers, but likely just a normal peer.  SK Telecom then announced these Facebook prefixes to its peers (IIJ, KDDI and  China Telecom) which started to use them.

China Telecom did not filter these Facebook announcements learned via SK Telecom and re-announced them happily to its peers. As a result traffic from several major global providers started using the path through China and Korea to reach some of Facebook’s networks.

Why the providers such as Level3 and Global Crossing did not have a shorter path to these two Facebook networks is unknown. But it’s likely that it had to do with the Facebook BGP setup / infrastructure.

Keep in mind….
It’s likely that both China Telecom and SK Telecom have a presence in the US. And as such it’s not necessarily the case that traffic was actually routed through China.  If someone has a traceroute from AT&T or any of the other networks taken at the time of the incident, please post this in the comments, this can helps us determining how the actual traffic flowed.

Facebook has a lot of data and does all kinds of CDN (content delivery) tricks to get data to its users.  Depending on your location you get a different DNS result and send to a different server. This might have been the reason why there were no reports of widespread outage.