Large hijack affects reachability of high traffic destinations
April 23, Update: NOC Team at innofield posted an explanation of the Incident in the comments section below.
Starting today at 17:09 UTC our systems detected a large scale routing incident affecting hundreds of Autonomous systems. Many BGPmon users have received an email informing them of this change.
Our initial investigation shows that the scope of this incident is widespread and affected 576 Autonomous systems and 3431 prefixes. Amongst the networks affected are high traffic prefixes including those of Google, Amazon, Twitter, Apple, Akamai, Time Warner Cable Internet and more.
All these events have either AS200759 “innofield AG” or private AS 65021 as the origin AS. In the cases where AS65021 appears as the origin AS, AS200759 is again the next-hop AS.
AS200759 “innofield AG” is a provider based out of Switzerland and normally only announces one IPv4 and one IPv6 prefix.
These are 2 example events:
Prefix 66.220.152.0/21 Is normally announced by Facebook AS32934 and during this event was announced by AS200759 as a more specific /22
Detected prefix: 66.220.152.0/22
Example aspath: 4608 24130 7545 6939 200759
And AS origin: 65021 behind AS 200759
Detected prefix: 66.220.152.0/22 Example aspath: 133812 23948 4788 6939 200759 65021
We saw the announcements via the following peers of AS200759 “innofield AG”:- 20634 “Telecom Liechtenstein AG”
- 6939 “Hurricane Electric, Inc.”
- 16265 “LeaseWeb Network B.V.”
10 comments
[…] was caused by a very large-scale network outage at a major Internet backbone company. It affected traffic to a significant percentage of the Internet, including as Amazon, Facebook, […]
Another route optomizer gone wrong?
Yes, as part of our investigation we did reach out and it sounds like it was related to a route optimizer.
thank for the monitoring and for the explanation 🙂
Hi all
Here is the NOC Team from innofield.
First of all: we sincerely apologize for the inconvenience this have caused.
Background of this incident:
– Yesterday (2016-04-22) SwissIX (Swiss Internet Exchange) has performed a maintenance on our port and we had to de-activate and re-activate the BGP sessions.
– After re-activating the sessions at approx. 17:10 UTC, for a currently unknown reason, we have redistributed other prefixes into SwissIX.
– This definitely shouldn’t happen, as a matter of course we have filters in place for SwissIX (which only allow to send our prefixes).
– Immediately after we received the first complains about the route-hijack, we have de-activated the BGP sessions again (at approx. 17:25 UTC)
Next steps at our side:
– We are in permanent contact with the router and the route-optimizer vendors. Together, we are deeply analyzing the system/logs why the filters were not considered after re-activating the BGP sessions.
– Currently (but not confirmed yet) it looks like a code-bug in the router software.
– We will definitely not re-activate the sessions until the root-cause is found!
– With this planned actions we will prevent this from happening again.
Again we are very sorry for this incident and we will do everything you can think of to avoid this situations in the future.
Respectfully,
NOC Team from innofield AG
Init7 AS13030 is one of the transit suppliers of Innofield AS200759. We double-checked the filters and it seems that we didn’t leak the hijacked prefix. Therefore we wonder why the path _13030_200759_ was seen during the hijack.
We also see only an increase of about 1gig inbound traffic during the hijack. I wasn’t even detected as this is below threshold. Considering the importance of the hijacked prefix of Facebook, I think the malicious traffic went mostly through SwissIX infrastructure.
Hi Freddy,
Thanks for reaching out. Yes you are correct, it appears AS13030 was erroneously mentioned, our apologies. AS13030 was detected as an upstream for legitimate AS200759 prefixes only. Good job filtering! I’ve updated the list accordingly.
At least HE and Core-backbone don’t usually peer with routeservers, so its more likely that we are talking about dedicated peering session. In fact, the peering with HE is documented in innofield’s aut-num object (meaning it was manually put there).
Also, the aut-num references a AS-SET that doesn’t exist: AS-FLOW.
Init7 is actually a transit of innofield, so it appears they don’t filter from their customers.
Also see remark re: init7 above. They filtered correctly.
[…] 前幾天 BGPmon 偵測到大規模的 routing-hijacking 事件：「Large hijack affects reachability of high traffic destinations」，範圍應該是最近最大的： […]