A BGP leak made in Canada

Posted by Andree Toonk - August 8, 2012 - Uncategorized - 2 Comments

A BGP leak made in Canada

Today many network operators saw their BGP session flap, RTT’s increase and CPU usage on routers spike.  While looking at our BGP data we determined the root cause to be a large BGP leak in Canada that quickly affected networks worldwide.

Dery Telecom
Based on our analysis it seems that Canadian ISP Dery Telecom Inc (AS46618) is the cause of what we observed today. AS46618 is dual homed to both VIDEOTRON and Bell. What seems to have happened is that AS46618 leaked routes learned from VIDEOTRON to Bell. This in itself is not unique and happens relatively often. However normally transit ISP’s like Bell have strict filters applied on these BGP sessions, limiting the number of prefixes they accept from their customers. In this case the filter failed to work or simply wasn’t (correctly) applied by both Bell and Dery Telecom.

Sequence of events
At 17:27 UTC  AS46618 ( Dery Telecom Inc) started to leak a ‘full table’, or at least a significant chunk of it to its provider Bell AS577. Bell selected 107,409 of these routes as best routes. Even though many of the ASpaths were much longer than other alternatives it was preferred because many ISP’s localpref customers higher than other peers and transit providers, so as a result customer routes are always preferred even when the ASpath is longer.

Bell then propagated the learned prefixes to its peers. Tata was one of the ones that accepted and used the bulk of these prefixes and re-announce these to its peers and customers.

Who was affected?
Interested if your prefixes were affected? We made a list of all prefixes and ASn’s that were leaked, feel free to see if your prefixes was one of them here: http://www.bgpmon.net/bell-leak.txt

BGP update storm
BGPmon routesevers saw a significant increase in BGP updates. A number of routers on the Internet were not able to keep up and experienced pegged cpu’s, some even had flapping BGP sessions.  Many Tata and Bell customers also reported performance and reachability problems.

BGP leaks
BGP leaks are relatively common, though the impact varies.  Earlier this year we reported about another large leak involving the Australian incumbent Telstra, causing most of the Internet in Australia to be affected.  The solution to the problem is simple, filter, filter, filter your BGP peers.

 

2 comments

  • hongtao says:

    please persuade me why TATA can pass the routes learning from Bell to their other customers and those routes are prefered by the other ISP, as the local preference will be only local significant. also I think the best way should be investigate to see if AS46618 has interminent throughput problem and packets drop when the symptom had happened. It;s easy to figure this out.

  • Andree Toonk says:

    Hi hongtao,
    What I meant to clarify is that even though the ASpath was much longer than other alternatives in the Bell and Tata routing tables, they both selected the ‘leaked’ routes because of local-pref policies.

    One example Aspath for 208.180.52.0/24 was:
    11039 11557 4436 2914 6453 577 46618 46618 46618 46618 46618 5769 19108
    Both bell and tata must have had much shorter paths, but didn’t choose these because of policies.

    Of course local-pref values only apply to decisions within an AS. However only one ‘best’ route is announced to peers. So in the case of bell, tata, verizon and others, the choices for customers are limited as Tata and bell only announced the longer path to them.

    Hope that explains.

    Cheers,
    -Andree

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>