January 10, 2024News
Guest Blog: Digging into the Orange España Hack
Exploring the Recent Hack on Spain's Second-Biggest Mobile Operator
by Doug Madory
This month, Orange España, Spain’s second-largest mobile operator, suffered a major outage. The outage was unprecedented due to the use of RPKI, a mechanism designed to protect internet routing security, as a tool for denial of service.
On January 3, 2024, Spain’s second-largest mobile operator, Orange España, experienced a national outage spanning multiple hours. The cause? A compromised password and an increasingly robust routing system. The network operator’s favorite defense tool (RPKI) can be a double-edged sword.
Using a password found in a public leak of stolen credentials, a hacker was able to log into Orange España’s RIPE NCC portal using the password “ripeadmin.” Oops! Once in, this individual altered Orange España’s RPKI configuration, rendering many of its BGP routes RPKI-invalid.
As demonstrated in our earlier analysis, the internet’s RPKI ROV deployment has reached the point where the propagation of a route is cut in half or more when evaluated as RPKI-invalid. Normally, this is the desired behavior, but when an RPKI config is intentionally loaded with misconfigured data, it can render address space unreachable, effectively becoming a tool for denial of service.
Using Kentik’s aggregate NetFlow, we observed the outage (illustrated above) as a significant drop in the volume of inbound traffic to Orange España (AS12479) between 14:20 UTC (3:20 pm local) and 18:00 UTC (7 pm local). However, there were more developments before this window of time and some lingering effects, which we will dig into in the post below.
So, What Exactly Happened?
We already know the outage and how the attacker pulled it off. Now, let’s trace the sequence of events using archived RPKI data from RPKIviews.
The story begins at 09:28 UTC on January 3, when someone (presumably the attacker) began tinkering with publishing and revoking ROAs for IP ranges belonging to the Spanish mobile operator. Then, at 09:42 UTC, they published three new ROAs for Orange España IP ranges with material impact.
Origin prefix maxLength ta expiration AS12479 18.104.22.168/22 22 ripe 1704355258 AS12479 22.214.171.124/21 21 ripe 1704355258 AS12479 126.96.36.199/16 16 ripe 1704355258
Given the fact that AS12479 all already originated 188.8.131.52/22, 184.108.40.206/21, and 220.127.116.11/16, those routes weren’t affected, but 18.104.22.168/16 had quite a few more-specifics that were now going to be evaluated as RPKI-invalid due to the max prefix length setting of 16.
Perhaps realizing this, minutes later, someone published a slew of additional ROAs to account for the more specifics of 22.214.171.124/16. These had the proper origin (AS12479), and as a result, all of those more-specifics became valid. All but one that is.
Origin prefix maxLength ta expiration AS12479 126.96.36.199/23 23 ripe 1704355258 AS12479 188.8.131.52/23 23 ripe 1704355258 AS12479 184.108.40.206/23 23 ripe 1704355258 AS12479 220.127.116.11/23 23 ripe 1704355258 AS12479 18.104.22.168/23 23 ripe 1704355258 (and many more)
Using Kentik’s BGP visualization, we can compare the impact in reachability (aka propagation) for two adjacent more-specifics of 22.214.171.124/16. Below, 126.96.36.199/22 was the route missed in that follow-up publication of ROAs. Its reachability dropped to as little as 20% of our BGP sources for over four hours.
Conversely, the rest of the specifics looked like 188.8.131.52/22 below: a partial drop in reachability between the first and second publications of ROAs mentioned above.
Although these prefixes were RPKI-invalid for several minutes, they only experienced a partial drop in reachability due to delays in the time to globally propagate ROAs, as documented in recent research on the topic. The act of blotting out a newly RPKI-invalid route is not instantaneous.
Wielding RPKI as a Weapon
Then the attacker took it a step further by creating ROAs with an origin other than that of Orange España’s. About the same time those additional ROAs were published covering the more-specifics of 184.108.40.206/16, four new ROAs were created for Orange España IP space with a deliberately incorrect origin of AS4958
Origin prefix maxLength ta expiration AS49581 220.127.116.11/16 16 ripe 1704355258 AS49581 18.104.22.168/21 21 ripe 1704355258 AS49581 22.214.171.124/20 20 ripe 1704355258 AS49581 126.96.36.199/16 16 ripe 1704355258
The addition of the bogus ROA for 188.8.131.52/16 had no effect because the attacker had previously created a ROA with the correct origin (AS12479) — as long as one ROA matches, a route is evaluated as RPKI-valid.
184.108.40.206/20 and 220.127.116.11/21 were only briefly invalid before the attacker published ROAs with correct origins.
Origin prefix maxLength ta expiration AS12479 18.104.22.168/20 20 ripe 1704355258 AS12479 22.214.171.124/21 21 ripe 1704355258
Only 126.96.36.199/16 (shown below) and its numerous more-specifics were rendered RPKI-invalid and had their reachability reduced for the duration of the outage due to the ROAs with bogus origins.
Thus far in the story, the attacker’s tinkering has led to the creation of a couple of RPKI-invalid routes and some minor reachability problems, but the major disruption was yet to come.
It wasn’t until about 14:20 UTC (3:20pm local) that things got ugly. The attacker went for it and published four more ROAs with bogus origins. Two of the ROAs were /12’s which covered over a thousand routes originated by AS12479 — all rendered RPKI-invalid by the publication of the following ROAs:
Origin prefix maxLength ta expiration AS49581 188.8.131.52/12 12 ripe 1704355258 AS49581 184.108.40.206/12 12 ripe 1704355258 AS49581 220.127.116.11/21 21 ripe 1704355258 AS49581 18.104.22.168/21 21 ripe 1704355258
It was here when the traffic graph at the beginning of this blog post began to take a nose dive. The number of globally routed routes originated by AS12479 dropped from around 9,200 to 7,400, as backbone carriers which reject RPKI-invalid routes stopped carrying a large chunk of Orange España’s IP space.
Reachability of 22.214.171.124/21 during the worst part of the outage.
It wasn’t until just before 18:00 UTC (7pm local) that things began to return to normal. Engineers from Spain’s second largest mobile operator regained control of their RIPC NCC account and began publishing new ROAs that would enable the carrier to restore service.
Origin prefix maxLength ta expiration AS12479 126.96.36.199/12 12 ripe 1704384768 AS12479 188.8.131.52/12 12 ripe 1704384768 AS12479 184.108.40.206/16 16 ripe 1704384768 AS12479 220.127.116.11/21 21 ripe 1704384768 AS12479 18.104.22.168/21 21 ripe 1704384768 AS12479 22.214.171.124/22 22 ripe 1704384768 AS12479 126.96.36.199/24 24 ripe 1704384768
While RPKI was employed as a central instrument of this attack, it should not be construed as the cause of the outage any more than we would blame a router if an adversary were to get ahold of the login credentials and start disabling interfaces.
It seems that prior to January 3rd, the Spanish mobile operator’s RIPE NCC account had never created a ROA (although other parts of Orange had created some on its behalf). If RPKI wasn’t on Orange España’s radar before, it sure is now.
Although the outage is over, there is still a lot of clean-up work to be done. As of this writing, over a thousand of the routes originated by AS12479 are still invalid, mostly due to the max prefix length setting on the ROAs for the two /12’s. Between yesterday and today, the number of unique IPv4 addresses originated by AS12479 dropped from 7 million to 5 million, and a few bogus ROAs with an origin of AS49581 are still in circulation.
I would remind those engineers cleaning up the ROAs that max prefix length is an optional field and can simply be left empty causing RPKI to only match on the origin of the ROA. This course of action was recently published as a best current practice.
Hopefully this incident can serve as a wake-up call to other service providers that their RIR portal account is mission-critical and needs to be protected by more than a simple password.
Thanks to Job Snijders of Fastly for his expert guidance on this blog post.
Protect Your RIPE NCC Account
Doug Madory is the director of internet analysis for Kentik where he works on internet infrastructure analysis. The Washington Post dubbed him “The Man who can see the Internet” for his reputation in identifying significant developments in the global layout of the internet. Doug is regularly quoted by major news outlets about developments ranging from national blackouts to BGP hijacks to the activation of submarine cables. Prior to Kentik, he was the lead analyst for Oracle’s internet intelligence team (formerly Dyn Research and Renesys).
February 21, 2024 • News
Attendee Shares Takeaways from NANOG's Most Recent Meeting
Internet technology and infrastructure is a constantly changing industry. Staying ahead of the curve is crucial for professionals and organizations alike. Enter NANOG, t…Learn More
January 18, 2024 • Stories
Mark Johnson Presents “Exploring the Internet History of North Carolina”
When contemplating the vibrant history of the Internet, California, home to the prestigious UCLA and the iconic "Birth of the Internet," may first come to mind. North Ca…Learn More