FYI So A Router in El Dorado, Arkansas Caused The Full Ground Stop

kenn10

Well-Known Member
Joined
Dec 16, 2007
Messages
4,768
Reaction score
3,155
Sounds like a contractor botched a router upgrade to me which resulted in a flood due to improper routing. I don't buy that a digital copper line vs. fiberoptic really has anything to do with the issue. Sounds like scare tactics to get the necessary funding. I don't disagree that the funding is needed to finally upgrade the ancient FAA equipment.

According to the FAA and Transportation Secretary Sean Duffy:

  • Legacy equipment became unstable when the FAA FTI network was flooded with data at a rate of about 1,000 connection attempts every 4–5 seconds. This disrupted aircraft communications and radar feeds.
  • Initial investigation pointed to two sites: a radio communications site in El Dorado, Arkansas, and a radar site in Amarillo, Texas. L3Harris (the FAA’s network service provider) was performing work at the El Dorado site that included deploying a new router as part of the ongoing, routine conversion of outdated copper-wire lines to fiber optics.
  • L3Harris blocked the affected locations, the network stabilized, and the ground stop was lifted. Officials are still determining the precise fault—whether installer/user error by L3Harris or a defective router. The broader context is the age and limitations of the legacy TDM/copper-based systems.⁠
For those old enough to remember "Green Acres".
1794793818942.png
 
Last edited:
Sounds like a contractor botched a router upgrade to me which resulted in a flood due to improper routing. I don't buy that a digital copper line vs. fiberoptic really has anything to do with the issue. Sounds like scare tactics to get the necessary funding. I don't disagree that the funding is needed to finally upgrade the ancient FAA equipment.


For those old enough to remember "Green Acres".
View attachment 6644
Sorry, but I don't buy the assertion that one defective or misconfigured router brought down the entire communications network of the FAA. In a former life, I was one of the architects of a nationwide communications network for the federal courts. We had a router at every site where there was a federal courthouse. Many were on copper backbones and probably still are. There are considerably more federal courthouses than there are airports in the United States. Do you think for a minute that none of these routers has ever failed until last Friday? Do you think for a minute that no router has ever starting spewing errant packets in the FAA's network or in the federal court's wide area network? Cisco and Juniper routers have some shortcomings, but a system-wide outage caused by a single point of failure is not one of them. So I'm calling bullsh*t on this one.

As for funding, they already have $50 $12.5 billion of which they have spent $1 billion according to current reporting.
 
Last edited:
Sorry, but I don't buy the assertion that one defective or misconfigured router brought down the entire communications network of the FAA.
Yet it did. 17 of the 22 control centers were impacted by this. The router in AR made a router down in Amarillo TX goes nuts too. Because all their network has Layer 2 overlays of some sort like VPLS or another method, a broadcast storm started and it was flooding with around 1,000 connection attempts ever 5 seconds.

Here's the thing, if all those other towers are on legacy equipment that's old and outdated then that equipment can't handle massive loads of data because it wasn't designed for it. So a 1,000 connection attempts to them or a broadcast storm can 100% impact a huge chunk of a network. Which it did in this case since only 5 sites weren't impacted.

So yeah, a single bad device (for whatever reason) has the ability to bring an entire network to a halt or seriously degrade it depending on what is happening.
 
Last edited:
Sorry, but I don't buy the assertion that one defective or misconfigured router brought down the entire communications network of the FAA. In a former life, I was one of the architects of a nationwide communications network for the federal courts. We had a router at every site where there was a federal courthouse. Many were on copper backbones and probably still are. There are considerably more federal courthouses than there are airports in the United States. Do you think for a minute that none of these routers has ever failed until last Friday? Do you think for a minute that no router has ever starting spewing errant packets in the FAA's network or in the federal court's wide area network? Cisco and Juniper routers have some shortcomings, but a system-wide outage caused by a single point of failure is not one of them. So I'm calling bullsh*t on this one.

As for funding, they already have $50 $12.5 billion of which they have spent $1 billion according to current reporting.

I can pretty much guarantee that the architecture wasn't updated to meet modern best practices, and it's entirely possible it had a single point of failure, that just happened not to fail in an unplanned way for a long time because Cisco is pretty amazing reliability wise (I accidently found a switch in a network the other day, uptime was 8 years). The packet storm explanation is both plausible, and likely, given the way things developed. Also, please don't change things on a weekday mid day . . . that's change control 101.
 
Last edited by a moderator:
1794856414982.png

For @Samot and others offering expertise without documenting credentials, here is the entire chronology including comments regarding the likelihood of a single router disrupting an entire nationwide network:

 
Last edited:
As for funding, they already have $50 billion of which they have spent $1 billion according to current reporting.
I'm not sure that is an accurate assessment. First, lets not forget that
Under the 2021 Bipartisan Infrastructure Law, the Federal Aviation Administration (FAA) was allocated a total of $25 billion over five years for airport and air traffic control modernization, rather than a lump sum given directly to U.S. Transportation Secretary Pete Buttigieg. [1]

Breakdown of the $25 Billion Allocation
According to the FAA Bipartisan Infrastructure Law Announcement, the funds were distributed across three main categories: [1]
    • $15 billion for Airport Infrastructure Grants (AIG) to upgrade runways, taxiways, and airport roads.
    • $5 billion for the Airport Terminal Program to expand terminals and improve accessibility.
    • $5 billion for the FAA’s Facilities and Equipment account to modernize air traffic control towers, radar systems, and navigation aids.
The FAA has not already received about $50 billion specifically for network/air traffic control (ATC) upgrades that remains largely unspent.

What actually happened with funding​

  • In July 2025, Congress appropriated $12.5 billion (via the One Big Beautiful Bill Act / related legislation) as a “down payment” for the Brand New Air Traffic Control System (BNATCS) modernization effort. Roughly $9 billion of that was directed toward systems modernization (telecommunications/fiber replacement, radars, radios, voice switches, etc.), with the rest for facilities.
  • The $50 billion figure is a new request from Transportation Secretary Sean Duffy (and the administration) made in mid-November 2026, after a nationwide ground stop caused by issues with aging network equipment (including routers during a copper-to-fiber transition). It was increased from a prior ~$30 billion ask. This is additional proposed funding, not money already provided.

Spending status of the existing $12.5 billion​

Progress has been made (e.g., roughly 50–68% of legacy copper telecommunications lines replaced with fiber optic cable in parts of the system, plus some radio and switch upgrades), but a substantial share remains unspent or only partially obligated:

  • As of March 2026 (GAO data), about $1.8 billion (roughly 20% of the relevant systems appropriation) had been obligated.
  • By mid-November 2026, Democratic lawmakers cited roughly $4.3 billion obligated and only about $1 billion actually spent/outlaid (around 8% of the total). Large portions remained unobligated.
GAO reports have noted challenges with cost estimating, integrated scheduling (thousands of individual project schedules), and underestimation of some Phase 1 costs (which rose toward or beyond the original allocation in places). The agency has been using some of its regular facilities-and-equipment budget to help cover gaps while seeking more funding for Phase 2 (automation/software) and a fuller overhaul.

In short: The FAA received a significant but much smaller appropriation ($12.5B) in 2025 and has spent only a fraction of it so far while making partial progress on telecom and related upgrades. The $50B figure is a current political request for substantially more money, not prior funding sitting unused.
 
The correct number allocated was $12.5 billion, not $50 billion. The $1 billion spent to date was accurate. My apologies.
 
Here are some examples of misconfigured routers causing network outages. I will be interested to see if the actual cause of the FAA network failure is fully described and publicly released.
Yes. Router (and related control-plane) misconfigurations have repeatedly caused large-scale outages in complex networks of routers and switches. These often involve BGP route leaks/announcements, filter removals, Flowspec rules, or simple policy errors that cascade through interconnected devices. Here are well-documented real-world instances:

CenturyLink / Level 3 (August 30, 2020)​

A misconfigured BGP Flowspec rule (intended to block traffic for a single customer IP address) was applied with wildcards instead of a specific address. This prevented BGP sessions from establishing properly across much of the network’s control plane.
The result was widespread packet loss and outages lasting roughly 5 hours that affected CenturyLink’s own infrastructure plus many downstream customers and services (Cloudflare, Hulu, Steam, Discord, Xbox Live, Reddit, Amazon-related traffic, and others). Global web traffic dropped by an estimated 3.5% at the peak. The network is a major Tier-1 backbone with extensive router interconnections.

Rogers Communications (Canada, July 8, 2022)​

During a multi-phase core IP network upgrade, staff removed an Access Control List (ACL) / routing filter from distribution routers. This flooded the core routers with excessive IP routing information, overwhelming their capacity and causing them to fail.
The outage lasted ~15–24+ hours, took down wireless and wireline services for more than 12 million customers nationwide (including emergency/911-related impacts in some cases), and disrupted internal management access. Independent reviews confirmed human configuration error plus missing overload protections and change-management gaps as key factors.

KDDI (Japan, July 2–4, 2022)​

A router path/route misconfiguration during maintenance on a nationwide transport/core network router (caused by using an outdated procedure manual) created one-way signal issues. This triggered massive retransmissions of location-registration requests, congesting VoLTE nodes and cascading into the subscriber database.
The outage lasted about 61 hours and affected tens of millions of voice and data users across Japan. It began with a relatively localized router config error but propagated through the interconnected network of routers and switches.

Cloudflare examples​

  • July 2020 (Atlanta router): A configuration change intended to relieve congestion on one router incorrectly directed essentially all backbone traffic through Atlanta. This overwhelmed the router and caused failures at many Points of Presence (PoPs) worldwide for ~27 minutes, with traffic drops of ~50% in affected parts of the network.
  • January 2026 (Miami BGP policy): Engineers adjusting export policies accidentally made them overly permissive. Internal IPv6 routes were advertised externally, creating a ~25-minute route leak that caused congestion, packet loss, and roughly 12 Gbps of dropped traffic.

Other notable cases​

  • Level 3 route leak affecting Comcast (November 2017): A configuration error led to Level 3 leaking over a thousand more-specific prefixes belonging to Comcast and its customers. Traffic was improperly attracted, producing nationwide outages and packet loss for Comcast users (and ripple effects on other providers).
  • AT&T Mobility (February 2024): A network element misconfiguration during maintenance triggered an automated shutdown. It disrupted service for ~125 million devices for about 12 hours and blocked tens of millions of voice calls (including thousands of 911 attempts). FCC findings pointed to the misconfiguration plus inadequate peer review, testing, and controls.
These incidents illustrate a common pattern: a single incorrect policy, filter, Flowspec rule, or route advertisement on one or a few routers can rapidly propagate through BGP or internal routing protocols, overwhelming control planes or data planes across a large mesh of routers and switches. Complex, highly interconnected networks (carrier backbones, CDNs, national mobile cores) amplify the impact because of the density of peering, redistribution, and dependency relationships. Change-management failures, missing validation/filters, and lack of overload protection frequently turn small errors into large outages.
 
@kenn10: If you review the examples you have documented, you will note that virtually all of them were initiated by system-level changes rather than an individual router going haywire. I can only speak to the system I used for more than two decades that was much like the FAA's network. It was managed by Sprint, a company that one of my employees discovered was using a single administrator password for every router in the federal judiciary. Don't ask how we knew. Despite Sprint's incompetence, I do not recall a single instance of our entire nationwide network going down ever, and individual router updates, upgrades, replacements, and new installs were commonplace. But maybe we were just lucky, and we certainly would not have rolled out anything new on a Friday afternoon.
 
Last edited:
We'll have to wait to see if a root cause analysis is released. HP Enterprise maintains their router was working as designed and indirectly implied that programming was not set up correctly.

I personally witnessed a complete county network shutdown which took all Cisco phone systems and the entire data network out of service. It was caused by a misconfigured router at the DOT camera monitoring center. Cisco experts were brought in quickly and discovered erroneous updates on one router that propagated across some 100+ routers in the network. Even the VPN's could not get enough bandwidth for the phones to work. The jail had its own IT people and was able to recover by unplugging the fiber link to the county network and rebooting their router. Their servers and Cisco voip servers were local in the building and resumed connection to the workstations and phones. Keeping track of 3,200 inmates is vital. Fortunately, the 911 center's computers were on a separate network with only passive one-way feeds into other computer systems.

For most of the county, the outage lasted most of the workday before the cause was found. The router change had not been submitted through change control or scheduling. Heads rolled, to say the least. A hotshot know-it-all new data network analyst had made the unapproved modification which caused a flood that choked the whole network as the routers went into an endless loop of some sort.
 
I would replace the paperclip with a smiling burger and change the last portion of the tagline to "Would you like to supersize it?"
 

Forum statistics

Threads
26,863
Messages
175,673
Members
20,402
Latest member
luxploit
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.
Back
Top