<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Signal Analysis]]></title><description><![CDATA[Signal Analysis]]></description><link>https://structuredsignal.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Wed, 18 Nov 2026 14:19:30 GMT</lastBuildDate><atom:link href="https://structuredsignal.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[When 2020s Fiber Meets 1980s Copper: Inside the FAA Telecommunications Ground Stop]]></title><description><![CDATA[Potential root-cause analysis of the FAA ground stop: how an L3Harris TDM-to-fiber cutover triggered a 200-req/sec Juniper-to-Cisco logon storm.]]></description><link>https://structuredsignal.hashnode.dev/when-2020s-fiber-meets-1980s-copper-inside-the-faa-telecommunications-ground-stop</link><guid isPermaLink="true">https://structuredsignal.hashnode.dev/when-2020s-fiber-meets-1980s-copper-inside-the-faa-telecommunications-ground-stop</guid><category><![CDATA[FAA]]></category><category><![CDATA[Outage]]></category><category><![CDATA[networking]]></category><dc:creator><![CDATA[structuredsignal]]></dc:creator><pubDate>Sat, 14 Nov 2026 19:14:30 GMT</pubDate><content:encoded><![CDATA[<blockquote>
<p><strong>TL;DR:</strong> According to the preliminary account given by DOT and FAA officials on 14 November 2026, the nationwide Federal Aviation Administration (FAA) ground stop of 13 November was not caused by a cyberattack. It began during a routine copper to fiber switchover by legacy sustainment contractor L3Harris: two network points began firing over 1,000 connection attempts every 4 to 5 seconds (~200 to 250 per second), flooding legacy Cisco routers across the FAA's single unified telecommunications backbone. Officials have not yet determined whether the trigger was a configuration or installation error or a faulty router. This article's reading is that it was an <strong>in band control plane signaling flood</strong> that turned automatic failover pathways into vectors for cascading instability. That reading is a hypothesis, not an official finding.</p>
</blockquote>
<h2>1. The Layered Network Architecture Behind the Outage</h2>
<p>To understand how a routine maintenance action at a remote facility could halt flights nationwide, it is necessary to separate the FAA's modern air traffic software applications from the aging physical transport backbone on which they ride. This article groups what Transportation Secretary Sean Duffy, FAA Administrator Bryan Bedford and the FAA's acting chief technology officer described in their briefing (<a href="https://www.youtube.com/watch?v=MOZuCknTxJ8">https://www.youtube.com/watch?v=MOZuCknTxJ8</a>; <a href="https://www.transportation.gov/briefing-room/icymi-us-transportation-secretary-sean-p-duffy-provides-update-restoration-airspace">DOT excerpts</a>) into four technological layers operating simultaneously within the National Airspace System (NAS).</p>
<h3>1.1 The FTI / TDMI Legacy Backbone (The Point of Failure)</h3>
<p>At the heart of the outage sits the <strong>FAA Telecommunications Infrastructure (FTI)</strong>, the network that L3Harris (originally Harris Corp.) <a href="https://www.telecomreviewamericas.com/articles/telecom-operators/att-to-drive-faas-next-generation-communications-backbone/">has run for the FAA since 2002</a>. Administrator Bedford described it as a 1980s <strong>Time Division Multiplexing (TDM/TDMI)</strong> "1G" telco network. That label refers to the circuit technology it still carries, typified by legacy <strong>T1 copper lines</strong> (1.544 Mbps serial circuits), rather than to the age of the FTI program itself. Bedford said the FAA spends roughly $1 billion a year merely maintaining and sustaining this infrastructure. Officials have previously also pointed to carrier "obsolescence fees" charged to keep discontinued copper circuits in service. Crucially, in Bedford's words the FAA has <strong>only one</strong> federal telephony infrastructure, not two physically and logically independent networks.</p>
<h3>1.2 The High Speed Fiber Transition Layer</h3>
<p>Because commercial telecom carriers are actively decommissioning legacy copper plant across the United States, the FAA is continuously migrating individual circuits from copper to fiber, and from TDM to IP. Roughly 70% of the copper has been converted, leaving about 30% on legacy circuits. According to officials, the nationwide fault originated at a remote FAA site undergoing a routine copper to fiber cutover, with a router being replaced as part of the work, around 3:00 PM ET on Friday.</p>
<h3>1.3 Operational Air Traffic Control Systems (ERAM &amp; STARS)</h3>
<p>Riding atop the FTI telecommunications transport layer are the mission critical air traffic management platforms:</p>
<ul>
<li><strong>En Route Automation Modernization (ERAM) &amp; ARTCCs:</strong> ERAM is the high altitude automation platform at <a href="https://www.faa.gov/speeches/en-route-automation-modernization">the 20 en route centers in the continental United States</a>. The outage was counted across the FAA's 22 En Route facilities. Officials described a telecommunications problem rather than an automation failure (Bedford said no facility "went dark"), but instability in the underlying FTI transport network degraded air to ground voice communications and radar surveillance feeds at <a href="https://www.cnn.com/2026/11/14/politics/faa-ground-stop-what-we-know">as many as <strong>17 of the nation's 22 En Route facilities</strong></a>.</li>
<li><strong>STARS (Standard Terminal Automation Replacement System) &amp; TRACONs:</strong> Low altitude Terminal Radar Approach Control facilities. Bedford said the instability in information feeds hit the En Route centers first and <a href="https://www.cnbc.com/2026/11/14/faa-ground-stop-flights-resume-delays.html">eventually reached facilities closer to airports</a>.</li>
</ul>
<h3>1.4 Newer Digital Systems (Not Implicated)</h3>
<p>Officials separated the legacy network from newer systems. Bedford said there was zero evidence of anything AI related. That covers the <strong>SMART system</strong> (Strategic Management of Airspace, Routes and Trajectories), the AI supported traffic planning tool the FAA <a href="https://avweb.com/aviation-news/faa-expands-ai-assisted-traffic-management/">expanded to all 22 En Route centers on 10 November</a>; its developer, Air Space Intelligence, also said it was not involved. Bedford cautioned against conflating the incident with the modernization program, and Duffy said on Friday night that newer IP switches stayed up while legacy equipment failed. In Saturday's briefing Duffy listed work already delivered under the initial $12.5 billion congressional allocation: <strong>Surface Awareness Initiative (SAI)</strong> technology at 143 towers, <strong>469 converted radio sites</strong>, <strong>Electronic Flight Strips</strong> at 25 towers, <strong>224 IP Voice Switch systems</strong>, and <strong>9 SMR4 Surface Movement Radars</strong>. This article has no data on how each of those systems behaved during the outage.</p>
<hr />
<h2>2. Contractor &amp; Vendor Ecosystem Matrix</h2>
<p>A critical point of confusion in public reporting is which contractor was responsible for the cutover and how hardware vendors fit into the architecture. Administrator Bedford drew a sharp boundary between legacy FTI sustainment work and the next generation modernization program.</p>
<table>
<thead>
<tr>
<th align="left">Contractor / Vendor</th>
<th align="left">Network Scope &amp; Role</th>
<th align="left">Involvement &amp; Status in the Outage</th>
</tr>
</thead>
<tbody><tr>
<td align="left"><strong>L3Harris</strong></td>
<td align="left">Prime contractor and owner of the legacy <strong>FTI telecom network</strong>.</td>
<td align="left"><strong>Direct Involvement:</strong> L3Harris owns and sustains the legacy FTI backbone. The fault occurred during L3Harris sustainment work switching a remote site from copper to fiber. The FAA is working to determine whether the root cause was installer or configuration error, or a faulty router.</td>
</tr>
<tr>
<td align="left"><strong>Cisco Systems</strong></td>
<td align="left">Legacy router hardware manufacturer (FTI backbone).</td>
<td align="left"><strong>Being Phased Out:</strong> Bedford said the FAA is replacing roughly 2,000 legacy Cisco routers, which the DOT transcript renders as "Cisco 600" units. <a href="https://www.cbsnews.com/news/faa-ground-stop-canceled-flights-travelers-what-we-know/">CBS News's government sources</a> describe Cisco 600 series routers built between 2001 and 2006. The exact model is unconfirmed. Officials said these routers were flooded during the event.</td>
</tr>
<tr>
<td align="left"><strong>Juniper Networks</strong></td>
<td align="left">Replacement router hardware manufacturer (fiber transition).</td>
<td align="left"><strong>Likely Involvement (inferred):</strong> Bedford said the legacy Cisco routers are being replaced with Juniper units, so the new router at the cutover site was presumably a Juniper. The FAA disabled a function, not publicly named, on the roughly 60 routers of the same type in use nationwide.</td>
</tr>
<tr>
<td align="left"><strong>Peraton</strong></td>
<td align="left">Prime integrator for the FAA's <strong>$12.5B</strong> modernization effort.</td>
<td align="left"><strong>Zero Involvement:</strong> Bedford said the work was routine FTI sustainment under L3Harris, <em>not</em> part of the modernization effort led by Peraton.</td>
</tr>
<tr>
<td align="left"><strong>AT&amp;T, Verizon, &amp; Lumen</strong></td>
<td align="left">Next generation commercial telecommunications partners.</td>
<td align="left"><strong>Future Architecture:</strong> <a href="https://www.foxbusiness.com/fox-news-air-space/who-l3harris-federal-contractor-sec-duffy-says-may-have-caused-faa-outage">Named by Duffy</a> as the companies working with the FAA on the new architecture that will replace the single L3Harris FTI backbone.</td>
</tr>
</tbody></table>
<hr />
<h2>3. Macro Architecture &amp; How Redundancy Failed</h2>
<p>FAA facilities are engineered with layered redundancy: Bedford described several backup channels behind each primary. How could a router swap at a remote site get past that redundancy and degrade 17 of 22 En Route centers? Points 1, 2 and 5 below come from the briefing. Points 3 and 4 are this article's hypothesis.</p>
<ol>
<li><strong>Shared Fate on a Single Network (per Bedford):</strong> While individual facilities maintain primary and backup channels for voice and radar data, all of those channels traverse <strong>one single federal telephony infrastructure (FTI)</strong>. In this article's reading, redundant channels protect against a clean physical line cut, but they do not isolate an active <strong>in band control plane flood</strong> across a shared routing domain.</li>
<li><strong>The 1,000 Attempt Connection Storm (per officials):</strong> During the L3Harris cutover from copper to fiber, <a href="https://techcrunch.com/2026/11/14/transportation-secretary-blames-routine-equipment-upgrade-for-grounding-flights-nationwide/">two network points</a> began generating <strong>over 1,000 connection attempts every 4 to 5 seconds</strong> (~200 to 250 attempts per second).</li>
<li><strong>Control Plane Exhaustion on Legacy Hardware (hypothesis):</strong> Officials said the legacy routers were flooded and overloaded. A plausible mechanism is control plane saturation: 200+ session establishment events per second driving the router CPU to 100% and starving legitimate radar and air to ground voice packets. The exact router model, and therefore its real capacity, is unconfirmed. (Cisco's historical 600 series was a DSL customer premises line, so the designation given at the briefing is probably shorthand for another model. This article's modelling assumes a 2600 class access router of that era: a single slow CPU that handles session setup in software, with no hardware enforced control plane protection.)</li>
<li><strong>Failover as a Contagion Vector (hypothesis):</strong> Because the fault was an active traffic flood rather than a dead wire, automatic failover logic may have treated the overwhelmed routers as degraded and shifted traffic, <strong>including the errant connection flood itself</strong>, onto backup channels and adjacent facilities' routers. That would explain how the instability reached 17 centers, but officials have not described the propagation path.</li>
<li><strong>Paper Logging Delays (The 45 Minute Search):</strong> Bedford said maintenance on the legacy network is logged on paper, so officials had to search paper records to work out who was working where. <a href="https://www.yahoo.com/news/us/live/faa-ground-stop-latest-updates-duffy-blames-contractor-l3harris-thousands-more-flights-canceled-saturday-no-sign-of-cyberattack-161329174.html">Tracing the source took about 45 minutes</a>. Once the two sources were disabled, the network stabilized, the FAA's acting CTO said. <a href="https://abcnews.com/US/timeline-faa-nationwide-ground-stop-developed/story?id=138141388">Publicly reported times</a> (ET): switchover work around 3:00 PM; instability in radar and communications from 3:36 PM; nationwide ground stop ordered at 3:54 PM; ground stop lifted at about 8:15 PM. Accounts differ on whether the 45 minutes was the time needed to trace the source or the period of unexplained instability that preceded the ground stop, and the exact time the sources were isolated has not been clearly reported.</li>
</ol>
<img src="https://cdn.hashnode.com/uploads/covers/6af89ee39518d753b71979d7/e158de45-e312-43a0-8162-392c7fc7de82.png" alt="" style="display:block;margin:0 auto" />

<p><em>Figure 1: Macro topology of the FAA FTI outage as reconstructed in this article. Primary and backup channels share a single FTI backbone (per Bedford). The hypothesis shown is that a connection flood of roughly 220 attempts per second from the fiber cutover saturated the control planes of legacy Cisco routers (modelled as 2600 class) and cascaded via automatic failover across 17 of 22 En Route centers.</em></p>
<h3>3.1 Four Stage Incident Progression &amp; State Timeline</h3>
<p>To capture the progression of the outage from normal operations to emergency containment, Figure 2 illustrates four sequential operational states of the FTI backbone between 3:00 PM ET and recovery, as reconstructed in this article. It is followed by a node by node readout that combines what officials said with this article's assumptions, marked as such.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af89ee39518d753b71979d7/4c241192-b5a0-4b48-865d-776f848838b1.png" alt="" style="display:block;margin:0 auto" />

<p><em>Figure 2: Four stage reconstruction of the FAA FTI incident, tracing how a remote cutover escalated into a nationwide ground stop. The paper maintenance logs are from the briefing; failover across the shared backbone as the propagation path is this article's hypothesis.</em></p>
<h3>3.2 Node by Node Readout (All 6 Architecture Nodes)</h3>
<table>
<thead>
<tr>
<th align="left">Architecture Node</th>
<th align="left">Hardware &amp; Transport Spec</th>
<th align="left">Owner / Integrator</th>
<th align="left">Role in Incident &amp; State Behavior</th>
</tr>
</thead>
<tbody><tr>
<td align="left"><strong>Node 1: Remote Cutover Site</strong></td>
<td align="left">Legacy T1 copper (1.544 Mbps) transitioning to fiber; new router (presumably Juniper) replacing a legacy Cisco unit.</td>
<td align="left">L3Harris (FTI sustainment)</td>
<td align="left"><strong>Origin of Fault:</strong> After the switchover at about 3:00 PM ET, two points fired &gt;1,000 connection attempts every 4 to 5 seconds (~220/sec), per officials. A tight retry loop is this article's interpretation. Traced in about 45 minutes, then disabled.</td>
</tr>
<tr>
<td align="left"><strong>Node 2: FTI Legacy Backbone</strong></td>
<td align="left">"1980s" TDM / TDMI "1G" telco network (Bedford's label); ~2,000 legacy Cisco routers ("Cisco 600" per Bedford; built 2001 to 2006 per CBS sources; exact model unconfirmed).</td>
<td align="left">L3Harris (about $1B/yr FAA sustainment spend)</td>
<td align="left"><strong>Contagion Medium:</strong> Single federal telephony backbone. Hypothesis: without hardware enforced control plane protection, a connection flood can saturate router CPUs.</td>
</tr>
<tr>
<td align="left"><strong>Node 3: Redundancy Layer</strong></td>
<td align="left">Primary and backup logical channels over shared FTI transport.</td>
<td align="left">FAA / L3Harris shared network</td>
<td align="left"><strong>Bypass Mechanism (hypothesis):</strong> Redundancy protects against line faults, though <a href="https://www.foxbusiness.com/economy/northeast-airport-operations-resume-telecom-outage-disrupts-flights">the 21 September Northeast outage</a> showed even that can fail (a primary circuit failed at Philadelphia TRACON and the backup fiber had been cut in New Jersey). It can also shift an active packet flood onto backup paths during failover.</td>
</tr>
<tr>
<td align="left"><strong>Node 4: En Route Centers (ARTCCs)</strong></td>
<td align="left">22 En Route facilities; ERAM automation at the 20 in the continental US.</td>
<td align="left">FAA Air Traffic Organization</td>
<td align="left"><strong>Primary Victim:</strong> No facility went dark (Bedford), but voice (air to ground) and radar feeds were degraded at as many as <strong>17 of the 22 En Route centers</strong>. Router saturation as the cause is this article's hypothesis.</td>
</tr>
<tr>
<td align="left"><strong>Node 5: TRACON Facilities</strong></td>
<td align="left">Low altitude terminal radar approach facilities running STARS automation.</td>
<td align="left">FAA Terminal Operations</td>
<td align="left"><strong>Downstream Cascade:</strong> Bedford said the instability eventually reached facilities closer to airports. The dependency path (via En Route FTI hubs) is assumed here.</td>
</tr>
<tr>
<td align="left"><strong>Node 6: Newer Digital Systems</strong></td>
<td align="left">SMART planning tool, 143 SAI towers, 469 radio sites, 224 IP voice switches, 9 SMR4 radars.</td>
<td align="left">FAA / modernization vendors ($12.5B initial allocation)</td>
<td align="left"><strong>Not Implicated:</strong> Officials reported zero evidence of AI involvement, and Duffy said newer IP switches stayed up. This article has no data on each individual system.</td>
</tr>
</tbody></table>
<h3>3.3 Architectural Comparison: Legacy FTI vs. Proposed Multicarrier IP Architecture</h3>
<p>A central policy theme of the DOT/FAA briefing was funding. Duffy called the $12.5 billion already appropriated a partial payment; after the outage he <a href="https://thehill.com/policy/transportation/6220399-duffy-billions-air-traffic-ground-stop/">asked Congress for $50 billion</a>, up from roughly $30 billion requested earlier. He tied that money to replacing the single L3Harris FTI network with a new architecture built with AT&amp;T, Verizon, and Lumen. Peraton is the prime integrator for the modernization effort.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af89ee39518d753b71979d7/4eaab7a4-e8c0-4b50-8c9c-a64ac299dabe.png" alt="" style="display:block;margin:0 auto" />

<p><em>Figure 3: Architectural comparison between the legacy L3Harris FTI backbone and the planned multicarrier IP architecture (Peraton as integrator; AT&amp;T, Verizon, and Lumen named by Duffy as partners).</em></p>
<hr />
<h2>4. Forensic Assessment: Why Misconfiguration Is the Leading Hypothesis</h2>
<p>During the briefing, officials stated they were investigating whether the root cause was <strong>user/installer misconfiguration</strong> or a <strong>faulty router</strong>. Neither has been ruled out. From a network engineering standpoint, three indicators favour a configuration or protocol mismatch loop during the cutover:</p>
<ul>
<li><strong>The ~60 Similar Units:</strong> The FAA's acting CTO said roughly 60 routers of the same type are in use nationwide, and no similar event has been publicly attributed to them. If the hardware design or base firmware were intrinsically defective, earlier deployments would be expected to have triggered the same flood. A fault erupting at the moment a remote site was cut over from copper to fiber points toward a provisioning error specific to that site. A single defective unit, which officials have explicitly left open, would look the same from outside.</li>
<li><strong>Two Network Points "Chattering" at 200+ Hz:</strong> A dead optical transceiver or failed ASIC typically stops passing traffic; it does not methodically generate over 1,000 connection attempts every 4 to 5 seconds from two points. That pattern is more characteristic of a protocol retry loop.</li>
<li><strong>Fleetwide Function Disable as the Fix:</strong> As a precaution, the FAA disabled a <em>specific function</em> on the roughly 60 routers of that type. The function has not been named. In enterprise telecom operations it would typically be an aggressive automated recovery, fast keepalive, or autonegotiation feature that acts as a force multiplier when paired with a legacy TDM link, but that is conjecture.</li>
</ul>
<hr />
<h2>5. Five Cutover Configuration Mistakes That Could Produce a 1,000 Attempt Connection Storm</h2>
<p>Bridging a 2020s high speed fiber router (assumed here to run Juniper Junos) into a legacy TDM T1 copper network populated by Cisco IOS routers built in the early 2000s requires careful timing alignment. Below are five cutover configuration mistakes capable of producing a connection storm of 200 to 250 attempts per second. These are hypotheses: nothing officials have released confirms or excludes any of them, and the legacy router model assumed (a low end IOS platform that forwards in software) is unconfirmed.</p>
<h3>5.1 Subsecond BFD (Bidirectional Forwarding Detection) Over Legacy Emulated Circuits</h3>
<p>Modern routers use <strong>BFD</strong> to detect path failures in milliseconds rather than waiting for routing protocol timers (OSPF's default dead interval is 40 seconds; BGP hold times are typically 90 to 180 seconds). On native fiber links, BFD can be run at intervals of <strong>10 ms to 50 ms</strong> with a multiplier of 3. If an installer applies a BFD template tuned for fiber toward a path that includes a legacy 1.544 Mbps T1 circuit or a slow legacy router CPU, and the legacy side runs BFD at all, serialization latency and processing delay can cause it to miss the detection window of 30 to 150 ms. The new router then declares the session down, tears down the dependent routing adjacency, and starts again. Repeated across many sessions or subinterfaces, that could add up to a high aggregate rate of connection attempts.</p>
<h3>5.2 Undamped Reconnects After an Authentication or Capability Mismatch</h3>
<p>Session protocols normally damp reconnection: BGP implementations wait between connection attempts (idle hold and connect retry timers, often with backoff), and PPP uses restart timers. A connection storm therefore implies that damping was absent, misconfigured, or defeated. One way that happens: the lower layer comes up cleanly every time (over fiber, link or TCP setup completes in a couple of milliseconds) and the failure only occurs one step later, at session setup. Examples are a PPP CHAP/PAP credential mismatch, or a BGP OPEN rejected for a wrong peer AS or for a capability that an IOS release from the early 2000s does not support. If each attempt is treated as a fresh start rather than a repeated failure, the endpoint retries immediately. One attempt every 4 to 5 ms is exactly 1,000 attempts in 4 to 5 seconds.</p>
<h3>5.3 Recursive Tunnel Routing ("Tunnel Over Self" Flapping Loop)</h3>
<p>To carry legacy radar and voice serial traffic over fiber, routers may encapsulate packets inside logical tunnels (GRE, mGRE, or IPsec). When cutting over from copper to fiber, the new fiber interface is assigned a preferable routing metric. If the technician forgets to apply a route filter preventing the tunnel from learning its own physical endpoint IP address <em>through the tunnel itself</em>, the following loop occurs: (1) the tunnel establishes over fiber; (2) a better route to the tunnel's own endpoint is learned through the tunnel; (3) the router shifts the tunnel's underlying transport into the tunnel itself; (4) the tunnel collapses; (5) the underlying fiber route becomes valid again and the tunnel comes back up. The tunnel flaps for as long as the misconfiguration persists, and every flap restarts the sessions riding on it.</p>
<h3>5.4 TDM Pseudowire / Circuit Emulation (CEM / SAToP) Clock Slippage</h3>
<p>Because about 30% of the FTI network remains on TDM copper, one way to carry the remaining T1 circuits across the new fiber is <strong>Structure Agnostic TDM over Packet (SAToP)</strong> or another form of circuit emulation, which converts T1 time slots into IP packets. TDM requires strict clock synchronization. If both the new router and the remote legacy endpoint are misconfigured as <code>clock source line</code> (loop timed, each waiting for the other to supply the master clock), or are left on unsynchronized internal clocks, buffer underruns and overruns ("frame slips") repeatedly corrupt frames on the emulated circuit. That can trigger repeated link resets and PPP CHAP/PAP reauthentication.</p>
<h3>5.5 Dual Active "Make Before Break" Cutover Clash</h3>
<p>During live cutovers, technicians often bring the new fiber router online before disconnecting the old T1 copper router ("make before break"). If both the new router and the legacy Cisco router remain active on the same local segment sharing the same <strong>loopback IP, OSPF/BGP router ID, or VRRP group</strong> without being isolated from each other, the two routers can continuously evict each other's sessions at the upstream En Route center, generating a bidirectional reauthentication storm between the two points.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af89ee39518d753b71979d7/f804b74f-59e7-4ed2-9c70-fd4ba01c847f.png" alt="" style="display:block;margin:0 auto" />

<p><br /><em>Figure 4: Hypothesised mechanics of a connection storm of 1,000 attempts per 5 seconds. If lower layer setup over fiber completes in about 2 ms and reconnects are not damped, a failure at the next stage (upper layer session setup, a BFD timer, or TDM clocking) produces an unthrottled retry loop of one attempt every 4 to 5 ms.</em></p>
<h3>5.6 Illustrative Model Across Operational Modes &amp; Example Configuration</h3>
<p>To illustrate the potential quantitative impact of such a storm on legacy routers, Figure 5 and Figure 6 compare three modelled operating modes and plot a cumulative connection flood over 5 seconds against an assumed CPU saturation curve for a legacy router. <strong>These are modelled, illustrative values, not FAA telemetry. No measurements from the incident have been published.</strong></p>
<img src="https://cdn.hashnode.com/uploads/covers/6af89ee39518d753b71979d7/14e5a539-4e02-4b28-9d0a-a042910786db.png" alt="" style="display:block;margin:0 auto" />

<p><br /><em>Figure 5: Modelled comparison across Mode 1 (misconfigured cutover storm), Mode 2 (legacy TDM baseline), and Mode 3 (mitigated cutover). Illustrative values, not measured telemetry.</em></p>
<img src="https://cdn.hashnode.com/uploads/covers/6af89ee39518d753b71979d7/1d62a0ea-7be9-4ee8-90fe-02f660047b26.png" alt="" style="display:block;margin:0 auto" />

<p><em>Figure 6: Modelled cumulative connection storm over 5 seconds (~1,100 attempts) vs. assumed legacy router CPU saturation (reaching 100% at t = 1.5 seconds). Illustrative, not measured.</em></p>
<table>
<thead>
<tr>
<th align="left">Operational Mode (modelled)</th>
<th align="left">Connection Attempts / 5 Sec</th>
<th align="left">Retry Interval</th>
<th align="left">Legacy Router CPU Load (assumed)</th>
<th align="left">Modelled Outcome</th>
</tr>
</thead>
<tbody><tr>
<td align="left"><strong>Mode 1: Misconfigured Cutover Storm</strong></td>
<td align="left"><strong>1,000 to 1,250 (~220/sec)</strong></td>
<td align="left"><strong>4 to 5 ms (unthrottled)</strong></td>
<td align="left"><strong>100% saturated (by t = 1.5s)</strong></td>
<td align="left">Voice &amp; radar packets dropped; failover spreads the storm (hypothesis).</td>
</tr>
<tr>
<td align="left"><strong>Mode 2: Legacy TDM Baseline</strong></td>
<td align="left"><code>&lt; 1 attempt / 30 sec</code></td>
<td align="left"><code>10s keepalive / 30s hold</code></td>
<td align="left"><strong>18% nominal load</strong></td>
<td align="left">Normal steady state operation over 1.544 Mbps T1 serial circuits.</td>
</tr>
<tr>
<td align="left"><strong>Mode 3: Mitigated Cutover</strong></td>
<td align="left"><code>&lt;= 1 attempt / 30 sec</code></td>
<td align="left"><code>30s, backing off to 120s</code></td>
<td align="left"><strong>22% stabilized load</strong></td>
<td align="left">Function disabled on the roughly 60 routers of the same type (as officials reported); network stable.</td>
</tr>
</tbody></table>
<p>Below is an illustrative Junos sketch of the aggressive timer and session mismatch mistakes described in 5.1 and 5.2, alongside a conservative alternative. It is not the FAA's or L3Harris's configuration: the group name, address and AS number are invented, and the function the FAA actually disabled has not been named.</p>
<pre><code class="language-javascript">/* 1. ILLUSTRATIVE MISCONFIGURED CUTOVER (Juniper Junos OS, aggressive fiber defaults toward a legacy peer) */
protocols {
    bgp {
        group FTI_ENROUTE_CORE {
            peer-as 65001;                 /* If this does not match the legacy peer's AS, every OPEN is rejected after TCP connects */
            hold-time 3;                   /* Aggressive hold time instead of a legacy compatible 90s */
            bfd-liveness-detection {
                minimum-interval 10;       /* 10 ms BFD: too aggressive for a legacy T1 path or slow legacy CPU */
                multiplier 3;              /* 30 ms detection: session torn down and retried on any delay */
            }
            neighbor 10.24.118.1;
        }
    }
}

/* 2. ILLUSTRATIVE CONSERVATIVE CONFIGURATION (hypothetical mitigation) */
protocols {
    bgp {
        group FTI_ENROUTE_CORE {
            hold-time 90;                  /* Legacy compatible 30s keepalive / 90s hold */
            damping;                       /* Route flap damping: limits churn propagated from a flapping peer */
            passive;                       /* Do not initiate sessions; wait for the legacy peer to open */
            /* BFD liveness detection removed */
        }
    }
}
</code></pre>
<hr />
<h2>6. Key Takeaways for Modernizing Safety Critical Networks</h2>
<p>Whatever the final root cause, the FAA outage offers a lesson for critical infrastructure operators transitioning from 20th century serial TDM circuits to modern IP fiber networks:</p>
<ul>
<li><strong>Logical Redundancy Is Not Fault Isolation:</strong> Running primary and backup channels over a <em>single</em> network (FTI) creates a shared fate vulnerability, a point Bedford himself made. True resilience requires partitioned routing domains across multiple carriers (as intended under the planned Peraton / AT&amp;T / Verizon / Lumen architecture) so that instability in one zone cannot spread to 17 En Route centers.</li>
<li><strong>Boundary Protection (Control Plane Policing):</strong> Whenever a modern high speed router is connected to a legacy network containing control planes that are 20 years old, strict ingress rate limiting, backoff enforcement, and route flap damping should be configured on the modern edge device to shield legacy upstream CPUs.</li>
<li><strong>Real Time Digital Change Management:</strong> Relying on manual paper maintenance logs in a national air traffic network helped turn what could have been a quick remote shutdown into a search of roughly 45 minutes and a nationwide ground stop. Automated real time telemetry and digital change tracking are just as critical as fiber optic bandwidth.</li>
</ul>
<hr />
<h2>7. Why Engineering Forensics Matters: Debunking Outage Conspiracy Theories</h2>
<p>Whenever a nationwide aviation ground stop or major infrastructure outage occurs, social media is immediately flooded with conspiracy theories alleging coordinated cyber warfare, electromagnetic attacks, or intentional sabotage. Disseminating unverified conspiracy theories during critical infrastructure incidents is harmful to the public and counterproductive to aviation safety for three concrete reasons:</p>
<ul>
<li><strong>Complex Hybrid Networks Fail in Predictable, Mechanistic Ways:</strong> As argued throughout this analysis, bridging a 2020s high speed fiber router (capable of passing packets at submillisecond intervals) into a legacy TDM copper network running routers built in the early 2000s creates extreme timing asymmetries. A simple configuration mismatch, such as a 10 millisecond BFD timer or an undamped reconnect loop, can mechanically produce hundreds of connection attempts per second without any malicious actor required.</li>
<li><strong>The Recovery Pattern Points to an Internal Fault:</strong> Destructive cyberattacks (such as wiper malware, firmware corruption, or distributed denial of service botnets) are not normally resolved by reading a paper maintenance log and disabling two sources tied to scheduled contractor maintenance. That the network stabilized once those sources were disabled, by officials' account, is strong evidence of an internal fault, and it is consistent with their statement that they found zero evidence of anything nefarious. It does not by itself establish which internal fault; that investigation is ongoing.</li>
<li><strong>Misinformation Distracts from Real Infrastructure Modernization Needs:</strong> Attributing every technical failure to shadowy external plots obscures the actual, mundane engineering challenge facing public infrastructure: <strong>technical debt</strong>. Paying to keep discontinued copper circuits alive, maintaining routers built two decades ago, and tracking maintenance on paper logs are real, solvable budget and engineering problems. Understanding the true mechanics of legacy network failures builds informed public support for the modernization investments needed to prevent them.</li>
</ul>
<hr />
<h2>References</h2>
<h3>Event record and official statements</h3>
<ul>
<li><a href="https://www.transportation.gov/briefing-room/icymi-us-transportation-secretary-sean-p-duffy-provides-update-restoration-airspace">U.S. DOT: Secretary Duffy's update on restoration of the airspace</a>: official excerpts from the 14 November 2026 briefing, with a link to the full press conference.</li>
<li><a href="https://www.faa.gov/newsroom/faa-statement-nationwide-ground-stop">FAA statement on the nationwide ground stop</a>: the agency's own statement on the event.</li>
<li><a href="https://abcnews.com/US/timeline-faa-nationwide-ground-stop-developed/story?id=138141388">ABC News: how the ground stop developed</a>: timeline of 13 November 2026.</li>
<li><a href="https://www.cbsnews.com/news/faa-ground-stop-canceled-flights-travelers-what-we-know/">CBS News: what we know</a>: summary of the ground stop, the canceled flights and the equipment involved.</li>
<li><a href="https://techcrunch.com/2026/11/14/transportation-secretary-blames-routine-equipment-upgrade-for-grounding-flights-nationwide/">TechCrunch</a>: Duffy attributes the grounding to a routine equipment upgrade.</li>
<li><a href="https://thehill.com/policy/transportation/6220399-duffy-billions-air-traffic-ground-stop/">The Hill</a>: the request for billions more in air traffic funding after the ground stop.</li>
<li><a href="https://www.bloomberg.com/news/articles/2026-11-14/duffy-calls-for-emergency-air-traffic-upgrade-funds-after-outage">Bloomberg</a>: the call for emergency upgrade funds and the open questions about the cause.</li>
<li><a href="https://www.foxbusiness.com/fox-news-air-space/who-l3harris-federal-contractor-sec-duffy-says-may-have-caused-faa-outage">Fox Business: who is L3Harris</a>: profile of the contractor named at the briefing.</li>
<li><a href="https://www.foxbusiness.com/economy/northeast-airport-operations-resume-telecom-outage-disrupts-flights">Fox Business: Northeast telecom outage</a>: report on the 21 September 2026 outage.</li>
</ul>
<h3>FAA network: FTI, FENS and modernization</h3>
<ul>
<li><a href="https://www.l3harris.com/all-capabilities/faa-telecommunications-infrastructure-fti">L3Harris: FAA Telecommunications Infrastructure (FTI)</a>: contractor overview with network scale and the current copper to fiber upgrade status.</li>
<li><a href="https://www.l3harris.com/sites/default/files/2025-03/fti-program-profile-sell-sheet-l3harris-sas-mn.pdf">L3Harris: FTI program profile (PDF)</a>: architecture summary covering IP and legacy TDM services, the optical backbone and the dual metro rings at the En Route centers.</li>
<li><a href="https://www.faa.gov/about/office_org/headquarters_offices/ang/offices/tc/library/storyboard/detailedwebpages/fens.html">FAA: FENS program description</a>: FAA Enterprise Network Services, the planned successor to FTI.</li>
<li><a href="https://www.faa.gov/testimony/testimony-bryan-bedford-faa-administrator-hearing-committee-appropriations-subcommittee">FAA Administrator testimony on the fiscal year 2027 budget</a>: frames FENS as the replacement for a 1980s era network architecture.</li>
<li><a href="https://www.faa.gov/media/80091">FAA Capital Investment Plan, fiscal years 2024 to 2028</a>: FTI service and site counts and the sustainment program that bridges to FENS.</li>
<li><a href="https://www.faa.gov/air_traffic/publications/cip/files/FY20-24/FY20-24_CIP_Overview.pdf">FAA Capital Investment Plan overview, fiscal years 2020 to 2024 (PDF)</a>: notes that most FTI services were still TDM based, and why.</li>
<li><a href="https://www.telecomreviewamericas.com/articles/telecom-operators/att-to-drive-faas-next-generation-communications-backbone/">Telecom Review: AT&amp;T and the FAA's next network</a>: the FENS award and the history of FTI since 2002.</li>
<li><a href="https://www.gao.gov/products/gao-24-107001">GAO: FAA Actions Are Urgently Needed to Modernize Aging Systems (September 2024)</a>: audit finding that most of the FAA's 138 air traffic control systems are unsustainable or potentially unsustainable.</li>
<li><a href="https://www.gao.gov/products/gao-26-107992">GAO: Ambitious New Modernization Effort Needs to Improve Cost and Schedule Planning (2026)</a>: progress and cost risks of the new air traffic control system program, with copper to fiber counts.</li>
<li><a href="https://www.oig.dot.gov/library-item/29670">DOT Inspector General: the 2009 FTI outage</a>: review of the 19 November 2009 outage, traced to contractor configuration and procedural errors.</li>
</ul>
<h3>Air traffic automation systems</h3>
<ul>
<li><a href="https://www.faa.gov/speeches/en-route-automation-modernization">FAA: En Route Automation Modernization (speech)</a>: FAA remarks describing the ERAM system.</li>
<li><a href="https://storyboard.faa.gov/detailedwebpages/eram.html">FAA: ERAM program description</a>: program page for the En Route automation platform.</li>
<li><a href="https://www.faa.gov/air_traffic/technology/tamr">FAA: Standard Terminal Automation Replacement System (STARS)</a>: the terminal automation platform at TRACONs and their towers.</li>
<li><a href="https://avweb.com/aviation-news/faa-expands-ai-assisted-traffic-management/">AVweb: FAA expands AI assisted traffic management</a>: coverage of the SMART planning tool.</li>
</ul>
<h3>Protocol standards (IETF)</h3>
<ul>
<li><a href="https://www.rfc-editor.org/rfc/rfc4271.html">RFC 4271: A Border Gateway Protocol 4 (BGP 4)</a>: session state machine, OPEN message checks, hold and connect retry timers.</li>
<li><a href="https://www.rfc-editor.org/rfc/rfc5492.html">RFC 5492: Capabilities Advertisement with BGP 4</a>: how peers negotiate capabilities and reject unsupported ones.</li>
<li><a href="https://datatracker.ietf.org/doc/html/rfc2385">RFC 2385: Protection of BGP Sessions via the TCP MD5 Signature Option</a>: segments that fail the check are dropped with no reply to the sender.</li>
<li><a href="https://www.rfc-editor.org/rfc/rfc2439.html">RFC 2439: BGP Route Flap Damping</a>: the damping mechanism for unstable routes.</li>
<li><a href="https://datatracker.ietf.org/doc/html/rfc5880">RFC 5880: Bidirectional Forwarding Detection (BFD)</a>: timer negotiation and detection time.</li>
<li><a href="https://www.rfc-editor.org/rfc/rfc6192.html">RFC 6192: Protecting the Router Control Plane</a>: filtering and rate limiting traffic sent to the route processor.</li>
<li><a href="https://www.rfc-editor.org/rfc/rfc1661.html">RFC 1661: The Point to Point Protocol (PPP)</a>: link establishment on serial circuits.</li>
<li><a href="https://www.rfc-editor.org/rfc/rfc1994.html">RFC 1994: PPP Challenge Handshake Authentication Protocol (CHAP)</a>: authentication on PPP links.</li>
<li><a href="https://datatracker.ietf.org/doc/html/rfc2784">RFC 2784: Generic Routing Encapsulation (GRE)</a>: the tunnel encapsulation behind the recursive routing case.</li>
<li><a href="https://www.rfc-editor.org/rfc/rfc4553.html">RFC 4553: Structure Agnostic Time Division Multiplexing over Packet (SAToP)</a>: carrying unframed T1 and E1 circuits over packet networks.</li>
<li><a href="https://rfc-editor.org/rfc/rfc5086">RFC 5086: Structure Aware TDM Circuit Emulation Service over Packet Switched Network (CESoPSN)</a>: carrying framed TDM circuits over packet networks.</li>
<li><a href="https://datatracker.ietf.org/doc/html/rfc9568">RFC 9568: Virtual Router Redundancy Protocol (VRRP) Version 3</a>: gateway redundancy and election of a single active router.</li>
<li><a href="https://www.rfc-editor.org/rfc/rfc2328.html">RFC 2328: OSPF Version 2</a>: router identifiers and adjacency formation.</li>
</ul>
<h3>Vendor documentation</h3>
<ul>
<li><a href="https://www.juniper.net/documentation/us/en/software/junos/cli-reference/topics/ref/statement/bfd-liveness-detection-edit-protocols-bgp.html">Juniper: BFD liveness detection statement for BGP</a>: minimum interval, multiplier and adaptation options.</li>
<li><a href="https://www.juniper.net/documentation/en_US/junos12.3/topics/topic-map/bgp-bfd.html">Juniper: configuring BFD for BGP</a>: worked example, with the vendor's caution that very short intervals can make BFD flap.</li>
<li><a href="https://www.juniper.net/documentation/us/en/software/junos/sample-bgp-user-guide/cli-reference/topics/ref/statement/hold-time-edit-protocols-bgp.html">Juniper: BGP hold time statement</a>: permitted range and the 90 second default.</li>
<li><a href="https://juniper.net/documentation/en_US/junos/topics/topic-map/bgp-session-flaps.html">Juniper: BGP session and route flaps</a>: damping parameters and how to enable damping.</li>
<li><a href="https://www.cisco.com/c/en/us/support/docs/ip/enhanced-interior-gateway-routing-protocol-eigrp/22327-gre-flap.html">Cisco: recursive routing error on GRE tunnels</a>: why a tunnel that routes to its own endpoint flaps along with its routing neighbors.</li>
<li><a href="https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_plcshp/configuration/15-mt/qos-plcshp-15-mt-book/qos-plcshp-ctrl-pln-plc.html">Cisco: Control Plane Policing</a>: rate limiting traffic aimed at the router CPU.</li>
<li><a href="https://www.cisco.com/c/en/us/support/docs/wan/t1-e1-t3-e3/10206-T1-layer1.html">Cisco: T1 Layer 1 Troubleshooting</a>: framing, line coding and clock source checks.</li>
<li><a href="https://www.cisco.com/c/en/us/support/docs/wan/t1-e1-t3-e3/14149-chapter15.html">Cisco: Troubleshooting Serial Line Problems</a>: detecting and isolating clocking problems on serial WAN links.</li>
<li><a href="https://www.cisco.com/c/en/us/support/docs/csa/cisco-sa-20020523-cbos-dos.html">Cisco security advisory on CBOS</a>: documents the Cisco 600 family as DSL customer premises equipment.</li>
</ul>
<hr />
]]></content:encoded></item></channel></rss>