US halts flights at busy East Coast airports, says fiber line cut
Posted by allanbreyes 1 day ago
Comments
Comment by chasd00 1 day ago
Comment by john_strinlai 1 day ago
>A man stranded in the bush in northern Saskatchewan was rescued last week after chopping down four power poles — knocking out electricity to surrounding communities. [...]
>But he had an axe and he knew SaskPower would have to check the downed line, so he went to work.
Comment by foxyv 3 hours ago
Comment by baud147258 7 hours ago
Comment by castillar76 21 hours ago
Comment by Onavo 1 day ago
Comment by cube00 1 day ago
Pretty grim that a life critical system wasn't designed to report that the backup fibre was unserviceable until they attempted to switch over to it.
I wonder how long it was down? Days, weeks, months?
Comment by newhotelowner 1 day ago
I am paying, $1000, $1800 & $1900 for the same service at 3 different location (20 mins from each other).
Two locations, I also have old coax lines that are still active, but not paying for it.
When I bought two businesses, I learned that they were paying for a dedicated fiber but using coax service.
At one of the location, we had fiber, and paying for backup coax and wireless. But if you turn off fiber box, it wont fail over to either one.
I wouldn't surprise it was down for weeks and no one bothered about it.
Comment by Neywiny 23 hours ago
Comment by tuvix 21 hours ago
Comment by Neywiny 7 hours ago
I thought about what I believe is your way until https://www.tomshardware.com/opinion/t-mobile-home-internet-... and https://www.tomshardware.com/news/t-mobile-misleads-home-int... .
Comment by tuvix 4 hours ago
Not my area of expertise though so I could be leaving out info
Edit: Starry is the company, I believe they were just purchased by Verizon
Comment by Neywiny 2 hours ago
Comment by onetokeoverthe 1 day ago
Comment by toast0 1 day ago
Comment by to11mtm 1 day ago
For a major trans-oceanic backbone provider, at least 15ish years ago they had a mile or two between Detroit and Chicago where both ends were on the same side of the interstate highway.
But it's more frequent on DAS (Distributed Antenna Systems, AKA small-cell or micro-cell) networks.
Also the challenge of when fibers are leased (if that's still a thing, based on the networks I helped design I'd say 'probably').
They really are analogous to Lamport's "Distributed System" quip; A damaged fiber owned by a company you have never heard of can wreck your day.
Comment by hdgvhicv 22 hours ago
Fibres on the Texas link die all the time (yearly) but that’s fine.
My circuits in the far east are often carried one via dues and one via the USA.
It is possible to guarentee diversity, you just need to get specific detail and have good contractual protections.
Comment by Havoc 1 day ago
For mission critical stuff like airports I would like to think they're go for a more rigorous methodology than hope for the best on paths
Comment by happyPersonR 1 day ago
Usually it’s a combo of a tier 1 that decides that a resell agreement is “good enough” and “well just eat the loss” and then by the time stuff like this rolls around “we’ll get back to you” + a bunch of silly explanations that boil down to “you’re not gonna sue us though” start coming out lol
Comment by iso1631 15 hours ago
Then there's another faction that doesn't trust a word suppliers say, who will insist on two different suppliers, as while in theory they could be sharing the same routes, it's far less likely.
Comes down to "do you trust contracts".
Of course this is for third tier sites like branch offices. Major sites require far more oversight, including visibility of cable routes and any hops on the internal network
Comment by koolba 1 day ago
Comment by oasisbob 1 day ago
Comment by hdgvhicv 22 hours ago
Had a few issues in some lamdlpcked counties on occasion - Kabul had all traffic routing via the same route into Pakistan for a few years, Kathmandu had an earthquake knock out both ISPs within 30 seconds of each other.
Backups via starlink global roaming work well now in particularly challenging environments.
Comment by jacquesm 1 day ago
Comment by konfusinomicon 1 day ago
Comment by burningChrome 1 day ago
Luckily we were on Century Link so weren't affected by his stupidity. lol
Comment by hdgvhicv 22 hours ago
Comment by lazide 17 hours ago
Comment by jonah 1 day ago
Comment by unethical_ban 1 day ago
Over ten years ago, my employer was spinning up a new DC across the state and had three links between it and the primary DC. Two were pretty direct, but we needed a third because at one point in the 300 mile path, the two main links went within 400 meters of each other.
So how national-security-adjacent critical systems like an airport system doesn't have a larger set of backup links, and validates that they are geographically separate up until the connections, and have realtime alerting on the connection status, is surprising to me.
Comment by Jtsummers 1 day ago
This isn't relevant in this case. The problems occurred at different places. The backup fiber was, separately from the issues with the primary system, cut by a construction crew. It's not two fiber lines both cut at the same spot.
Comment by hdgvhicv 22 hours ago
You can get two separate failures in short succession, where the first one hasn’t been fixed. For critical devices that’s why you have tertiary backups, ideally on different technology compel you. For one major event I had diverse fibre and satelite and microwave. Lost microwave and satelite to electronic warfare but the fibres were ok. Ofcom were really pissed about it, but nothing they could do.
Comment by justsomehnguy 19 hours ago
No?
Like, okay, monitoring reports what there is no link.
Now what? Somebody gets a hi vis vest, a hard hat and goes along the cable to investigate the reason?
There is nothing monitoring could help, especially if the cable was cut recent enough.
Comment by benzible 19 hours ago
> An RFTS enables fiber to be automatically tested from a central location. A central computer is used to control the operation of OTDR-like test components located at key points in the fiber network. The test components scan the fiber to locate problems. If a problem is found, its location is noted and the appropriate personnel are notified to begin the repair process.
Comment by iso1631 15 hours ago
If it's down for more than 15 minutes in country or 2 hours internationally it gets flagged for manual attention. Shorter outages are logged but only looked at monthly as part of the operational report process.
Now sure you can have two independent faults on the same day, but the reports are that they didn't know the backup line was down.
Comment by bonestamp2 22 hours ago
Comment by unethical_ban 7 hours ago
Comment by mikeweiss 1 day ago
Comment by krashidov 1 day ago
Comment by kqgnkqgn 1 day ago
Comment by vkou 1 day ago
You're going to be hard-pressed to find any point in the American empire's life when it doesn't have 'current tensions' with someone or other.
Comment by wiml 1 day ago
Comment by rylando 1 day ago
Comment by imglorp 1 day ago
Recently, I tried calling 811 before digging in my yard. The webpage was broken and the hotline kept me on hold forever. I gave up. Small wonder.
https://blackhydrovac.com/underground-utility-strikes-learn-...
Comment by ChrisMarshallNY 21 hours ago
Comment by expedition32 15 hours ago
Comment by imglorp 7 hours ago
Comment by lazide 9 hours ago
Plenty of people don’t bother and YOLO it, and usually it’s fine - until it isn’t.
Comment by jasonjayr 23 hours ago
The punchline is that it's about 2000 ft from the local CO, where (I believe) half the town's lines terminate.
Comment by pvaldes 1 day ago
Similar events happened in Europe disguised as thieves stealing fiber optic. This does not have any sense economically, as the value in the market is zero so... either is an honest accident and is cleared in a few days, or is sabotage
Comment by MC995 1 day ago
Do you honestly think that crackheads think that far in advance?
I've seen fiberoptic cables stolen from 2 (city) jobsites in the last 5 years, once by tweakers later caught trying to sell them as scrap copper and the second thief was never caught. This happened even though the spools had big signs on them saying "Fiber Optic Cable - NO COPPER".
Comment by pvaldes 13 hours ago
Crackheads can be manipulated easily, or bribed. Maybe the value was in that they were paid with $50 in drugs for doing that.
People too dumb to plan X and to inconsistent to spend several days honing the plan, somehow avoided all the traps and locks, go for the precise location and do X... either they worked in the place and are very familiar with each room, or there is a director manipulating them.
With the added value that addicts can be overdosed easily if they try to strike back and treat to getting too talkative. Nobody would bat an eye.
Comment by rcostin2k2 1 day ago
Comment by 12876qj 1 day ago
Comment by someothherguyy 1 day ago
Comment by zoomablemind 1 day ago
Comment by pixl97 1 day ago
I worked for a regional ISP that had a major outage when the redundant fiber provided by the telephone company was cut in one place triggering a full loss of connectivity. It also caused a massive 911 outage for 200,000 people as it isolated the 911 center from the city core.
Turns out the phone company didn't connect one side of the ring topology even though they certified they did. Needless to say lawsuits abounded.
Comment by lovich 1 day ago
The amount of shit I was given everytime I went through a checklist working at a fedramp certified company working with emergency alerts on the phone system made me prematurely gray.
I could feel the barely contained seething rage everytime I told the execs that the reason this release will take 3 days and not be instantaneous like your friends releases at a faang and that information made you embarrassed at your dinner party, is because you agreed to this process contractually years ago and now it’s a crime if I just sign off on it being ok without actually checking that it’s ok.
Comment by pixl97 1 day ago
Comment by readthenotes1 1 day ago
A lot of people are incompetent.
Comment by andrewjf 1 day ago
Comment by bell-cot 1 day ago
Comment by eichin 23 hours ago
Comment by thewebguyd 1 day ago
Comment by bonoboTP 1 day ago
Comment by lovich 1 day ago
I worked at a company worth a few billion and the leadership balked when they told engineering they wanted a near instantaneous failover system and our department informed them that would require paying for a second environment that could be rolled over to.
It is rare to find leaders who can accept the cost of redundant infrastructure that is there for emergency backup.
What puzzles me is why they can’t accept it when they are perfectly fine with insurance costs and I can’t see much of a difference between the two when looking at a spreadsheet of costs other than possibly tax differences between the type of expenditure.
Comment by bonoboTP 23 hours ago
Redundancy has costs and those costs can be spent elsewhere like having higher quality or bigger disks, or a faster network switch or better CPUs etc.
In theory, it would also be better to own two cars instead of one, because what if the first one gets in an accident or just breaks down. Yet, not everyone can afford that. Should you just buy two half-as-expensive cars than what you can buy one of, so you can say you have a "backup"? Likely the two half-price ones would be so much crappier that the one good car would cause you less trouble in expectation than driving a shitty one and then having another shitty spare one, both of which will constantly have issues.
Comment by 486sx33 11 hours ago
Comment by jimt1234 1 day ago
There's generally two wrong responses: (1) We spent a lot of money on 'blah blah blah', a lot of other companies use it, so yeah, we've got a backup/failure system. And, (2) inadequate testing - either, we tested 1 of 50 services, and it worked, so the whole system can be restored; or, we gracefully tested, and it worked, so it will obviously work during not-graceful incidents.
And the root cause of this is generally that no one gets promoted for implementing an adequate backup/failure system, or it's extremely rare.
Comment by ranger_danger 1 day ago
Not saying that's what is going on here, just that it's possible.
Comment by aaron695 1 day ago
Comment by kqgnkqgn 1 day ago
Sometimes the level of incompetence / lack of care in organizations like this astounds me. I understand issues like this can be complicated and systemic but it honestly makes me think very poorly of the technologists building these systems in government.
Comment by woodruffw 1 day ago
Nothing I've ever seen or experienced with FAA indicates a lack of care; the parsimonious explanation is almost always that people who care a great deal are working within complex systems that don't always have a consistent or externally legible set of priorities. Or more intuitively, the failures we see are the "acceptable" ones versus the unacceptable ones (like planes falling out of the sky).
Comment by jcrawfordor 21 hours ago
Comment by markdown 1 day ago
Hard to fathom someone still holding that view after the 737MAX disasters.
The NTSB has repeatedly and publicly criticized the FAA for failing to implement their recommendations, including as recently as last week (Amazon Prime crash resulting in 5 deaths). They've explicitly called out the FAA's failure to act as a contributing factor in crashes, multiple times.
Regulatory capture ruined the FAA.
Comment by OutOfHere 1 day ago
Comment by woodruffw 1 day ago
Comment by rasz 11 hours ago
Comment by RyJones 1 day ago
BT guy ends that part of the presentation with "the IRA will really have to get shit together to take you off the net"
This was during the troubles. Same planet, different world.
Comment by dgellow 1 day ago
Comment by kqgnkqgn 1 day ago
Comment by sc68cal 1 day ago
https://www.airwaysmag.com/new-post/faa-smart-first-deployme...
Comment by cmiles8 1 day ago
Comment by jcranmer 1 day ago
The fundamental problem with DCA, and one of the two main causes of the crash [1], is that they are required by political pressure to operate at a higher operational tempo than they can safely operate at. DCA has essentially 1½ usable runaways--large planes can only use the larger runway, and given the capacity restrictions, airlines have been pushing to use fewer small planes at the airport. This shift in plane size means the effective safe slot capacity has gone down, but people still keep citing the same number as justification for safe numbers, and the politicization of the issue has shut down everyone who complained that the actual traffic just couldn't be safely handled.
If DCA's theoretical slot capacity (36 landings and takeoffs each per hour) is to be reached one just one runway, you have about 100 seconds to go from plane 1 touchdown; exit the runway, letting plane 2 on to take off; plane 2's wheel leaving the runway, clearing plane 3 touchdown. That's doable, but with essentially 0 margin for error. Alternating the runways used for each landing would give you closer to 30s of margin, but if only 10-20% of the planes can use the alternate runway, you can't divert enough planes to use the alternate runway.
Now DCA and the FAA aren't stupid enough to actually schedule 36 landing slots every hour, but the problem is that because of a few various factors, what was scheduled as, say, 30 slots for an hour (giving 120 seconds between landings, probably sufficient margin) ends up being 12 slots used in the first half-hour and 18 slots used in the second half-hour, which means a lot of the actual operation ends up having no safety margin even though on paper you have sufficient margin.
One of the ways you can rectify that is to assign slots further in advance so that you don't get the bunching. Effectively saying "oh, if you leave right now, you'll arrive at 3:30 with three other planes, but if I hold you for 10 minutes, I can push you into a less busy arrival time." Doing this requires good, accurate prediction of the actual flight travel times, and my understanding is that this is what the new software is meant to provide.
[1] The other main cause is essentially that the DoD's aviation practices in the area is a giant clusterfuck that endangers lives, and unfortunately that sentence is not relegated to the past tense.
Comment by kj4211cash 21 hours ago
Comment by whatever1 1 day ago
Good luck to all of us.
Comment by gkedzierski 1 day ago
Comment by dzhiurgis 14 hours ago
Comment by tencentshill 1 day ago
Comment by macintux 1 day ago
Comment by lovich 1 day ago
The government politicians who approved this are frequent flyers out of there, so they’re at least getting their lives out on the line before the rest of the country if it proves ill managed or conceived.
It’s basically dogfooding.
Comment by atonse 1 day ago
Or is it that these ATC networks are their own air-gapped network with less redundancy? That just doesn't add up. Or maybe there was only one line going to the ATC, with no multiple "ISPs" like a datacenter would have?
Comment by bri3d 1 day ago
* There is supposed to be a primary and a secondary link, in this case the primary failed and the fail-over also failed. It's unclear from the reporting if they were damaged in the same incident or if the failover was not tested or monitored adequately.
The Philadelphia TRACON site has been notoriously unreliable and was supposedly improved in 2025, it's also unclear if these issue actually could stem from that implementation.
Comment by cmiles8 1 day ago
Comment by wrs 1 day ago
Comment by Polizeiposaune 1 day ago
First big oops of this form that I remember:
"In December 1986, the ARPANET had 7 dedicated trunk lines between NY and Boston, except that they all went through the same conduit -- which was accidentally cut by a backhoe. "
Comment by wrs 1 day ago
One example was a site that had fiber and coax, from different companies. They might have shared a pipe at some point along their length, hard to say. But both connections went down at the same time for digital reasons, not physical. The providers had simultaneous unrelated backend router problems and the site lost contact to both gateways.
This was with a nice SD-WAN system that routed everything dynamically across both pipes to address latency and errors, but if the packets get dropped at the first hop in both networks, there's not much you can do!
The saving grace in that case was the third redundant connection, a 4G cell modem, with a lot less bandwidth but able to keep the critical transactions going. These days I'd certainly want Starlink on the roof as well.
Comment by FireBeyond 1 day ago
Comment by pixl97 1 day ago
In any system where both sides of redundancy always carry traffic then loss of one link can cause congestion failure if any link fails. This is a very common means of failure in electrical networks that requires load shedding. Well, you can't load shed air traffic.
A more complex system that I like, but comes with it's own set of constraints and implementation issues is a system where both lines carry all the traffic at all times. This way the default state of the system is always working and your first failure isn't invisibly critical.
But this is very hard as we see in TCP when things get out of order and high latency creeps in. You have to manage a lot more state at the data level.
Comment by readthenotes1 1 day ago
That seems like an extremely foolhardy thing to do.
Comment by ssl-3 1 day ago
It seems like it would present a less-chaotic solution than that provided by having no data communications at all.
Comment by kqgnkqgn 1 day ago
Comment by bell-cot 1 day ago
Comment by topspin 1 day ago
Comment by bell-cot 22 hours ago
Comment by topspin 20 hours ago
I'm rather certain the FAA does indeed either employ or contract people that are capable of providing the solution. I'm also quite certain those people live in a perpetual blizzard of responsibility diffusion bullshit: the FAA is a venerable and highly risk-adverse operation. If someone told me it takes months of meetings and sign-offs to so much as log into a router and perform a benign check of some setting, I wouldn't blink.
Comment by bell-cot 16 hours ago
In this case, that outsourcing could easily look like "call local ISP's, get connections, set up tunnel".
Comment by Hikikomori 1 day ago
Comment by cmiles8 1 day ago
Comment by librasteve 1 day ago
one day the backbone (E1 bidirectional ring iirc) was broken north of london … my boss drove out to find that travellers had entered the field, opened the inspection cover in the ground, stood a pole for their clothes line and filled with cement
very savvy, he didn’t let on that we had about £1m revenue per day close to collapse and convinced them to hang their clothes somewhere else (perhaps a few quid changed hands)
Comment by tomasphan 1 day ago
Comment by Hikikomori 1 day ago
Comment by LastTrain 1 day ago
Comment by AnimalMuppet 1 day ago
Comment by adamm255 1 day ago
Comment by quickthrowman 1 day ago
(The actual easiest method is to install the conduit or direct burial cable with tracer wire)
Comment by bluGill 1 day ago
Before fiber a lot of things were put into the ground with no thought how you would find it again.
Comment by quickthrowman 1 day ago
I sometimes hire directional boring and excavation contractors and if there’s any doubt as to where an electrical conduit or natural gas pipe is, I opt for the hydrovac truck to minimize risk.
Comment by wat10000 1 day ago
Comment by dboreham 1 day ago
And of course there are marker posts that say "buried fiber optic cable do not dig".
Comment by pixl97 1 day ago
Most cheap hired labor doesn't realize you shouldn't dig up the market posts and throw them in the dumpster on construction sites. Add a little bit of rain and suddenly the contractor showing up with an excavator that was told everything was properly marked, and can see markings in other places but not where they are digging, and disaster occurs.
Also soils aren't static, I've been on sites where the midpoint of a buried telecommunications cable had drifted over 8 feet from where the markers were a few hundred yards apart. Just looking at the markers and assuming a straight line isn't a safe bet. Looking at the fence rows to the left and right of the cable and you could see a matching bow in the fence rows.
Comment by throwaway173738 1 day ago
Comment by pixl97 1 day ago
Comment by ssl-3 16 hours ago
Reps from the local telco showed up to one of the meetings to tell everyone that they didn't know where their copper lines were buried and that they were unsuccessful at locating them outside at all. They knew where both ends were but the middle was a mystery to them.
This statement made the architect's face contort into strange shapes. I joked that they call 811; some people thought that was rather hilarious, others much less so.
Anyway, the building wasn't in service yet and the telco said they'd fix it if we managed to naff them up, so away we dug. No telco cabling was damaged in this process. (We did dig up some plastic drainage tile but that was a simple farm store fix.)
Comment by wnevets 1 day ago
Press X for doubt.
[1] https://www.npr.org/2026/09/21/nx-s1-5976816/faa-ai-manage-a...
Comment by jsrozner 21 hours ago
Comment by dctoedt 1 day ago
[QUOTE:]
Russia has been staging covert operations and deniable provocations against European countries for many years. Since its full-scale invasion of Ukraine began in 2022 they have increased. These have included incursions by reconnaissance drones, hacking, arson, parcel bombs targeting cargo flights and plots to kill executives of leading European arms firms. In recent months the pace of such shadowy attacks appears to have risen again. The Russian operations have three main goals, according to analysts and security officials: trying to coerce European countries into halting their support for Ukraine; raising the costs of providing that support; and attempting to make NATO look powerless.
[/QUOTE]
And of course there've been "suspicious" cable cuts in the Baltic Sea and other European waters ....
https://www.economist.com/europe/2026/09/21/russias-grey-zon...
Comment by jacquesm 1 day ago
Comment by hdgvhicv 22 hours ago
Salami tactics.
Comment by simulator5g 18 hours ago
Comment by jacquesm 16 hours ago
Comment by aaron695 1 day ago
Comment by Johnny555 1 day ago
https://www.cnn.com/2025/02/25/business/musk-faa-starlink-co...
Comment by 0xbadcafebee 21 hours ago
Comment by platevoltage 1 day ago
Comment by cyanydeez 1 day ago
right. of course, no ones going to cut corners on the safe use of AI and secure infrastructure in this administration.
Comment by platevoltage 1 day ago
Comment by basisword 1 day ago
Comment by bell-cot 1 day ago
With low-enough engineering quality, you don't actually need saboteurs.
Comment by smcin 23 hours ago
Comment by basisword 1 day ago
While I agree, that’s just what they told us. Not sure I believe it anymore. The fact the first UK one happened immediately after they sanctioned Israel is also just a big coincidence.
Comment by copritic 19 hours ago
Not sure what they would gain from a one-off shutdown of UK airports for a few hours (which included flights to and from Israel being cancelled).
Israel has actually openly retaliated to the UK measures with far more sensible and long-lasting strategic measures, including shutting down the UK consulate for Palestine in East Jerusalem.
Comment by cynicalpeace 1 day ago
Just sayin
Comment by kittikitti 1 day ago
In places like these, it's not the best contractor that wins, but the one that supports genocide against Palestinians.
Comment by ofcrpls 1 day ago
Comment by MadrasThorn 1 day ago
Comment by dboreham 1 day ago
Comment by baggachipz 1 day ago
Comment by ye4yey 1 day ago
Comment by chresko 1 day ago