Worldwide IT Outage (Crowdstrike update)

[MarkWi72]

Apparently happened recently and was on a newsflash from the BBC. It’s affected driver diagrams for Thameslink, Gatwick Express and Northern Trains. Flights are grounded in Sydney and Berlin.

 

[Mcr Warrior]

More here...

Hospitals, banks, media, airlines hit by major IT outage

Cyber-security firm Crowdstrike said its update had caused the problem on computers running Windows.

www.bbc.com

Extract...

Mass IT outage affects airlines, media and banks

A raft of global institutions - including major banks, media outlets and airlines - have reported being hit by a mass IT outage.

The US state of Alaska has warned its emergency services are affected, while there are also reports of the London Stock Exchange being impacted.

Govia Thameslink Railway also advising of issues, as @MarkWi72 has hinted at...

x.com

x.com

Extract...

We are currently experiencing widespread IT issues across our entire network. Our IT teams are actively investigating to determine the root cause of the problem.

We are unable to access driver diagrams at certain locations, leading to potential short-notice cancellations, particularly on the Thameslink and Great Northern networks.

 

[signed]

Crowdstrike, a leading Antivirus provider, shipped a borked update to their agents that leads to immediate crashing of the clients

https://supportportal.crowdstrike.c...s-crashes-related-to-Falcon-Sensor-2024-07-19 (login required)

Microsoft cloud arm Azure is also suffering from massive outages due to that Crowdstrike events.

https://www.wsj.com/tech/microsoft-reports-major-service-outage-affecting-users-worldwide-328a2f40

This sounds like why there is pure madness

https://www.reuters.com/technology/frontier-says-operations-impacted-by-microsoft-outage-2024-07-19/

---

The US airlines are asking for a US-wide global grounding of flights.

IT outages reported across globe as airlines, airports, banks and media companies experience disruptions

IT outages have been reported across the globe as airlines, airports, banks and media companies have suddenly experienced ongoing disruptions.

abcnews.go.com

"We’re aware of a technical issue with CrowdStrike that is impacting multiple carriers. American is working with CrowdStrike to resolve the issue as quickly as possible and apologize to our customers for the inconvenience," American Airlines said in a statement obtained by ABC News.

American Airlines, United and Delta have asked the FAA for global ground stop on all flights, according to an alert from the FAA on Friday morning.

Meanwhile, flights in the air will stay in the air, but no American, United or Delta flights will take off.

 

[dan4291]

Typically this would happen on potentially one of the busiest days of the year for the travel industry, with it being the start of the school summer holidays for most!

 

[Beebman]

The online services of Belgian national rail operator SNCB have been affected according to this article from RTBF TV - Google translation:

SNCB is experiencing various technical problems on Friday morning, notably preventing passengers from buying tickets online. Data on train delays and track changes may also not be updated. The cause is a global problem at Microsoft, the Belgian railway company said.

" Since this morning, we have been faced with a general IT problem. However, this has no impact on train traffic ," said spokesperson Bart Crols. " Our services are doing everything in their power to resolve the problem as quickly as possible ." Due to this technical problem, information on the website and the application, such as data on delays, track changes and carriage occupancy, may not be up to date, SNCB added.

Passengers are also reporting that they are no longer able to order tickets online. " We advise passengers to rely on the screens and announcements in stations for the most up-to-date information on journeys ," Crols recommended. The cause of the problem is a Microsoft outage, leaving many companies around the world facing technical problems. Many users report seeing a blue screen with an error message.

Meanwhile, tech giant Microsoft said it was taking " mitigation measures " after an outage affected several businesses, including in Australia.

“ Our services are being improved as we continue to take mitigation measures ,” the company wrote on the social network X (formerly Twitter).

 

[dk1]

Typically this would happen on potentially.one of the busiest days of the year for the travel industry, with it being the start of the school summer holidays for most!

Sod’s Law that.

 

[OuterDistant]

Feeling pretty lucky typing this on MacOS!

 

[Rail Quest]

Crowdstrike, a leading Antivirus provider, shipped a borked update to their agents that leads to immediate crashing of the clients.

I'm baffled by the irony of a large cyber security firm causing one of the most wide spread security incidents that I've ever heard of.

 

[signed]

That will probably make the banks go down soon I expect

 

[OscarH]

Feeling pretty lucky typing this on MacOS!

Bet the companies don't use that brand of corporate spyware are feeling smug!

 

[uglymonkey]

There was massive problems on the ECML out of KX last night, all trains north cancelled at one point, wonder if that had anything to do with it? "Signalling at Welyn" was said to be the cause. Took me 4 hours to get home via Liverpool Street

 

[Adam Williams]

x.com

x.com

Will be due to the botched Crowdstrike update last night that's taken out a metric ton of machines and left them boot-looping.

Merseyrail @merseyrail
⚠️Due to a Microsoft outage, we are currently experiencing a technical issue which is affecting our passenger information boards and our ability to print 3rd party tickets at stations.
❗️Some delays across the network might not be displayed. We are working hard to fix this issue

It's a great shame they don't have any Ticket Machines, isn't it. The S&B machines I saw this morning at other stations were all still working!

Shame they don't accept E-Tickets either...

I hope affected passengers will be permitted to travel.

 

[DanNCL]

There was massive problems on the ECML out of KX last night, all trains north cancelled at one point, wonder if that had anything to do with it? "Signalling at Welyn" was said to be the cause. Took me 4 hours to get home via Liverpool Street

Unrelated.

 

[uglymonkey]

Thanks

 

[kristiang85]

All SWR TVMs down this morning, but very efficient service from the staff at Basingstoke.

It s a good job it happened on a sunny Friday morning when commuting loads are likely reduced.

 

[Flying Snail]

There was massive problems on the ECML out of KX last night, all trains north cancelled at one point, wonder if that had anything to do with it? "Signalling at Welyn" was said to be the cause. Took me 4 hours to get home via Liverpool Street

I would hope that signalling systems are very strongly firewalled against general internet traffic if not entirely airgapped.

 

[Bletchleyite]

If anyone is experiencing this themselves, there is apparently a workaround - boot into safe mode, find the Crowdstrike installation folder and rename it then reboot. This will prevent Crowdstrike loading, which at the expense of slight added vulnerability to attack (Windows Defender will kick in instead so you won't have nothing at all) will get your PC working again.

I suspect everyone will need to do that - they won't be able to push an update to fix it if the PC isn't online.

If you don't use Crowdstrike anti-virus, you won't have the problem.

 

[signed]

It looks like the worst IT in the railway space (SNCF) is not impacted (for once).

Airports are going down like Kapla falling though :

• Amsterdam and all the other Dutch airports
  • https://www.lbcgroup.tv/news/world-...iphol-airport-says-hit-by-global-it-outage/en

Amsterdam's Airport Schiphol, one of the busiest airports in Europe, said on Friday it was affected by a global IT outage, which also affected Eindhoven airport, the Transavia airline, and Dutch hospitals.

 

[SWT_USER]

How long before the first thread in the disputes and prosecutions forum where some poor soul is being taken to court by the railway because they were unable to collect their tickets as a result of this?

 

[m0ffy]

There’s also a major Microsoft Azure issue going on.

 

[signed]

How long before the first thread in the disputes and prosecutions forum where some poor soul is being taken to court by the railway because they were unable to collect their tickets as a result of this?

I'd say that will not happen, especially if they are on notice (which they clearly are). Except of course if you're talking about Merseyrail or TfL

 

[ainsworth74]

It looks like the worst IT in the railway space (SNCF) is not impacted (for once).

Surely that's becuase SNCF are still using Windows XP?

 

[westv]

There’s also a major Microsoft Azure issue going on.

The MS issue is what has caused it all.

 

[m0ffy]

The MS issue is what has caused it all.

I’d know that if I read the second/third post!

 

[Howardh]

How long before the first thread in the disputes and prosecutions forum where some poor soul is being taken to court by the railway because they were unable to collect their tickets as a result of this?

According to Simon Calder on the news this morning, TPE (maybe others) are "allowing you to buy your tickets on the train" but that doesn't help those already bought who can't access them? Unless they buy fresh and get refunded on the original?

 

[signed]

Surely that's becuase SNCF are still using Windows XP?

That wouldn't sound that far off. Friends that work at SNCF told be some things about their IT (that I can't share) that is scary how bad it is behind the scene

It usually works great for revenue (I have yet never seen a handheld failure in scanning my tickets on either TGV or TER), but on the ticketing side it's absolute chaos (I had to wait at a ticket office for 20min because the internal ticket office portal refused to issue me with a Interrail pass holder reservation while the very sweet agent was wrestling with it)

 

[PG]

Tech news site The Register reporting on this issue which contains a report of a possible workaround:

There is a faulty channel file, so not quite an update.
There is a workaround...

1. Boot Windows into Safe Mode or WRE.

2. Go to C:\Windows\System32\drivers\CrowdStrike

3. Locate and delete file matching "C-00000291*.sys"

4. Boot normally.

 

[Lytham Local]

Bet the companies don't use that brand of corporate spyware are feeling smug!

We are, but we also aren't getting much in the way in from customers, so it works both ways.

 

[eoff]

The MS issue is what has caused it all.

Care to explain?

I saw the BBC website report this morning so straight to The Register (my go to for over the top IT coverage) to hear of the Crowdstrike isue, Not clear to me why anyone with business critical infrastructure would allow untested updates onto their internal machines.
Anyway, I'm not working today but otherwise would have not started up my work laptop without looking into this further.

 

[uglymonkey]

I think our Trident missile submarines work off microsoft products