Worldwide IT Outage (Crowdstrike update)

[WelshBluebird]

There are other ways of accessing cash. For example you could walk into your bank's local branch and ask to withdraw cash in person . *

Branches that use systems that have just a high chance of being down in the same way as card machines (and again the tills shops use to take payment).

The reality is that cash use these days has the same reliance on technology as card use does for all but the very very few who horde cash at home.

 

[signed]

For example you could walk into your bank's local branch and ask to withdraw cash in person . *

Which you wouldn't be able to do if the system used to check your account is down

Cash is just moving the problem to a even more centralized system. Which could as well have been impacted by the CS incident

Cash is needed, but fully mobile payments, with as much of a decentralized infrastructure as possible while maintaining the possibility of government oversight (I.e no blockchain or whatever), are the future, not cards.

 

[OscarH]

Does it matter which company is to blame if the whole incident demonstrates why a cashless society is a bad idea?

Yes, sloppy bad journalism is a slippery slope even if the underlying point can still be made with the correct facts in this case

 

[Egg Centric]

Except, as has already been pointed out to you, it would still cause similar levels of chaos because cash machines and automated tills are also affected.

Only if you're not sensible enough to have a few hundred quid for emergencies in both your wallet and house - and perhaps car if you have one. This is common sense surely?

Tills may be down but you won't be stranded.

Similarly - unless you're in a house share or tiny studio in which case commiserations - it's common sense to have a reasonable amount of tinned food in your home.

Random problems always happen every so often and it costs nothing other than the opportunity cost of storing the above items and potential risk of theft to be prepared.

I appreciate this is probably not practicable if you're a 20 year old graduate who's just landed your first job, and similarly a whole generation are screwed in the south east housing wise*, but outside of these edge cases proper adults should be able to cope.

*the fact I wouldn't be able to provide for my family like this without unreasonable (imo) costs is one of the two main reasons I moved back up north.

 

[Lemmy99uk]

Only if you're not sensible enough to have a few hundred quid for emergencies in both your wallet and house - and perhaps car if you have one. This is common sense surely?

Fine for those people with ‘a few hundred quid’ to spare.

 

[Bantamzen]

Fine for those people with ‘a few hundred quid’ to spare.

I was just going to say that myself, not everyone is in that position to be able to do so.

 

[ivorytoast28]

Only if you're not sensible enough to have a few hundred quid for emergencies in both your wallet and house - and perhaps car if you have one. This is common sense surely?

How many people actually do this?

Let's be honest if everything has gone down for long enough to get to that point that last £200 in cash is needed imminently then that £200 is not going to solve that chaos that would ensue from no companies being able to access/use their money.

Could I afford this? Of course. Do I do it? No I'd rather have it invested that lying around the house somewhere

 

[DarloRich]

been at a history festival all weekend - missed all this! Are things fixed?

 

[pokemonsuper9]

been at a history festival all weekend - missed all this! Are things fixed?

Most companies have likely reset everything by now (bad weekend for tech support staff), but not all probably are done yet, especially if they've got lots of systems that they can't wipe remotely.

 

[lxfe_mxtterz]

been at a history festival all weekend - missed all this! Are things fixed?

Ticket machines still seem to be down. Had many people from all different stations along the Portsmouth Direct Line asking to buy tickets on my train earlier today.

 

[DarloRich]

Most companies have likely reset everything by now (bad weekend for tech support staff), but not all probably are done yet, especially if they've got lots of systems that they can't wipe remotely.

Ticket machines still seem to be down. Had many people from all different stations along the Portsmouth Direct Line asking to buy tickets on my train earlier today.

thanks chaps!

 

[davews]

SWR website says 30% of ticket machines are still down, waiting engineers. The ones at Martins Heron which were faulty yesterday were working fine this afternoon.

 

[sor]

Most companies have likely reset everything by now (bad weekend for tech support staff), but not all probably are done yet, especially if they've got lots of systems that they can't wipe remotely.

I bet a lot of IT depts will come to realise what that "intel vpro" sticker actually means now and are wishing they enabled it on all those hybrid/WFH laptops

 

[Egg Centric]

How many people actually do this?

Let's be honest if everything has gone down for long enough to get to that point that last £200 in cash is needed imminently then that £200 is not going to solve that chaos that would ensue from no companies being able to access/use their money.

Could I afford this? Of course. Do I do it? No I'd rather have it invested that lying around the house somewhere

Fine for those people with ‘a few hundred quid’ to spare.

I absolutely accept that some people will not be able to because they are students, are starting out on their career, live "paycheque to paycheque" (does anyone still use cheques?) or are disabled or otherwise unable to work (plus what I previously mentioned about the housing market). But most people can do it at very little opportunity cost. Having £250 in cash is costing you, what, £15 a year tops? I needed this as recently as Saturday actually, when I discovered that the barbers I was using didn't take card.

been at a history festival all weekend - missed all this! Are things fixed?

I really doubt it. I think some horror stories will be coming out. Actually I know of some very serious trouble some people might be in depending on whether their payroll department manages work arounds but I cannot put it publicly for obvious reasons.

This will be a slow burner.

You can look at e.g. r/sysadmin/ for some current tales of woe as well.

(Unrelatedly a "history festival" sounds awesome - what are the details?)

 

[JohnMcL7]

I bet a lot of IT depts will come to realise what that "intel vpro" sticker actually means now and are wishing they enabled it on all those hybrid/WFH laptops

I'm sure most IT departments are very aware what that vpro sticker means because of the number of security headaches it's caused in the past and have disabled it for good reason.

 

[dgl]

Here is a video on the incident, how it affected windows and why it blue screens. Seems the fix is to simply delete one file (the borked update) in safe mode.

 

[pokemonsuper9]

Seems the fix is to simply delete one file (the borked update) in safe mode.

I don't blame you for not knowing, the last few days have been chaotic, but this was known since early in.

 

[Crossover]

It is 100% their fault for designing business-critical infrastructure that has such a vulnerability. At one time companies tested and tested any updates, now they rely on online updates by third parties and are paying the price.

There are choices on how systems are accessible, what networks they are on, if any, what those networks allow, how staff are trained, who has physical access etc. This is one of the aspects that plays into Availability / Business Continuity in any Information Security Management System, having security at the risk or no availability is not great. Someone made that decision and I don't think you should complain if something like this happens and you decided to do no testing. You may wish to relax some requirements for very high risk threats of course. I don't see how doing at least an install and restart test is going to take that long, presumably that can be automated in virtual machines pretty quickly.

Mass testing of updates is generally nigh on impossible and some trust has to be placed in the third parties

I manage IT for an SME and I would need an army of people to test all the updates (including definition updates for the likes of AV which release regularly) before going into production. Some of our kit is also one of a kind within the business so have to take a bit of a flier on updating them (it being too cost prohibiltive to have multiple of them), be it they are patched manually.

What doesn't help, in some ways, is compliance requirements necessitating systems being patched within x period of time (14 days from patch release for some of the certifications, I believe), so things have to move fast by necessity

 

[bakerstreet]

Just tried to buy a ticket (SWR) today. Still out. Had forgotten about Friday’s events!

 

[nlogax]

Just tried to buy a ticket (SWR) today. Still out. Had forgotten about Friday’s events!

Either it's a Safe Mode recovery for each TVM or someone is desperately having to dig out Bitlocker keys.. presumably TVMs can be remotely accessed out-of-band, right..?

 

[Energy]

Either it's a Safe Mode recovery for each TVM or someone is desperately having to dig out Bitlocker keys.. presumably TVMs can be remotely accessed out-of-band, right..?

If they have Intel vPro or similar then yes it can be accessed remotely, though removing the file for each TVM is still a slow process.

If not then someone will need to physically go to each TVM.

Software based management solutions (e.g InTune) will not work as the system cannot boot.

 

[Russel]

So, I was on annual leave on Friday so missed all the fun, I went back into work on Monday to find two out of five of our PC's in the office are in recovery mode, despite IT sending out a step by step guide for how to fix them, no one bothered...

Sometimes I really dislike being the one "who knows about computers".

 

[eoff]

Seems that Crowdsrike has finally realised the obvious...

Falcon Content Update Remediation and Guidance Hub | CrowdStrike

Access consolidated remediation and guidance resources for the CrowdStrike Falcon content update affecting Windows hosts.

www.crowdstrike.com

(snippet...)

How Do We Prevent This From Happening Again?

Rapid Response Content Deployment

• Implement a staggered deployment strategy for Rapid Response Content in which updates are gradually deployed to larger portions of the sensor base, starting with a canary deployment.

 

[Egg Centric]

Seems that Crowdsrike has finally realised the obvious...

Not really.

This isn't the place to go into a lot of detail but those who are following this closely will know exactly the meaning of this image!



(Image shows a man, labelled "Crowdstrike", pouring petrol labelled "$10 Uber coupon", on a fire labelled "Angry customers")

 

[dangie]

So, I was on annual leave on Friday so missed all the fun, I went back into work on Monday to find two out of five of our PC's in the office are in recovery mode, despite IT sending out a step by step guide for how to fix them, no one bothered...

Sometimes I really dislike being the one "who knows about computers".

Why?
Surely in this day & age it’s better to have more strings to your bow. Make it so they can’t do without you.

 

[Russel]

Why?
Surely in this day & age it’s better to have more strings to your bow. Make it so they can’t do without you.

That is true, but it's annoying that in this day and age so many people are still helpless when it comes to I.T., especially when the instructions to solve the issue are in front of them...

 

[dangie]

That is true, but it's annoying that in this day and age so many people are still helpless when it comes to I.T., especially when the instructions to solve the issue are in front of them...

Switch it off. Wait five minutes. Switch it on again. Usually works for me

 

[GRALISTAIR]

Seems like Delta airlines are back to normal after suffering more cancellations over CrowdStrike than it did in the whole of 2019!

 

[eoff]

Not really.

This isn't the place to go into a lot of detail but those who are following this closely will know exactly the meaning of this image!

View attachment 162315

(Image shows a man, labelled "Crowdstrike", pouring petrol labelled "$10 Uber coupon", on a fire labelled "Angry customers")

I'm lost on the relevance of this. Crowdstrike won't care if lots of people want to moan on social media because they sent some vouchers to staff and partners.
Their customers are mostly very large corporations and businesses and those are the ones they should be trying to keep happy and that won't be happening in public.

 

[Egg Centric]

I'm lost on the relevance of this. Crowdstrike won't care if lots of people want to moan on social media because they sent some vouchers to staff and partners.
Their customers are mostly very large corporations and businesses and those are the ones they should be trying to keep happy and that won't be happening in public.

I assure you that it's annoyed a lot of staff in said very large corporations and businesses. Staff with power to decide if they keep Crowdstrike.