All Northern self-service ticket machines off line 13/7/2021

[skyhigh]

Having seen the real reason for the outage - it was not an update that caused the outage, the machines have deliberately been switched off (they don't require individual visits to restore software) and there was no option for the supplier other than to turn the machines off immediately until the issue was sorted. The issue is entirely on the side of the supplier.

 

[Dai Corner]

Not if this update has caused some problem with the network drivers or something else related.

Indeed, though I've not seen one of the broken machines in person, from the descriptions it does seem a bit like the machines are actually unable to even boot.

"If its not broken, don't fix it" comes to mind.

Or " test twice, deploy once" to paraphrase a carpenter's saying.

 

[yorksrob]

Aren't Northern ticket machines supplied with a back up CD ?

 

[DB]

"If its not broken, don't fix it" comes to mind.

The issue there is when a gaping security hole comes to light - leaving it alone is not an option.

 

[flitwickbeds]

"If its not broken, don't fix it" comes to mind.

...is the worst attitude when it comes to technology, and usually only spoken by those who have no real understanding of the issues being (or at least attempting to be) fixed during the change/upgrade.

 

[DB]

Make an 'image' by setting up a machine exactly as you want it. Take a copy. Replace whatever's on each faulty machine with that copy.

Indeed, or variations such as deployment systems like Intune.

For anyone interested, Intune is a Microsoft cloudy service which deploys programs and settings onto computers - so a new computer is enrolled into Intune, and put into whichever configuration groups are needed. A clean install of Windows is then put onto the machine, and Windows will then see that it's an Intune-managed machine and allow Intune to deploy the software and settings.

...is the worst attitude when it comes to technology, and usually only spoken by those who have no real understanding of the issues being (or at least attempting to be) fixed during the change/upgrade.

Quite - the reality is you leave it alone if you can, but recognise and act quickly when that's not an option - i.e. when there's a security hole which needs patching fast.

 

[Dai Corner]

Indeed, or variations such as deployment systems like Intune.

For anyone interested, Intune is a Microsoft cloudy service which deploys programs and settings onto computers - so a new computer is enrolled into Intune, and put into whichever configuration groups are needed. A clean install of Windows is then put onto the machine, and Windows will then see that it's an Intune-managed machine and allow Intune to deploy the software and settings.
[

The problem comes when your machines are so FUBAR'd your remote management system stops working.

 

[skyhigh]

The problem comes when your machines are so FUBAR'd your remote management system stops working.

It's the complete opposite here - the machines themselves are fine (there's been no software changes to them). The management system is the issue.

 

[flitwickbeds]

Indeed, or variations such as deployment systems like Intune.

For anyone interested, Intune is a Microsoft cloudy service which deploys programs and settings onto computers - so a new computer is enrolled into Intune, and put into whichever configuration groups are needed. A clean install of Windows is then put onto the machine, and Windows will then see that it's an Intune-managed machine and allow Intune to deploy the software and settings.



Quite - the reality is you leave it alone if you can, but recognise and act quickly when that's not an option - i.e. when there's a security hole which needs patching fast.

But, you also act when you WANT to as well, not just NEED to. Most of the technology we take for granted in 2021 wouldn't have happened if nobody had moved on because "it wasn't broken".

 

[Vespa]

Well... Its broken now by trying to "fix it"

 

[flitwickbeds]

Well... Its broken now by trying to "fix it"

Yes. The risk you take by fixing or changing anything, even tiny things. Sometimes it goes wrong, sometimes it goes catastrophically wrong. But that's no reason to never change anything, ever.

There will have been dozens/hundreds/thousands of times these machines were successfully upgraded without anyone knowing any different.

 

[yorksrob]

I must admit, I started using Northern's ticket machines last year and I quite like them.

 

[py_megapixel]

Well... Its broken now by trying to "fix it"

That's because either fixing it was handled incompetently, or Flowbird's systems do not facilitate testing adequately. Nothing inherently wrong with the idea of updating software.

Honestly given how poor quality the software on those machines was to start with, I'm not entirely surprised that this has happened

 

[DB]

But, you also act when you WANT to as well, not just NEED to. Most of the technology we take for granted in 2021 wouldn't have happened if nobody had moved on because "it wasn't broken".

Yes, but you have a lot more time to plan it then, so major issues are less likely (although not, of course, impossible!)

Well... Its broken now by trying to "fix it"

But sometimes issues have to be fixed. I don't know what is being addressed in this case, but urgent fixes are normally due to security issues which have come to light - what you you prefer: that it's left alone because it's not "broken", and there's a major security breach as a result, or an attempt is made to fix it, with a small chance of something going wrong?

 

[Killingworth]

---

It's the complete opposite here - the machines themselves are fine (there's been no software changes to them). The management system is the issue.

The machines are switched on and humming as usual but with a totally blank screen offering quite a good mirror, see picture at 12.50. No information at the majority of stations to explain what to do if collecting tickets. It might be useful to have a permanent printed notice on or near the machines to explain what to do if out of order.

 

[roversfan2001]

Both machines at Leyland are showing an "out of service" message, rather than being totally blank.

 

[skyhigh]

Well... Its broken now by trying to "fix it"

No, it's not. There was an issue (not caused by a software update), that meant the only option was to take all the machines offline until it could be fixed. This isn't an issue caused by poor testing, software updates or people trying to 'fix things'.

 

[SteveM70]

There will absolutely be penalties for the supplier who is contracted to manage these machines

Unless you’ve seen the contract, I’d replace “will” with “should”. This is Northern after all

 

[Emyr]

An up-to-date set of infrastructure for testing, close enough to the live/production/realworld kit for testing to be meaningful, has a significant capital and operating cost. Those who forego this cost by downsizing or de-featuring the test environment aren't necessarily the ones who would bear the cost of an outage. Applies even more when a system is created or changed by a contractor chosen via competitive tendering.

No, it's not. There was an issue (not caused by a software update), that meant the only option was to take all the machines offline until it could be fixed. This isn't an issue caused by poor testing, software updates or people trying to 'fix things'.

A software update is an attempt to "fix things".

 

[Vespa]

I don't know what is being addressed in this case, but urgent fixes are normally due to security issues which have come to light - what you you prefer: that it's left alone because it's not "broken", and there's a major security breach as a result, or an attempt is made to fix it, with a small chance of something going wrong?

Of course I would like it fixed, my initial comment was tongue in cheek, it has sprung a lot of defensive responses.

As I said before it should have been tested before roll out or somebody said above " Test twice, deploy once"

There is clear failure in the "fixes"

I always say "Prior Planning and Preparation Prevents Poor Performance"

 

[py_megapixel]

Seems we're wrong about them being completely shut down. Just passed through a Northern station now and the machine was displaying an exclamation mark accompanied by some text which was too far away to read.

I shall be alighting at another Northern station shortly and will try to upload a picture of the machine there.

 

[skyhigh]

A software update is an attempt to "fix things".

Yes, but as I said, no software updates have been done. That's not the cause of this.

 

[DelW]

No, it's not. There was an issue (not caused by a software update), that meant the only option was to take all the machines offline until it could be fixed. This isn't an issue caused by poor testing, software updates or people trying to 'fix things'.

I realise that you're limited in what you can say publicly, but I take your comments as meaning that it's the central ticketing system that has been compromised, not the "outstations", I.e. the ticket machines.

On a different note, I haven't seen anyone suggesting a deliberate attack, but is that a possibility? We have heard quite a bit recently of both attempted and /or successful attacks on software systems, either by criminal or foreign power hackers. I can't see why a ticket system would be particularly a target, but the same applies to some systems which have been attacked.

 

[py_megapixel]

The ones I've seen today have been on and displaying the standard background image (i.e. blue with some terms and conditions in the bottom right), but instead of buttons to buy tickets there is a message stating that the machine is out of service. It advises that one tries another machine - which is unhelpful at the best of times, when the other machine, if it exists at all, could not be easy to find. But it's even worse when the other machine is also guaranteed to be out of service!

 

[DB]

The ones I've seen today have been on and displaying the standard background image (i.e. blue with some terms and conditions in the bottom right), but instead of buttons to buy tickets there is a message stating that the machine is out of service. It advises that one tries another machine - which is unhelpful at the best of times, when the other machine, if it exists at all, could not be easy to find. But it's even worse when the other machine is also guaranteed to be out of service!

Probably a default message when it can't connect to the bac-end service.

 

[the sniper]

Will somebody please think of the lost revenue! WE ARE ALL PAYING FOR THIS NOW!!!

 

[py_megapixel]

Probably a default message when it can't connect to the bac-end service.

Indeed, but ideally whoever wrote the default message would have considered phrasing it to encompass a scenario where the backend is at fault (and therefore none of the machines are working!)

Actually what I'd hoped would have happened is that something else would be advising people of the circumstance. Surely it can't take long for someone at Northern head office to bash out an appropriately worded sign in Word, email a PDF to the station staff, and give an instruction to print and affix a copy to each TVM? That doesn't deal with the unstaffed stations of course, but it's a start at least.

 

[Vespa]

On a different note, I haven't seen anyone suggesting a deliberate attack, but is that a possibility? We have heard quite a bit recently of both attempted and /or successful attacks on software systems, either by criminal or foreign power hackers. I can't see why a ticket system would be particularly a target, but the same applies to some systems which have been attacked.

Disabling a ticket system would be disruptive to ordinary peoples lives, I wouldn't underestimate low level targets, disrupting an infrastructure would have a knock on effect beyond the immediate event such as banking, security, government staff being unable to attend work......

 

[Killingworth]

Still totally blank screen at Dore & Totley.

 

[DB]

Indeed, but ideally whoever wrote the default message would have considered phrasing it to encompass a scenario where the backend is at fault (and therefore none of the machines are working!)

The machine is unlikely to be able to tell whether or not other machines are affected - and in most cases they won't be; it'll be an issue with that particular machine's network connection.