All Northern self-service ticket machines off line 13/7/2021

Killingworth

Established Member
Joined
30 May 2018
Messages
3,171
Location
Sheffield
A major fault as they seem to have all been offline all day, see website.

We are currently experiencing technical difficulties with our self-service ticket machines which mean all have to be taken off-line. We are investigating the issue and are working hard to have the machines up and running again as soon as possible.

We are sorry for any inconvenience this causes. In the meantime we are advising customers to either use our mobile app or website to purchase tickets in advance and, where necessary, to collect those tickets from one of our ticket offices.

Customers who have already bought tickets to be collected at a machine, or who would normally use ‘promise to pay’ slips, should board their booked service and either speak to the conductor or to Northern staff at their destination station.
 
Sponsor Post - registered members do not see these adverts; click here to register, or click here to log in
R

RailUK Forums

_toommm_

Established Member
Joined
8 Jul 2017
Messages
4,660
Location
Yorkshire
They did (or attempted to do) an upgrade yesterday, so it seems it maybe didn’t go as well as they wanted…
 

Vespa

Member
Joined
20 Dec 2019
Messages
820
Location
Merseyside
Ideally when you do a software fix or upgrade, you test it first on an isolated machine, then a dry run before roll out.

So what went wrong here ? Was it a rush job ?
 

Killingworth

Established Member
Joined
30 May 2018
Messages
3,171
Location
Sheffield
Seems they're still offline and have been since before their Tweet at 5.07pm on 13th. Must be quite a big problem.
 

Ken H

On Moderation
Joined
11 Nov 2018
Messages
3,276
Location
N Yorks
Ideally when you do a software fix or upgrade, you test it first on an isolated machine, then a dry run before roll out.

So what went wrong here ? Was it a rush job ?
and you have a recovery plan in case it goes wrong.
 

Killingworth

Established Member
Joined
30 May 2018
Messages
3,171
Location
Sheffield
It wasn't an upgrade, it was an urgent fix that couldn't be postponed
It's clearly proving to be a complex fix, the sort that has impacts for the business if not urgently done that outweigh the considerable inconvenience. It must be severe to leave machines with totally blank, dead screens.
 

DB

Guest
Joined
18 Nov 2009
Messages
5,036
Ideally when you do a software fix or upgrade, you test it first on an isolated machine, then a dry run before roll out.

So what went wrong here ? Was it a rush job ?

No idea, but depends where the issue is - if it's with the back-end service which the machines connect to then it will be far more difficult to test it first.

Testing one machine only works if it's an issue with the actual machines themselves.
 

Ken H

On Moderation
Joined
11 Nov 2018
Messages
3,276
Location
N Yorks
No idea, but depends where the issue is - if it's with the back-end service which the machines connect to then it will be far more difficult to test it first.

Testing one machine only works if it's an issue with the actual machines themselves.
you have a test system with a TVM dedicated to testing.
 

DB

Guest
Joined
18 Nov 2009
Messages
5,036
you have a test system with a TVM dedicated to testing.

You might well do, but if it's a big back-end infrastructure and that's where the upgrade is then it might be prohibitively expensive to have a duplicate system for like-for-like testing.

There should be a rollback plan though, in any case.
 

Philip

Established Member
Joined
27 May 2007
Messages
2,410
Location
Manchester
Passengers having to go to the ticket office...all of a sudden they are buying the correct ticket for their journey and being given good travel and fares advice. Perhaps they should be left off for good? ;)
 

yorksrob

Veteran Member
Joined
6 Aug 2009
Messages
30,294
Location
Yorks
Passengers having to go to the ticket office...all of a sudden they are buying the correct ticket for their journey and being given good travel and fares advice. Perhaps they should be left off for good? ;)

Only if there's a ticket office to go to !
 

py_megapixel

Established Member
Joined
5 Nov 2018
Messages
4,676
Location
Northern England
I have to wonder what this update was.

The last thing I think we saw affecting ticket machines on this scale was the Scheidt & Bachmann backend outage a few years ago, which had an even wider impact. But that was fixed, as I recall, within a matter of hours. This has dragged on for almost 4 days...
 

johntea

Established Member
Joined
29 Dec 2010
Messages
2,034
If they run Windows in the background it could be to do with trying to apply a fix for the recent 'PrintNightmare' security vulnerability - I know initially applying this knocked out a lot of Zebra label printers so it could be a similar situation with whatever hardware they use to print the tickets in the machines

Or they've rolled out an update which has knocked out their remote connectivity to the machines, that would be a headache!
 

py_megapixel

Established Member
Joined
5 Nov 2018
Messages
4,676
Location
Northern England
If they run Windows in the background
Oh god I hope not!

Or they've rolled out an update which has knocked out their remote connectivity to the machines, that would be a headache!
It would, and would presumably involve having to send a technician out to several hundred stations to apply a fix. That could be a long process I'd have thought
 

Dai Corner

Established Member
Joined
20 Jul 2015
Messages
3,631
Oh god I hope not!


It would, and would presumably involve having to send a technician out to several hundred stations to apply a fix. That could be a long process I'd have thought
Where do I apply? Decades of IT experience but little of the Northern rail network. Getting paid to put that right sounds ideal. :D
 

DB

Guest
Joined
18 Nov 2009
Messages
5,036
If they run Windows in the background it could be to do with trying to apply a fix for the recent 'PrintNightmare' security vulnerability - I know initially applying this knocked out a lot of Zebra label printers so it could be a similar situation with whatever hardware they use to print the tickets in the machines

Or they've rolled out an update which has knocked out their remote connectivity to the machines, that would be a headache!

The actual ticket machines when installed ran Windows 8 - I saw one being set up (quite unuusual for anything to use Windows 8 - most suppliers of kiosk equipment used 7 and then moved to 10). Not sure whether they still do. The back end could be anything - don't know what system they use.

The print spooler issue has been patched now by Microsoft, so should just be a case of installing the patch - and for good measure disabling the print spooler service on any server which doesn't need to be able to print.

It would, and would presumably involve having to send a technician out to several hundred stations to apply a fix. That could be a long process I'd have thought

I would be very surprised if they can't install patches remotely.
 

PeterC

Established Member
Joined
29 Sep 2014
Messages
3,278
and you have a recovery plan in case it goes wrong.
It just takes somebody reluctant to admit failure on the day. In my IT days I have known an operator make several attempts to apply a fix rather and end up overwriting the fall back.
 

py_megapixel

Established Member
Joined
5 Nov 2018
Messages
4,676
Location
Northern England
I would be very surprised if they can't install patches remotely.
Not if this update has caused some problem with the network drivers or something else related.

Indeed, though I've not seen one of the broken machines in person, from the descriptions it does seem a bit like the machines are actually unable to even boot.
 

3rd rail land

Member
Joined
30 Jan 2019
Messages
392
Location
Where the 3rd rail powers the trains
Not if this update has caused some problem with the network drivers or something else related.

Indeed, though I've not seen one of the broken machines in person, from the descriptions it does seem a bit like the machines are actually unable to even boot.
As already alluded to if they have lost remote connectivity to the TVMs then they have no choice but to visit each and every machine to apply whatever fix needs to be applied. They'll undoubtedly be treating this as a P1 incident, which is the highest priority incident there is.

Hopefully by now they will have identified a fix for this issue and it's just a case of applying said fix as quickly as practicable.
 
Last edited:

skyhigh

Established Member
Joined
14 Sep 2014
Messages
1,557
As already alluded to if they have lost remote connectivity to the TVMs then they have no choice but to visit each and every machine to apply whatever fix needs to be applied. They'll undoubtedly be treating this as a P1 incident, which is the highest priority incident there is.

Hopefully by now they will have identified a fix for this issue and it's just a case of applying said fix as quick as practicable.
Also worth noting it's not Northern staff responsible for the machines- they're maintained (both software and hardware/supplies) under a contract with the manufacturer. As I understand it they're the ones who decided the update was required.
 

DB

Guest
Joined
18 Nov 2009
Messages
5,036
It's Northern who will get the negative PR.

I expect there are penalties in the contract for this sort of thing, so the suppliers won't go unscathed either.

If Windows is completely broke, reimaging the machines might well be easiest.
 

lincolnshire

Member
Joined
12 Jun 2011
Messages
854
Talking to the man who came to look at the machine at the station I was at that day told me that the ticket re-stocking etc. was been done by another company and he only now deals with the computer etc side of the equipment and his company that he works for has now made 3 employees redundant.
So could now end up been a slow job if someone has to visit every Northern ticket machine to sort the problem.
 

3rd rail land

Member
Joined
30 Jan 2019
Messages
392
Location
Where the 3rd rail powers the trains
I expect there are penalties in the contract for this sort of thing, so the suppliers won't go unscathed either.

If Windows is completely broke, reimaging the machines might well be easiest.
There will absolutely be penalties for the supplier who is contracted to manage these machines.

Reimaging may indeed be the easiest and quickest option at this point. Actually they may have a backup they can used and restoring from that may be a better option.

When the change was raised they will have stated a backout plan however it almost certainly doesn't cover a complete outage such as this.

Personally I can't see there being any option but to visit every TVM to apply a fix.
 

Top