Nationwide signal issue reported on 6th December

[Class15]

North London, Suffragette and Midland lines chaotic this morning. Made travelling from Gospel Oak area very difficult.

 

[Harpo]

Going to be a monster TRUST delay TIN!

 

[AndrewE]

Should've seen Crewe station this morning, heavy delays and cancellations across the board

it's not looking too bad now, just a couple of locals cancelled (including the 0705 to Newark) most EMUs not far off right time or (the Lpool Brums) up to 1/2 hr late, Pendolinos mostly on time apart from a Brum-Edinburgh and one Eus -Glasgow being 1/2 late.
Looks like a fairly normal day on the WCML to me!

 

[skyhigh]

3 mobile network for internet has been down since around 1am

It isn't down here.

not sure if the two share any infrastructure

Highly doubt it.

 

[yorkie]

https://www.southernrailway.com/service-updates/service-updates?uq=638690715250240000#INCE8C222BEA5C34575A3B583977D8DD2A3

There is a nationwide fault with the train's onboard communication system. As a result, services across the Gatwick Express, Great Northern, Southern and Thameslink network may be cancelled, delayed by up to 15 minutes or revised.

We expect this to continue until 12:00.

Customer advice:

Services across the Southern, Thameslink, Great Northern and Gatwick Express network are expected to be disrupted this morning.

For most journeys, you can travel using your normal route. However, if your planned train is affected you may need to change trains en route, or use an alternative route. Please be aware that this will delay your journey by at least 15 minutes. Please check journey planners for the latest information regarding your intended train service.

The following services will be amended:

• Brighton to Southampton Central will only be running to either Bognor Regis or Chichester
• London Victoria to Ore will terminate and restart at Hastings
• London Victoria to Brighton (Gatwick Express) will terminate and restart at Gatwick Airport
• Kings Lynn to London Kings Cross will only run between Kings Lynn and Cambridge

Tickets will be accepted for no extra cost on the following:

• Southern, Thameslink and Great Northern services via any reasonable route
• London Underground on any reasonable route between London Terminals
• South Western Railway tickets between Southampton Central, Portsmouth Harbour, Portsmouth & Southsea and London Waterloo (please note, that South Western Railway are also experiencing disruption and their services will be extremely busy)

Please note that there is no ticket acceptance on Gatwick Express services at this time.

Planning your journey:

Live departure boards can be found here.

You can view a live map of the network here.

You can also see an alternative route guide here.

If you're travelling in the London area, you can plan journeys on alternative routes by using the TfL Journey Planner here.

You can also use GTR's app to find up-to-date information.

Can you tell me more about the incident?

A fault with the radio system between the driver and the signaller means that trains are unable to run to / from Moorgate this morning and services across the network may be delayed. Investigations are ongoing as to why this has occurred and efforts underway to get it rectified as soon as possible.

We'll provide further updates on this as soon as we get them.

Check before you travel:

You can check your journey using the National Rail Enquiries real-time Journey Planner.

Compensation:

You may be entitled to compensation if you experience a delay in completing your journey today. Please keep your train ticket and make a note of your journey, as both will be required to support any claim.

From the above: "Please note that there is no ticket acceptance on Gatwick Express services at this time.", I see GTR are making things awkward for their customers by trying to deter people using their GX branded services again.

Is anyone here affected by this, or experiencing this?

I would like to hear if anyone who has a ticket that GTR considers to be brand-restricted, that they claim is not valid on GX, who catches a GX train to minimise their delay, is charged an additional fare by GTR for their inconvenience!

 

[cheekybifta]

North Wales Coast seems fine but the Cambrian lines with their GSMR-dependedent signalling are completely shut.

 

[ThameslinkUser]

What on earth is that thread title?

 

[yorkie]

What on earth is that thread title?

Agreed; I have now amended it. Any issues can be brought to our attention with the 'report' link at the bottom of the relevant post (the opening post in the case of thread title issues).

 

[PG]

https://www.southernrailway.com/service-updates/service-updates?uq=638690715250240000#INCE8C222BEA5C34575A3B583977D8DD2A3

From the above: fault with the radio system between the driver and the signaller means that trains are unable to run to / from Moorgate this morning

IIRC Moorgate NR services run on the ENTC system. Is that dependent upon GSM-R to function?

 

[68000]

IIRC Moorgate NR services run on the ENTC system. Is that dependent upon GSM-R to function?

ETCS is reliant on GSM-R to communicate movement authorities to the train

 

[SCDR_WMR]

it's not looking too bad now, just a couple of locals cancelled (including the 0705 to Newark) most EMUs not far off right time or (the Lpool Brums) up to 1/2 hr late, Pendolinos mostly on time apart from a Brum-Edinburgh and one Eus -Glasgow being 1/2 late.
Looks like a fairly normal day on the WCML to me!

Yeah much better now, it was 'fun' at 5am this morning! Believe the wildcards are still not working on most trains there though but minimal impact as drivers and signallers are well aware

 

[jfollows]

Reported at https://www.theguardian.com/uk-news...rain-disrpution-network-delays-latest-updates:

8m ago10.37 GMT
The BBC’s transport correspondent Sean Dilley has reported that the cause of the communications fault that affected the British train network this morning was “a new hardware card installed overnight at the telecommunications node in Stoke.”

Again, what is it about IT upgrades of all sorts these days - why are they not properly tested when implemented and reversed if they don’t work? Or why is the infrastructure so complex that this isn’t possible?

 

[Western Sunset]

Reported at https://www.theguardian.com/uk-news...rain-disrpution-network-delays-latest-updates:


Again, what is it about IT upgrades of all sorts these days - why are they not properly tested when implemented and reversed if they don’t work? Or why is the infrastructure so complex that this isn’t possible?

I was just going to mention that.
"Overnight hardware/software upgrades" often seem to be a recipe for disaster in so many IT-related systems such as banking, air traffic control, logistics, etc.

 

[Efini92]

No, as long as the driver is connected to the network they seem to be getting permission to run. Though 1 unit this morning couldn't make or receive calls so that didn't run until it could.

Do you remember when we had this issue not long after the system was rolled out?

 

[merseydrew]

Are issues on WCML expected to last throughout the rest of the day?

 

[StoneRoad]

Yeah, complicated IT-based systems are fine, until they fall over.

[anyone recall the chaos when a crowdstrike update failed ?]

 

[Belperpete]

North Wales Coast seems fine but the Cambrian lines with their GSMR-dependedent signalling are completely shut.

According to Journey Check, all Cambrian services were suspended, on both the mainline and Coast.

The lines that seem to have been significantly affected seem to be those with ETCS signalling, which relies on radio to transmit the movement authorities. Hence the Cambrian. Thameslink core. And Lizzy line to Heathrow. And of course significant problems on those routes would have had knock on effects on other routes.

 

[jfollows]

From the BBC: (https://www.bbc.co.uk/news/live/cv2g2rrjywqt)

The BBC has been told the cause of the fault in national rail radio communications has been located and is being fixed.

Engineers have identified that a new hardware card installed overnight at the telecommunications node in Stoke is the source of the problem.

The hardware card was installed as part of an upgrade to the system - but drivers found this morning that the digital communications system connecting them to signals, known as GSM-R, lost communications and needed to be reconnected manually.

The system at Stoke is being rebooted and a new hardware card installed.

All communications traffic is being managed successfully through a different communications node in Didcot.

 

[Unixman]

Talk about single points of failure. A swap out of one card in one telecoms centre caused this? Are they saying that there is no resilience in the system?

Edit to add. Just seen the BBC report saying that there was a failover to a centre in Didcot. Not very successfully it seems.
I suspect that the DR plan is being updated right now.

 

[Zontar]

Latest conspiracy theory from the mess room, it's purposely been simulated as an attack on infrastructure to see how we would cope.....

 

[Palmerston]

To make matters worse, on the SWML, they turned the electricity back on this morning before the tracks were clear. I've heard no fatalities but did cause damage.

 

[LNW-GW Joint]

According to Journey Check, all Cambrian services were suspended, on both the mainline and Coast.
The lines that seem to have been significantly affected seem to be those with ETCS signalling, which relies on radio to transmit the movement authorities. Hence the Cambrian. Thameslink core. And Lizzy line to Heathrow. And of course significant problems on those routes would have had knock on effects on other routes.

Nevertheless, RTT is showing trains moving on the Cambrian.

Realtime Trains | 1I08 0729 Aberystwyth to Birmingham International | 06/12/2024

Real-time train running information for 1I08 0729 departure from Aberystwyth to Birmingham International on 06/12/2024. From Realtime Trains, an independent source of train running info for Great Britain.

www.realtimetrains.co.uk

 

[bleeder4]

Latest conspiracy theory from the mess room, it's purposely been simulated as an attack on infrastructure to see how we would cope.....

So we failed the test then.

Russia now knows that if it wants to take out our transport infrastructure just before an attack, it just needs to bomb Stoke.

 

[Cricketer8for9]

The reporting of it is frankly bizarre. BBC News refers to a 'nationwide fsult', but then say that afffected services include South Western Railway, Gatwick Express, Great Northern, Southern, Southeastern and Thameslink.

Very long way from nationwide then...

Did the BBC report contain the typos as well?

 

[Class 317]

I think it's unfair to say the contingency plans not worked today. Most delays around 15 mins and majority of services operating normally.

 

[Belperpete]

So we failed the test then.

Russia now knows that if it wants to take out our transport infrastructure just before an attack, it just needs to bomb Stoke.

I suspect that if Stoke had been bombed, it would have been noticed much quicker, and remedial action taken much quicker. I suspect the issue was more subtler than that, with the faulty card appearing to be working but actually screwing the system up.

 

[Unixman]

I think it's unfair to say the contingency plans not worked today. Most delays around 15 mins and majority of services operating normally.

As an IT person who deals in major infrastructure I don't think it was unfair. Why wasn't the system switched over BEFORE the work was carried out?

 

[Belperpete]

Nevertheless, RTT is showing trains moving on the Cambrian.

Realtime Trains | 1I08 0729 Aberystwyth to Birmingham International | 06/12/2024

Real-time train running information for 1I08 0729 departure from Aberystwyth to Birmingham International on 06/12/2024. From Realtime Trains, an independent source of train running info for Great Britain.

www.realtimetrains.co.uk

Presumably, after they got the system working again.

 

[Horizon22]

I was just going to mention that.
"Overnight hardware/software upgrades" often seem to be a recipe for disaster in so many IT-related systems such as banking, air traffic control, logistics, etc.

There are of course probably hundreds of updates that work perfectly normally overnight everyday.

It’s just that when one goes wrong it really goes wrong and the redundancy / contingency seems minimal.

 

[Belperpete]

As an IT person who deals in major infrastructure I don't think it was unfair. Why wasn't the system switched over BEFORE the work was carried out?

They may well have done so, with the problem only becoming evident some time after they switched back.

There are of course probably hundreds of updates that work perfectly normally overnight everyday.

It’s just that when one goes wrong it really goes wrong and the redundancy / contingency seems minimal.

No, it's just only the one that goes really wrong makes the headlines.