Total shutdown at Paddington from 1600 on 6 July

[Marklund]

For the record, yes "****" indicates that the Track Circuit is occupied, but that there is no Train Describer data to show which train it is.

Whilst signallers might enter some unothodox text into berths from time to time, I'm far from convinced swear words would be among them!

It was very much tongue in cheek.
And you'd be surprised what can be put in a train describer

 

[Sunset route]

It was very much tongue in cheek.
And you'd be surprised what can be put in a train describer

Not much these days now that everybody is watching

 

[455driver]

Anyone have any idea what the underlying problem(s) with the signalling on the GWML on Wednesday and Thursday were?

Yes, same fault both days, it wasnt anything to do with the track or signals.

 

[philthetube]

A computer going bang causes much bigger issues than a cable breaking, there were probably very few ways that such a big failure could happen.

 

[JN114]

Anyone have any idea what the underlying problem(s) with the signalling on the GWML on Wednesday and Thursday were?

Datalink failure.

For signalling of the vintage at Airport Jn (1990s design) the signal box sends coded messages along telecoms cabling (fibre optic these days) to trackside location cases to control the various bits of signalling furniture. The layout is somewhat tree-like: big datalink from Signal Box to the various interlocking "super" locations; which then splits down to the datalinks for smaller trackside functional modules which control just a small handful of pieces of equipment each - I seem to recall something like eight elements per TFM - where a Track Circuit is worth one; point 'end' two and signal four.

Wednesday the communications equipment that receives and sends messages over the datalink for the modules that have the Main Line points and track circuits at Airport Jn stopped working. This causes signals to show red, track circuits occupied and points out of correspondence (not normal or reverse) - impossible to run trains essentially. The Relief lines were available.

Thursday a similar failure occurred slightly further up the chain, which knocked out all lines at Airport Jn.

 

[Gerald Fiennes]

Thanks, JN114.

I can see stopping the datalink failing is crucial, but any ideas on why it takes so long to restore service when it does go wrong? Does each connected device have to be individually correlated back onto the link? In other words, since it sounds like no hardware was damaged, isn't this just a software thing? And is there no faster recovery approach given the massive impact these incidents have (3rd or 4th one in recent memory)?

 

[jimm]

What looks like a GWR staff briefing from Mark Hopwood (the managing director) has appeared on WNXX and Great Western Passengers' Forum explaining the fault and saying that Siemens and Alstom are working with Network Rail on the issue.

This week’s infrastructure problems

As you’ll be aware we’ve had two days of catastrophic infrastructure failure in the Thames Valley, principally around the Hayes - Airport Junction area. This has massively impacted on our evening peaks on Wednesday and Thursday evenings, delaying and frustrating our customers across our network. It has also put huge strain on colleagues in GWR and in Network Rail.

The failure is a repeat one, based upon information between the points giving detection, i.e. if they are fully pointing in the right direction, and therefore how the signalling can work. This information all must be present for signallers in the Thames Valley Signalling Centre to signal trains.

For two nights running, as we’ve gone into the evening peak, as the air temperature has gone up and we’ve lost all the track circuits and detection on all the points in that area due to a problem with data transmission. Without that detection, the Signallers can’t signal trains and move them. The Signallers need confirmation that the points are in position. To do that they are ‘clipped’ or clamped and a process called Temporary Block Working (TBW) put in place. TBW effectively gets a reduced number of trains through a section. So when it goes in, we know the timetable plan isn’t going to work.

Over the last two nights this TBW process has gone badly and been too slow to implement. This has meant huge delays and confusion and ultimately the distressing scenes that we’ve witnessed.

I’ve been speaking to the Chief Operating Officer for Network Rail Western, Route Gareth Vest, who is committed to improving the situation and offers a full apology. As someone who has worked for a train operator, he knows the impact this has on people and how they react. He has put a huge amount of resource together to get this right.

Fundamentally NR have an issue around understanding the technical failure and as a result have brought in additional tech support from the manufacturers of the signalling equipment (Siemens and Alstom). I am in regular contact with Gareth and his team to ensure they get to the bottom of the failure and the mitigation for the failure.

In order to protect ourselves this evening, we have a number of points ‘clipped’ already, whilst the NR team watch data streams from the point work (transmitted to the TVSC that tell them they are working). They have local teams watching and patrolling the area in case of a repeat failure.

I can’t promise that we won’t have any issues this evening but I can say that at all levels of Great Western and First Group, we are working with Network Rail to ensure they get this right.

I want to thank all of you on behalf of the Exec Team for your effort and professional conduct throughout a really tough couple of days. I am always amazed by what GWR can do in a crisis, from manning stations, to crew resourcing through to the energy and robustness of our Control and everything in between.

 

[PHILIPE]

Although praising GWR staff, he doesn't mention what was the biggest beef among passengers which was the lack of information. I know it's difficult to provide accurate information and the forecast of a normal working resumption time in such circumstances but there is so much conflicting information from different staff

 

[SpacePhoenix]

Had the datalink overheated and shut itself down to prevent damage to itself?

 

[GodAtum]

Can anyone confirm how one could have gone from marlow to paddington from 1700 on 6 July? Looks like

Marlow - Bourne End 1747 - 1754
Bourne End - Maidenhead 1803 - 1832
Maidenhead - paddington 2128 - 2211

 

[JN114]

Can anyone confirm how one could have gone from marlow to paddington from 1700 on 6 July? Looks like

Marlow - Bourne End 1747 - 1754
Bourne End - Maidenhead 1803 - 1832
Maidenhead - paddington 2128 - 2211

That tallies with the level of delay scene along the Thames Valley on both days.

Bus Ticket acceptance was in place with First Berkshire which could have got you Maidenhead to Heathrow for Piccadilly line which was also accepting GWR tickets. Similarly First Berkshire Maidenhead to Windsor thence SWT from Riverside.

But regrettably I would not be surprised if the quoted itinerary was the best achievable solely with GWR trains on the day.

 

[father_jack]

What looks like a GWR staff briefing from Mark Hopwood (the managing director) has appeared on WNXX and Great Western Passengers' Forum explaining the fault and saying that Siemens and Alstom are working with Network Rail on the issue.

It was from Rob Mullen, ops director.

 

[83G/84D]

What looks like a GWR staff briefing from Mark Hopwood (the managing director) has appeared on WNXX and Great Western Passengers' Forum explaining the fault and saying that Siemens and Alstom are working with Network Rail on the issue.

Whilst I am not in possession of the timeline of events I think it is important to say that Temporary block working is slow to implement. People have to be found then get to the right locations with the right equipment.

I doubt even the London area has a unlimited number of Network Rail response staff waiting to respond to such an incident and the ones that are on duty may be miles away carrying out other duties.

When they get to the locations concerned they have to access the track in a safe manner and Temporary block working introduced in a safe way.

Whilst everyone wants trains moving as soon as possible after such a catastrophic signalling failure it has to be done safely without cutting corners.

Whilst I am unsure if anything could be done better in these circumstances you cannot expect instant solutions and people have to be patient no matter how frustrating it is if you are sat on a stationary train or standing on a crowded platform.
I am sure if there are lessons to be learnt they will be but fixing things like this when it goes wrong are not as straightforward as it may seem.