Loss of signalling control at Thames Valley Signalling Centre

[tsr]

As of around 0015 tonight (Monday 04/11/2013), Thames Valley Signalling Centre lost all control of signalling equipment between London Paddington & Didcot Parkway, plus applicable surrounding routes, such as the line to Earley (North Downs Line).

All trains which remain in the area tonight are at a stand by means of the failsafe mechanisms and systems, and have been for the last 40-45mins (at time of posting). Taxis have been ordered at Reading and possibly other stations, which will take passengers where numbers permit.

This will be a massive problem if it carries on towards the morning peak and disrupts things - stock movements included.

To add insult to injury, so to speak, animals on the line disrupted XC services through Lapworth (Chiltern Main Line route) earlier today, meaning some passengers will now have been stuck at Reading much later than they first imagined...
--- old post above --- --- new post below ---
Update: It now looks like most trains are now able to move in the wider area, especially around Reading. Phew...!

 

[jon0844]

Maybe someone accidentally unplugged the control system to charge up their phone!

 

[the sniper]

Well that nearly ruined a few people's prospect of a quiet night shift!

Welcome to the future folks, the age of the world ending Rail Operating Centre collapses...

 

[Temple Meads]

It definitely makes me feel rather concerned about huge areas under the control of a single box..

 

[Captain Chaos]

Ah, you beat me to the punch! Just woke up to see this. On way to go to Reading now. Still failed apparently. No trains have been able to get on or off depots either so even if it gets fixed immediately this is now going to have some impact I would have thought. Brace yourselves everyone. The impact from this could be huge. Ticket acceptance with Virgin, SWT, Southern, Chiltern and buses etc, etc. First Monday early in weeks and this happens! Looks like it's going to be a bad day at the office for everyone!

 

[Dieseldriver]

If they insist on putting all their eggs in one basket.... :roll:

 

[MadCommuter]

NRE reporting power restored.

 

[Oliver]

Not too bad now:

http://ojp.nationalrail.co.uk/service/ldbboard/dep/RDG

 

[Captain Chaos]

Indeed. I think the words nick of time spring to mind!

 

[Manchester77]

I thought these ROCs had to have backup transformers in the case of power loss?

 

[Nym]

I thought these ROCs had to have backup transformers in the case of power loss?

Wouldn't account for a system failure between the gennie and equipment, repeating breakers or hot cables come to mind.

 

[thelongestroad]

Bit of a running joke really. Some embarrassingly frustrating reliability yet again.

 

[NSEFAN]

It would be interesting to see what caused this. One would hope that Fort Knox would have enough redundancy to cover for most failures!

 

[dysonsphere]

As I have mentioned before 1 large center is always going to be prone to a failure shutting down large areas at once. Even having a duplicate center next door there is always going to a single point of failure somewhere. I was working a few years ago at a well known TV station and we had an overnight test of standbye power. Mains was dropped and dropped off air, turned out that lots of bits and pieces had been plugged into non maintained supplys. When this was fixed next time we tried a few nights later (its always done at 0300) genny tripped out. No one had checked load for years, cue delivery next day of 1/2 mega wattt genny and a lot of cablle. so the morale is dont put everyone thing in 1 place.

 

[Flamingo]

Same thing happened with the new super-dooper South Wales Signalling Center in Cardiff a few months ago.

 

[ainsworth74]

How do they cope on the continent with large signalling centres?

 

[YorkshireBear]

How do they cope on the continent with large signalling centres?

They do what we are planning and each one can take the workload of 3 if needed.

 

[NSEFAN]

I believe in the Netherlands, for example, there are 2 redundant control centres should the main one go down. That said, I have heard that there have been days where everything has fallen over, the redundant centres couldn't take over for some reason and passengers were told to come back tomorrow!

 

[ushawk]

Three Bridges had an emergency shutdown a few months ago too which shutdown pretty much all trains South of London for an hour. Once all ROCs are up and running, then id expect other ROCs to take over the running of the centre which has issues.

 

[ainsworth74]

Right so the doomsayers are perhaps jumping the gun in predicting the end times once the ROCs are in use?

 

[NSEFAN]

There's always going to be days where things just break. The key is that NR are learning from when these things happen, to help reduce the chance reoccurence!

 

[Tomnick]

We're still a fair way off being able to transfer control from one ROC to another though - the 'hard wired' nature of each workstation, items such as LC controls specific to a workstation and the need for signalmen to learn and maintain competence on everything that they operate are some of the main barriers: not impossible to overcome, in time, but I'm not sure that anyone's yet certain whether it's a realistic prospect.

 

[IanXC]

They do what we are planning and each one can take the workload of 3 if needed.

I do wonder whether the risk would be better managed if there was a contingency site for each ROC rather than an ability for another ROC to take over its work, as this is likely to be difficult from a resourcing and signaller knowledge point of view. If the ROC assuming control is resource limited then there is the potential for impacts in its own operating area.

I suppose it all depends what the risk assesments look like, however chucking all the signallers on a bus from (for example) York to Doncaster would seem to me to offer a much quicker service restoration than expecting (perhaps) Derby to take over the work.

 

[Bald Rick]

There is no current plan for one ROC to back up another, in exactly the same way as there is no current plan for, say, Upminster IECC to oback up Liverpool St IECC.

 

[YorkshireBear]

There is no current plan for one ROC to back up another, in exactly the same way as there is no current plan for, say, Upminster IECC to oback up Liverpool St IECC.

Only relaying what i read in MR.

 

[TOCDriver]

Three Bridges had an emergency shutdown a few months ago too which shutdown pretty much all trains South of London for an hour. Once all ROCs are up and running, then id expect other ROCs to take over the running of the centre which has issues.

LOL Not a chance. You wouldn't have the staffing for one thing!

 

[NSEFAN]

LOL Not a chance. You wouldn't have the staffing for one thing!

That's not the kind of redundancy people had in mind.

 

[455driver]

That's not the kind of redundancy people had in mind.

But thats the whole point of these ROCs, to minimize the number of signallers (or whatever they are called this week) employed.

 

[NSEFAN]

But thats the whole point of these ROCs, to minimize the number of signallers (or whatever they are called this week) employed.

True, but there should still be some redundancy in the system otherwise it will hinder recovery from when there is a major failure (which will happen, no system is immune to a major outage). Given how much NR gets fined for each delay minute it causes and how large an area these ROCs cover, is it really cheaper just to let things fall apart than hire a few more people?

 

[rdeez]

My expectation is that there won't be enough redundancy initially, but following several high profile outages (that are bound to happen sooner or later when everything is up and running...) they will be forced to put better arrangements in place...