Rail Operating Centre reliabilty

Status
Not open for further replies.

Sleepy

Member
Joined
15 Feb 2009
Messages
1,118
Location
East Anglia
Today Romford ROC had a power failure which stopped all services to/from London Liverpool St. Presumably there will some difficult questions asked as to why backup supply did not work. How unusual is a complete meltdown in ROC PSB etc ?
 
Sponsor Post - registered members do not see these adverts; click here to register, or click here to log in
R

RailUK Forums

Elecman

Established Member
Joined
31 Dec 2013
Messages
2,258
Location
Lancashire
I cant comment on specifics of Romford but the ROCs I dealt with have 2 independant routed DNOs from Primary substations both of which are rated to supply the whole building but are normally set up to provide half the building each. Both with HV and LV busbar interconnectors as well as 2 independant Emergency Generators so mitigating against total loss of supply.
 

LowLevel

Established Member
Joined
26 Oct 2013
Messages
5,356
EM ROC at Derby has died completely in the past on a couple of occasions. Can't recall what the root cause was in the end.
 

HSTEd

Veteran Member
Joined
14 Jul 2011
Messages
13,531
I'm afraid this is the sort of thing where design or operational oversights only become apparent through actual experience.

Hopefully whatever caused the failure this time will be discovered and mitigated against moving forward.
 

Bald Rick

Veteran Member
Joined
28 Sep 2010
Messages
19,549
Interesting, as the routes from Liverpool St are not signalled from Romford.

They are, however, signalled from Liverpool St IECC (est. 1989). I suspect that the power failure was there, although that’s not to say there may not have been related issues at Romford ROC also.
 

Shwam3

Member
Joined
13 Jun 2016
Messages
100
Location
Norwich
Interesting, as the routes from Liverpool St are not signalled from Romford.

They are, however, signalled from Liverpool St IECC (est. 1989). I suspect that the power failure was there, although that’s not to say there may not have been related issues at Romford ROC also.
Although the signallers are still located in Liverpool Street & Upminster IECCs, the actual interlockings are now located in the ROC (NLL excluded for the time being).
The entire c2c/LT&S area was also at a stand during this incident.
 

Bald Rick

Veteran Member
Joined
28 Sep 2010
Messages
19,549
Although the signallers are still located in Liverpool Street & Upminster IECCs, the actual interlockings are now located in the ROC (NLL excluded for the time being).
The entire c2c/LT&S area was also at a stand during this incident.

Of course (bangs head on table). :oops:
 

Ediswan

Member
Joined
15 Nov 2012
Messages
958
Location
Stevenage
I'm afraid this is the sort of thing where design or operational oversights only become apparent through actual experience.
Agreed. I have no railway experience, but I do have some with computer data centers. 'Cannot fail' multiply redundant power supply systems fail all the time.

I came across one that failed over to battery then diesel generators just fine, but refused to revert to mains power. It required some live working on the control logic to overcome that design flaw.
 

DB

Guest
Joined
18 Nov 2009
Messages
5,036
Agreed. I have no railway experience, but I do have some with computer data centers. 'Cannot fail' multiply redundant power supply systems fail all the time.

I came across one that failed over to battery then diesel generators just fine, but refused to revert to mains power. It required some live working on the control logic to overcome that design flaw.

The level of centralisation is perhaps going a bit far these days - on the railways it's ROCs, in the general business world it's datacentres (e.g. Microsoft's three in the UK). If something were to happen which took one or more of these out for an extended period the disruption would be considerable.
 

ainsworth74

Forum Staff
Staff Member
Global Moderator
Joined
16 Nov 2009
Messages
23,678
Location
Redcar
The level of centralisation is perhaps going a bit far these days - on the railways it's ROCs, in the general business world it's datacentres (e.g. Microsoft's three in the UK). If something were to happen which took one or more of these out for an extended period the disruption would be considerable.
We've still not gone as far as Union Pacific who run their entire 36,000 mile network from one facility. So it could be worse ;)
 

Gloster

Established Member
Joined
4 Sep 2020
Messages
2,351
Location
Up the creek
The level of centralisation is perhaps going a bit far these days - on the railways it's ROCs, in the general business world it's datacentres (e.g. Microsoft's three in the UK). If something were to happen which took one or more of these out for an extended period the disruption would be considerable.
I have no doubt that the Russians, Chinese, Americans, ISIS, etc. all know this: the MoD probably do too. However, this is a political problem: we (not just the UK) are governed by those who put short term financial gain over long-term planning, either by themselves or those in the business world who back them.

More to the point. If I have understood things correctly, there is no facility for one ROC to take over another’s area, even if only sufficiently to keep main routes operating. Nor are there any emergency control panels at major stations or junctions to allow things to be kept moving. If there was a major failure it would be back to talking trains from signal to signal, but if the ROC is out, where are you going to do that from?

I hope I am wrong in my understanding of the current situation, but I fear I am not. If a thing can go wrong, it will.
 

Bald Rick

Veteran Member
Joined
28 Sep 2010
Messages
19,549
I have no doubt that the Russians, Chinese, Americans, ISIS, etc. all know this: the MoD probably do too. However, this is a political problem: we (not just the UK) are governed by those who put short term financial gain over long-term planning, either by themselves or those in the business world who back them.

More to the point. If I have understood things correctly, there is no facility for one ROC to take over another’s area, even if only sufficiently to keep main routes operating. Nor are there any emergency control panels at major stations or junctions to allow things to be kept moving. If there was a major failure it would be back to talking trains from signal to signal, but if the ROC is out, where are you going to do that from?

I hope I am wrong in my understanding of the current situation, but I fear I am not. If a thing can go wrong, it will.

There are, now, very few control locations that have ‘emergency’ panels at other . In any event they were rarely at a major station.

And ROCs could, in theory, back each other up. But there are three crucial points:

1) where do you put the spare control equipment ? Does each ROC have to be twice the size for the blue moon occasion?
2) who controls the spare equipment? Does each ROC have to have twice the number of signallers for the blue moon occasion?
3) if you have lost power to the interlockings (as today, apparently) it doesn’t matter where you control it from, trains aren’t going to move.
 

Highlandspring

Established Member
Joined
14 Oct 2017
Messages
2,779
An interesting one I have personally experienced a few times is that when one of the ROCs is working on the standby generator there’s no cold or hot running water anywhere in the building due to the vagaries of the plumbing system. It’s fine for an hour ir two but not for days on end...
 

Skie

Member
Joined
22 Dec 2008
Messages
586
I'm afraid this is the sort of thing where design or operational oversights only become apparent through actual experience.

This is where a lack of failover and disaster recovery tests will bite you on the bum.
 

Elecman

Established Member
Joined
31 Dec 2013
Messages
2,258
Location
Lancashire
An interesting one I have personally experienced a few times is that when one of the ROCs is working on the standby generator there’s no cold or hot running water anywhere in the building due to the vagaries of the easily system. It’s fine for an hour ir two but not for days on end...
I can’t work out why you would lose running cold water in that situation unless its pumped sealed break tank system and someone has connected the pumps to the non essential distribution system which is easily rectified
 

HSTEd

Veteran Member
Joined
14 Jul 2011
Messages
13,531
There is also the problem that it doesn't take all signals being out to collapse the service in the area.

Imagine the signalling system responsible for Clapham Junction goes out.
It doesn't matter if the signalling systems on the attached lines worked, very rapidly you will have total gridlock and half the former Southern Region is paralysed regardless.
 

Bald Rick

Veteran Member
Joined
28 Sep 2010
Messages
19,549
Hasn't it even been moved to a IECC?!

Wasn't it resignalled in the IECC era?

South Central side controlled by Victoria ASC since the early 80s which is at, err Clapham (although soon to be resignalled and relocated to TB ROC)

South Western side controlled from Wimbledon ASC since 1989, although the interlocking is at Clapham. No plans to move that, yet. It was just about the last non SSI big resignalling.

Both controlled by NX Panels, both have big Relay interlockings. Westinghouse for South Central side, I forget on the SW.
 

MAV39

Member
Joined
10 May 2017
Messages
44
This is where a lack of failover and disaster recovery tests will bite you on the bum.

The classic data centre failure I came across in the past was where every test worked beautifully, but when a longer term power outage happened, the generators failed after several hours as no-one had arranged to refill the fuel tanks when live running.
 

MadMac

Member
Joined
13 Jun 2008
Messages
401
Location
Moorpark, CA
South Central side controlled by Victoria ASC since the early 80s which is at, err Clapham (although soon to be resignalled and relocated to TB ROC)

South Western side controlled from Wimbledon ASC since 1989, although the interlocking is at Clapham. No plans to move that, yet. It was just about the last non SSI big resignalling.

Both controlled by NX Panels, both have big Relay interlockings. Westinghouse for South Central side, I forget on the SW.
Not wishing to get too far OT, but I recall that Victoria was GEC, as it was then known, as they were robbing stuff (particularly FDM equipment) from there for Edinburgh.

The classic data centre failure I came across in the past was where every test worked beautifully, but when a longer term power outage happened, the generators failed after several hours as no-one had arranged to refill the fuel tanks when live running.
I recall that when the standby generator for Dundee SC was tested pre-opening, the CCTV picture for Camperdown barriers went haywire. We quickly deduced that the generator wasn’t quite putting out 50 Hertz and the timebase circuitry in the monitor couldn’t cope with what it was getting. Minor adjustment necessary
 
Last edited:

Surreytraveller

Established Member
Joined
21 Oct 2009
Messages
1,813
There are, now, very few control locations that have ‘emergency’ panels at other . In any event they were rarely at a major station.

And ROCs could, in theory, back each other up. But there are three crucial points:

1) where do you put the spare control equipment ? Does each ROC have to be twice the size for the blue moon occasion?
2) who controls the spare equipment? Does each ROC have to have twice the number of signallers for the blue moon occasion?
3) if you have lost power to the interlockings (as today, apparently) it doesn’t matter where you control it from, trains aren’t going to move.
There's also maintaining the competence of the staff in each area
 

rd749249

Member
Joined
15 Sep 2015
Messages
115
This situation at the Romford ROC has halted Crossrail SIDT. Today, the only movement was the removal of a unit by a rescue train from Bond Street that had been there since the power was switched off yesterday. Testing tomorrow has been cancelled and Thursday is in doubt too.
 

mikeg

Established Member
Joined
20 Apr 2010
Messages
1,362
Reminds me of that time York ROC was hit by lightning whilst I was on my way to work. Cue 2 hours stuck on the train,a reverse at Tollerton and being taken back to Thirsk. It got me thinking too, would smaller overlapping ROCs be the solution with one able to cover for another? Would that work and if so how?
 

Horizon22

Established Member
Associate Staff
Jobs & Careers
Joined
8 Sep 2019
Messages
2,697
Location
London
The level of centralisation is perhaps going a bit far these days - on the railways it's ROCs, in the general business world it's datacentres (e.g. Microsoft's three in the UK). If something were to happen which took one or more of these out for an extended period the disruption would be considerable.

On the control side, having controllers from different TOCs together and NR personnel in once place in an integrated centre is definitely a plus. I thought ROCs were meant to have UPS (uninterrupted power supplies) though?
 

zwk500

Established Member
Joined
20 Jan 2020
Messages
2,273
Location
Milton Keynes
There is also the problem that it doesn't take all signals being out to collapse the service in the area.

Imagine the signalling system responsible for Clapham Junction goes out.
It doesn't matter if the signalling systems on the attached lines worked, very rapidly you will have total gridlock and half the former Southern Region is paralysed regardless.
A considerably bigger problem was caused about 2 years ago at East Croydon when the power tripped for Windmill Bridge Junction. It was, to put it mildly, an almighty task to even get a basic service moving again. But move it did, that night, and once it begun running the TRC & Signalling staff were able to devise and implement a recovery plan that allowed an almost full AM peak the following morning.
 

Horizon22

Established Member
Associate Staff
Jobs & Careers
Joined
8 Sep 2019
Messages
2,697
Location
London
A considerably bigger problem was caused about 2 years ago at East Croydon when the power tripped for Windmill Bridge Junction. It was, to put it mildly, an almighty task to even get a basic service moving again. But move it did, that night, and once it begun running the TRC & Signalling staff were able to devise and implement a recovery plan that allowed an almost full AM peak the following morning.

Was this also the one that shut down Victoria completely in the peak?
 

zwk500

Established Member
Joined
20 Jan 2020
Messages
2,273
Location
Milton Keynes
Was this also the one that shut down Victoria completely in the peak?
It may have been, there was also a complete panel failure at Streatham Common within 6 months of the windmill bridge incident. Streatham common was late morning into the afternoon, Windmill bridge happened about 5pm in winter, I think.
 

Ashley Hill

Established Member
Joined
8 Dec 2019
Messages
1,222
Location
The West Country
There is also the issue of fires and alarms. Should a fire alarm go off then the building would be evacuated pending investigation. What happens to the trains? What if it were a real fire,how easily repairable/replaceable is a ROC?
 
Status
Not open for further replies.

Top