ECML Significant Disruption

[Ferret]

Not cable theft at all, but a major power/systems failure in York IECC. I've had a narrow escape today given I just got through before the farce started.

 

[142094]

I've just been through York and didn't see much disruption - thing is I was on a train from Manchester going to Scarborough which wasn't delayed at all. Did see a GC HST sitting in Holgate Sidings and wondered what had happened.

Is it just me or are delays/cancellations seem to be getting more frequent? Seems as if everytime I look at the departure boards on any given day there are some delays, and problems like this seem to happen more often.

 

[ralphchadkirk]

Not cable theft at all, but a major power/systems failure in York IECC. I've had a narrow escape today given I just got through before the farce started.

Network Rail have said it was a software failure.

 

[YorkshireBear]

I've just been through York and didn't see much disruption - thing is I was on a train from Manchester going to Scarborough which wasn't delayed at all. Did see a GC HST sitting in Holgate Sidings and wondered what had happened.

Is it just me or are delays/cancellations seem to be getting more frequent? Seems as if everytime I look at the departure boards on any given day there are some delays, and problems like this seem to happen more often.

I personally think signal failues or similar happen far too often and serious steps should be made to stop it happening.

 

[142094]

I personally think signal failues or similar happen far too often and serious steps should be made to stop it happening.

I can understand that things like this happen occasionally and that you can't have a perfect railway, but it does seem to me that they are happening a lot more frequently since the beginning of the year. Perhaps that is just my perception, although I wonder if the bad weather we had earlier has made things worse.

 

[YorkshireBear]

I can understand that things like this happen occasionally and that you can't have a perfect railway, but it does seem to me that they are happening a lot more frequently since the beginning of the year. Perhaps that is just my perception, although I wonder if the bad weather we had earlier has made things worse.

i agree, i do accept that these things happen and usually not bothers me. But i do check service disruptions on national rail just cos im nosy and i feel i see it alot more and far too often now. (signal failure)

 

[DaveNewcastle]

There is (was) also a delayed 20:00 northbound departure, so hopefully many stranded pax are getting away, and ultimately home, from London tonight.

Southbound travellers appear to be less fortunate. Although there are some movements now, I'll guess that staff are well into overtime/out of hours. But for whatever logistical reason, its presumably unlikely that anything other than the Leeds 18:40 and 20:15 will get through tonight (picking up pax from the north at Donny where poss).

--- old post above --- --- new post below ---

Network Rail have said it was a software failure.

I personally think signal failues or similar happen far too often and serious steps should be made to stop it happening.

So which is it?

I'm advised that there has, indeed been a system fault which (it can be hard to isolate domains) appears to be within the computer procedures (i.e. software).
I also understand that faults in network software of any significance are very rare, affecting railway signalling. On the other hand, signalling failures, due to physical damage, decay, damage, error and other human interventions are much more frequent (though in the overall scale of mainland rail signalling are arguably not very significant).

Process Control Software in recent times is readily testable for its resilience, exception handling and robustness in the face of improbable circumstances. There are globally adopted standards for software verification which have long passed their development stages and have led to the adoption of very reliable and mature standards of verification.

I'd be very surprised if you can provide any evidence to show that NR signalling software fails "often". Very surprised indeed.
Can you?

 

[ralphchadkirk]

So which is it?

I'm advised that there has, indeed been a system fault which (it can be hard to isolate domains) appears to be within the procedures (i.e. software).
I also understand that faults of significance are very rare, within railway signalling. On the other hand, signalling failures, due to physical damage, decay, error and other human interventions are more frequent (though in the overall scale of mainland rail signalling are arguably not very significant).

Process Control Software in recent times is readily testable for resilience, exception handling and robustness in the face of improbable circumstances.
I'd be very surprised if you can provide any evidence to show that NR signalling software fails "often". Very surprised.
Can you?

They have specifically stated that it's a software failure. Makes sense, as York is an IECC. This must be a very rare event for a failure to happen in this way. But the passengers won't care whether the software has failed, or the signals. To them, it's a signal failure. So they think that they are becoming more common.

 

[142094]

They have specifically stated that it's a software failure. Makes sense, as York is an IECC. This must be a very rare event for a failure to happen in this way. But the passengers won't care whether the software has failed, or the signals. To them, it's a signal failure. So they think that they are becoming more common.

To most passengers a delay is a delay, doesn't matter what caused it. Sad as even rare delays such as what has happened today may put off a lot of people from travelling by rail ever again.

 

[caliwag]

I certainly recall, in the 90s, everything on ECML was blamed on points failure. I once spent several hours between KX and York watching coal trains trundle by, getting to York at 3.00 am ( a train that left KX at 8pm) blamed on points failure!! It's definitely better now, software/signalling...bah.

Radio 4 has just reported (9.00pm) that a train has been stationary for 6 hours...what can that possibly be about...?

 

[ralphchadkirk]

I certainly recall, in the 90s, everything on ECML was blamed on points failure. I once spent several hours between KX and York watching coal trains trundle by, getting to York at 3.00 am ( a train that left KX at 8pm) blamed on points failure!! It's definitely better now, software/signalling...bah.

Radio 4 has just reported (9.00pm) that a train has been stationary for 6 hours...what can that possibly be about...?

A serious software failure. That's what "it's" about.

 

[DaveNewcastle]

To most passengers a delay is a delay, doesn't matter what caused it. Sad as even rare delays such as what has happened today may put off a lot of people from travelling by rail ever again.

I quite agree with you. That is exactly how passengers will remember today's unfortunate disruption - as they do with all major incidents on our fragile network.
What bothered me, is that I'd expect more interest on THIS highly specialised forum in the ACTUAL causes, the logistical impact and the disruption management procedures which apply. But, what we did get on here, is an unfounded accusation of cable theft, a complaint that signals fail far too often, a proposal of homicide, a complaint about football, and a nice video of delayed trains.

I have a very, very great deal of sympathy for the affected EC passengers and have had contact with many today, and would wish to have assisted, and to demonstrate sympathy for the staff having to deal with such an unpredictable disruption to service (front line, back-office and on emergency call), and more than all of these, sympathy for the IT personnel who with no forewarning, have had to diagnose, reverse engineer the causal events, fix or bodge, and reinstate a functioning process, all on a day off work!

On here, I had hoped for a more informed technical debate than what we've got.

 

[caliwag]

Well IT'S very poor...Is this lack of investment, incompetance, or some other bl++dy excuse? The ECML has been running superbly for years and this is a huge setback.

Didn't they blame a software problem on the recent massive disruption in South Wales?

Come on you techie boys...get your heads together.

 

[142094]

I quite agree with you. That is exactly how passengers will remember today's unfortunate disruption - as they do with all major incidents on our fragile network.
What bothered me, is that I'd expect more interest on THIS highly specialised forum in the ACTUAL causes, impact and disruption management procedures which apply. But what we did get on here, is an unfounded accusation of cable theft, a complaint that signals fail far too often, a proposal of homicide, a complaint about football, and a nice video of delayed trains.
I have a very, very great deal of sympathy for the affected EC passengers and have had contact with many today, and would wish to have assisted, and sympathy for the staff having to deal with such an unpredictable disruption to service, and more than all of these, sympathy for the IT personnel who with no forewarning, have to diagnose, reverse engineer the causal events and reinstate a functioning process, all on a day off work!

On here, I had hoped for a more informed technical debate than what we've got.

Good point and well made there Dave. The main problem in this day and age of mobiles and wireless internet is that as soon as something happens, everyone will know about it within minutes. Of course on train X someone might be told it is due to points failure and on train Y it is a signalling failure. BBC News will come up with something different, then the Daily Mail will blame it on rising house prices or something along those lines.

Since it has been confirmed that it is a software issue - what can be done about it in future? Is there some sort of backup that can be used, or something else that could be done? Also does the closure of smaller local boxes and amalgamating everything into one large IECC pose a higher risk of this exact same problem happening again?

York IECC covers quite an area (also IIRC the same building has Leeds signalling as well), so perhaps putting all our signalling eggs in one basket was a wrong move?

 

[amcluesent]

>Come on you techie boys...get your heads together.<

I'd speculate there was -

1) A s/w upgrade to ready the system for 22 May Eureka timetable and it went pear shaped.

2) So many charter trains had been scheduled for the London demo that the s/w couldn't cope and crashed and the fail-safes tripped-in; hence NR insisting that no trains moved.

 

[Minilad]

Well IT'S very poor...Is this lack of investment, incompetance, or some other bl++dy excuse? The ECML has been running superbly for years and this is a huge setback.

Didn't they blame a software problem on the recent massive disruption in South Wales?

Come on you techie boys...get your heads together.

Why, when things like this happen and either a TOC or NR come out and give the reason why it happened people call it an excuse ?
It's not a bl++dy excuse, it's a reason

 

[CliveJones]

Well IT'S very poor...Is this lack of investment, incompetance, or some other bl++dy excuse?

Or maybe non of the above?

S**t happens. That's a part of life Why do people think that in this universe that its reasonable to expect nothing ever to go wrong ever?

The ECML has been running superbly for years and this is a huge setback.

So you are saying its generally been running great apart from this? See above.

Why, when things like this happen and either a TOC or NR come out and give the reason why it happened people call it an excuse ?
It's not a bl++dy excuse, it's a reason

I think people believe we just delay things for fun. Like the signal man is that chappy out of Airplane!, just pulling the plug out for lulz. Then we have to make 'excuses' to cover up the 'truth'. Because its not in the industry's best interests to have trains on time or anything like that.

 

[Liam]

When something like this happens what options do passengers who are not yet on their train have?Ccould someone travelling London - Newcastle take a VT service up to Carlisle and across from there? Edinburgh passengers should have been OK as London-Edinburgh changing at Preston or Carlisle (or via Reading and Manchester IIRC) is a permitted route, provided, of course, they weren't already on the train.

What about advance ticket holders?

I'm just glad I was on the 1600 from KX yesterday, not today.

 

[DaveNewcastle]

. . . .
Since it has been confirmed that it is a software issue - what can be done about it in future? Is there some sort of backup that can be used, or something else that could be done? Also does the closure of smaller local boxes and amalgamating everything into one large IECC pose a higher risk of this exact same problem happening again?

In terms of system design at a macro level, that is quite a profound question. Not amenable to quick off-the-cuff reponses.

There is a science of system-scale which can help with questions like this. (i.e. is it more robust to have a myriad of uncoordinated and self-sustsaining local systems forming an emergent system with complex properties, or to have a single over-arching system which is responsive to all localised data but is also culpable for every and any local fault?)
But there is little evidence (so far) to help determine whether today's very significant events on the ECML would favour isolated local line management or global network management.
For what its worth, my personal view is that the ECML, as it stands at present merits global control.
But for as long as the ECML hosts freight and local stopping services alongside the high speed long distance services on the same infrastructure, then I acknowledge good arguments for robust local control and intervention too.

But surely, and despite some of the comments above, we have to acknowledge that today's disruption is very very exceptional. No?

 

[silentone]

Travelling via the West Coast didn't seem to be an option on offer as I'm guessing with the Scotland game the route was already heavily loaded. The advice was not to travel unless you have to. Advance ticket holders are usually allowed to travel on the next available service when possible or use the ticket the next day, or use an alternative route that have approved passengers to use the route.

And just the throw my hat in I think technically today was a software failure, power failure, signal failure and points failure. If I'm following it correctly it was a power failure that prevented the software from operating anything on that stretch of route! They did manage to fix it briefly for around 15 minutes, of course when things are ongoing there is normally mixed information, it was being described as a signal failure to begin with which really covers a multitude of sins. And the official line is it was power related, anything else is going to be connected.

These signal failures happen far more often than you're even aware of, only the most serious disruption catches peoples attention.

 

[The Planner]

I'd speculate there was -

1) A s/w upgrade to ready the system for 22 May Eureka timetable and it went pear shaped.

2) So many charter trains had been scheduled for the London demo that the s/w couldn't cope and crashed and the fail-safes tripped-in; hence NR insisting that no trains moved.

Why on earth would you need an upgrade for a timetable change ?? or have too many trains running ??

As an aside, anyone got the TRUST incident number for this ?? bet the delay minutes are shooting up.

 

[lyndhurst25]

Network Rail have said it was a software failure.

Have they tried turning it off and then turning it back on again?

 

[First class]

Why on earth would you need an upgrade for a timetable change ?? or have too many trains running ??

As an aside, anyone got the TRUST incident number for this ?? bet the delay minutes are shooting up.

Can get the TRUST log when the control log is available at 05:15am

 

[YorkshireBear]

There is (was) also a delayed 20:00 northbound departure, so hopefully many stranded pax are getting away, and ultimately home, from London tonight.

Southbound travellers appear to be less fortunate. Although there are some movements now, I'll guess that staff are well into overtime/out of hours. But for whatever logistical reason, its presumably unlikely that anything other than the Leeds 18:40 and 20:15 will get through tonight (picking up pax from the north at Donny where poss).

--- old post above --- --- new post below ---


So which is it?

I'm advised that there has, indeed been a system fault which (it can be hard to isolate domains) appears to be within the computer procedures (i.e. software).
I also understand that faults in network software of any significance are very rare, affecting railway signalling. On the other hand, signalling failures, due to physical damage, decay, damage, error and other human interventions are much more frequent (though in the overall scale of mainland rail signalling are arguably not very significant).

Process Control Software in recent times is readily testable for its resilience, exception handling and robustness in the face of improbable circumstances. There are globally adopted standards for software verification which have long passed their development stages and have led to the adoption of very reliable and mature standards of verification.

I'd be very surprised if you can provide any evidence to show that NR signalling software fails "often". Very surprised indeed.
Can you?

I was not neccesarily reffering to this event hence the or similar part of my sentance. I was merely stating that i believe they happen often.
I am looking on a U.K wide scale and my only evidence is that i look at service disruptions daily on NR website and see at least 2 cases daily of signal failure.

 

[boing_uk]

What the travelling public won't understand is why trains were held at signals at all, when it is entirely within the wit of man (if not the rulebook) to move a train, such as the one in the video, to a convenient location at a speed at which it can stop if the line ahead is obstructed.

There is really no need, in this day and age, to delay a train between stations, for hours on end due to a failure like this.

I suppose the difficulty comes on whether NR can reasonably guarantee that pointwork will not move injudisciously during failure mode such as this.

 

[David Dunning]

What the travelling public won't understand is why trains were held at signals at all, when it is entirely within the wit of man (if not the rulebook) to move a train, such as the one in the video, to a convenient location at a speed at which it can stop if the line ahead is obstructed.

There is really no need, in this day and age, to delay a train between stations, for hours on end due to a failure like this.

I suppose the difficulty comes on whether NR can reasonably guarantee that pointwork will not move injudisciously during failure mode such as this.

and i think that could have been the problem at that location .The line at Tollerton has point work to allow swapping from fast to slow or slow to fast in both directions .

 

[Crossover]

What the travelling public won't understand is why trains were held at signals at all, when it is entirely within the wit of man (if not the rulebook) to move a train, such as the one in the video, to a convenient location at a speed at which it can stop if the line ahead is obstructed.

There is really no need, in this day and age, to delay a train between stations, for hours on end due to a failure like this.

I suppose the difficulty comes on whether NR can reasonably guarantee that pointwork will not move injudisciously during failure mode such as this.

My thought as well - was talking to a friend about this earlier (having seen said video) and as much as trains can be moved under danger if approval is given, I would have thought that all visibility had been lost of everything at track level and therefore, suspect they wouldn't be able to prove the points as a result.

Likewise, I guess the points could have been locked in place, but if such a large visibility of track had been lost then you only move the problem further down the line when you get to another set of points, especially when it gets complex.

(I don't work for the railway so not sure on viability of point locking etc, merely a little (hopefully vaguely educated) speculation)
--- old post above --- --- new post below ---

In terms of system design at a macro level, that is quite a profound question. Not amenable to quick off-the-cuff reponses.

There is a science of system-scale which can help with questions like this. (i.e. is it more robust to have a myriad of uncoordinated and self-sustsaining local systems forming an emergent system with complex properties, or to have a single over-arching system which is responsive to all localised data but is also culpable for every and any local fault?)
But there is little evidence (so far) to help determine whether today's very significant events on the ECML would favour isolated local line management or global network management.
For what its worth, my personal view is that the ECML, as it stands at present merits global control.
But for as long as the ECML hosts freight and local stopping services alongside the high speed long distance services on the same infrastructure, then I acknowledge good arguments for robust local control and intervention too.

But surely, and despite some of the comments above, we have to acknowledge that today's disruption is very very exceptional. No?

Thanks for your response to this DaveNewcastle and have to admit I'd not thought of the whole picture of centralised vs. non centralised control.

Slightly away from railways but I'm currently studying a degree in Computer Networking - and high availability is a key part of it, whilst still retaining something that is vaguely serviceable when it comes to troubleshooting! In some ways it bears a resemblence and even a well planned network can go belly up if hit with enough problems simultaneously.

I agree that from what I have heard of today at York, it is exceptional for failure on such a scale and with technology, to a degree we do have to accept that occasionally things aren't going to work quite as we want or expect to. Dare I go so far as to say that its what keeps some of us in a job?

 

[spoony]

Surely a software system that keeps a vital artery of the country open has a backup system? Even if it means that everything runs to a max speed of 30-40mph? Obviously not if today is anything to go by!

 

[Crossover]

Surely a software system that keeps a vital artery of the country open has a backup system? Even if it means that everything runs to a max speed of 30-40mph? Obviously not if today is anything to go by!

On the whole I'd agree that there should be backup systems, and I'm sure there are. However, if I have read into it right, it appears that it may have possibly been a number of different elements that failed in succession and backup systems can't cover every eventuality.

 

[Anvil1984]

Sorry to divert away from the techie stuff and just paint a picture from the staff side of things.

its been a really bizarre day from our point, at least one point between 1930 - 2000 (could have been more times but I was working a shift I'd be away from the depot) the queue for the replacement buses services was snaking from Eurocar next to platform 1 to the far end of the long stay car park snaking around by the train crew messrooms as well, none of us in the messroom have seen it that bad.

Signalling was lost between York and Danby Wiske which I believe is near Northallerton (don't sign north of York so can't confirm location), which meant Northern services on the Harrogate line were stuffed (starting / terminating at Poppleton), XC, EC,GC and TPE couldn't run north of York (barring the Scarboroughs which run to and from the east). It was bedlam

Services started to trickle in and out of York to the North as of about 2000 hours however a couple of services were still bustituted, the Northern 2017 service to Hull was worked by a guard who had actually finished his shift but stayed behind to help out (this service was originally cancelled for some reason).

A day to forget methinks