ECML/MML major power problems (09/08)

[ashkeba]

Anyone been compensated yet?

 

[Nick Nation]

Anyone been compensated yet?

No! A friend put in a claim the next day and it still hasn't been processed.

 

[takno]

Anyone been compensated yet?

The last claim I put into LNER they took right up until the deadline to ask for more information, then another couple of weeks to agree the payment and a full week to then make the payment. And that wasn't immediately following a period of massive disruption.

 

[mrmatt]

Anyone been compensated yet?

Put in claim for 2 hours delay (was longer than that in reality) on the day after, was approved the day after that. I rarely have an issue with GTR and delay repay being approved.

 

[Hadders]

Anyone been compensated yet?

My GTR claim for a 2+ hour delay was paid in full last week.

GTR are pretty efficient at dealing with my delay repay claims (they've had enough of them mind!). Northern on the other hand...

 

[OwenB]

My delay repay has been approved and paid for 2+ hours. However, my claim for a £25 train fare to stay at a friend's place (cheaper than a hotel or a taxi home to Hatfield from London) has been approved in theory, and they have even asked for my full address to issue the cheque to. However, I've heard nothing since that email from 22nd August, I presumed they would issue the cheque on confirmation of my address.

 

[717001]

My delay repay has been approved and paid for 2+ hours. However, my claim for a £25 train fare to stay at a friend's place (cheaper than a hotel or a taxi home to Hatfield from London) has been approved in theory, and they have even asked for my full address to issue the cheque to. However, I've heard nothing since that email from 22nd August, I presumed they would issue the cheque on confirmation of my address.

My approval email, received this week for a similar amount, said I should receive the cheque by post in the next couple of weeks.

 

[SAPhil]

My delay repay claim was processed very quickly, however when another power failure happened on the 23rd people were complaining to a staff member behind the barriers at STP that their taxi claims from the 9th were being rejected and they wern't doing that again! He did say that he'd heard similar complaints before!

 

[OwenB]

So, despite the fact that they had agreed to issue a cheque for 25.00 (as mentioned before, I went to stay with a friend and was claiming for the price of the train ticket only, cheaper than any taxi home or hotel) and were asking my address to issue it to, I had an email from someone else in Customer Relations saying they had been referred the case and to confirm the amount my claim was for. I have no idea why this is happening given: a) I had already provided photos of the ticket (including receipt) and b) they had already agreed to issue a cheque for that amount.

Are they trying to weasel out of paying or do the left arm just not know what the right arm is doing? I'm so confused by their lack of process.

 

[43096]

Anyone been compensated yet?

Yes. My claim to EMT for that day has been paid (100% refund) despite the change of franchise.

 

[Nicholas Lewis]

GTR have inputted to the NG final report there feedback on why the class 700/717’s became stranded.

https://www.nationalgrideso.com/document/152351/download see p48

Basically the trains shouldn’t have locked out at the frequency excursion experienced but they are designed to self protect below 49Hz should have been 48.5Hz. The drivers should have been able to do a battery reset boot BUT circa 30 units were on a later software mod which had removed the ability of the driver to do this!!.

Siemens now reverting that mod in next software drop so drivers can self recover should the train go into protective shutdown.

No units suffered a loss of 25Kv

Apparently they had 17 technicians deployed as soon as drivers began reporting unable to do a reset and a further 24 were mobilised within an hour so a bit surprising it so long to clear all the trains but until GTR/NR, if they ever do, report back on incident management we wont know which trains were prioritised.

 

[Hadders]

Apparently they had 17 technicians deployed as soon as drivers began reporting unable to do a reset and a further 24 were mobilised within an hour so a bit surprising it so long to clear all the trains but until GTR/NR, if they ever do, report back on incident management we wont know which trains were prioritised.

This makes interesting reading. It could easily have taken the technicians 2+ hours to have reached the stranded trains given that the incident happened at the height of the Friday evening rush hour. Add to that the fact that some of the trains were in locations that were difficult to reach.

Also bear in mind that a train can only be moved if those in front had been restarted. Sod’s law would probably mean the train in front would be the last to be restarted, for whatever reason.

 

[Greybeard33]

In summary, then, 18000 delay minutes were caused by the combination of a specification non-compliance (4QC Permanent Lockout below 49.0Hz not 48.5Hz) and a software regression error (Permanent Lockout not driver recoverable by Battery Reset).

Nice one, Siemens!

 

[Surreytraveller]

In summary, then, 18000 delay minutes were caused by the combination of a specification non-compliance (4QC Permanent Lockout below 49.0Hz not 48.5Hz) and a software regression error (Permanent Lockout not driver recoverable by Battery Reset).

Nice one, Siemens!

Leave the software engineers alone. I was getting in trouble on the NRES thread whingeing about them breaking the NRES site!

 

[bahnause]

In summary, then, 18000 delay minutes were caused by the combination of a specification non-compliance (4QC Permanent Lockout below 49.0Hz not 48.5Hz) and a software regression error (Permanent Lockout not driver recoverable by Battery Reset).

Nice one, Siemens!

These software changes (permanent lockouts) are usually in agreement with the TOC. We had similar issues in our fleet. After complaining with the manufacturer, someone finally figured out the change was approved beforhand. But nobody remembered...

 

[Surreytraveller]

These software changes (permanent lockouts) are usually in agreement with the TOC. We had similar issues in our fleet. After complaining with the manufacturer, someone finally figured out the change was approved beforhand. But nobody remembered...

Trouble is, changes are agreed without realising the potential implications

 

[Greybeard33]

These software changes (permanent lockouts) are usually in agreement with the TOC. We had similar issues in our fleet. After complaining with the manufacturer, someone finally figured out the change was approved beforhand. But nobody remembered...

According to the GTR report linked in Post #371:

6. Importantly Siemens have also clarified that there should not have been a Permanent Lockout
on the train following a protective shutdown caused by a supply voltage frequency drop. All trains
should have been recoverable via Battery Reset whereas 30 trains were not recoverable. This was not
the intended behaviour of the train.
7. Therefore, the affected Class 700 and 717 sets did not react according to their design intent in
these circumstances. The risk of this happening was not known prior to the power event on
Friday 9 August.
8. Separately as a part of the new TCMS (Train Control Management System) software version 3.27, the
ability for the driver to recover from a Permanent Lockout by using the Battery Reset process was
removed.
9. On the 9th of August all the units which required a Technician to recover power were at software
level 3.27 or above. The 28 units recovered by the driver performing a Battery Reset were at the
previous TCMS software level of 3.25 or below.

And:

Planned Mitigation
1. Siemens are developing a software patch to allow units which protectively shutdown below 49Hz
supply frequency to recover themselves without the need of a reboot or laptop when the frequency rises
above 49.5Hz.
...
3. In addition to this Siemens will investigate how the train could be made to operate for a short time with
supply frequency falling to 48.5Hz.

 

[Nicholas Lewis]

ORR have published there report into the events of 09/08/19 aka the wrong kind of frequency

https://orr.gov.uk/__data/assets/pd...ay-power-disruption-on-2019-08-09-report.pdf?

Reasonable inciteful and fills in some of the gaps from the night as well as giving the full breakdown of impact on each route. Largely exonerates NR and explains Siemens rationale for the software change but doesn't discuss the impact of making this change and its potential operational consequences not being understood by the TOC and its drivers. Given the scale of software penetration into modern trains i would have though this would have been an opportunity to explore this event further.

 

[edwin_m]

There's also this story about electricity companies being fined - presumably the ORR report was held back to be released at the same time.

https://www.bbc.co.uk/news/business-50971717

RWE Generation, Orstead and UK Power Networks will pay into a redress fund run by the UK's energy watchdog, Ofgem.

Ofgem says it will continue to look into the role National Grid played.

 

[ashkeba]

Can passengers claim from the redress fund where the TOC has refused to pay the full cost of their last-minute hotel stay?

Notably, passengers heading for Cambridge were told at King's Cross that all terminals were effected for the rest of the day and indeed both King's Cross and St Pancras were but of course we know now that Liverpool Street services resumed later that evening making a hotel stay unnecessary strictly speaking although passengers had no way to know from information available at King's Cross at the time they had to decide before rooms sold out. A £26 Delay Repay refund barely dents the price paid for one of the last hotel rooms or the £90 taxi fare - if you could find a taxi willing to do it then for that!

 

[Sniffingmoose]

My son who was Kings Cross got home by black cab taxi to Cambridge that evening. It cost him £300. He said Uber were flcharging more. He felt there were no other options apart from a hotel. Luckily as he was in London for work, his employer paid.

 

[edwin_m]

ORR have published there report into the events of 09/08/19 aka the wrong kind of frequency

https://orr.gov.uk/__data/assets/pd...ay-power-disruption-on-2019-08-09-report.pdf?

Reasonable inciteful and fills in some of the gaps from the night as well as giving the full breakdown of impact on each route. Largely exonerates NR and explains Siemens rationale for the software change but doesn't discuss the impact of making this change and its potential operational consequences not being understood by the TOC and its drivers. Given the scale of software penetration into modern trains i would have though this would have been an opportunity to explore this event further.

There is the recommendation:

Siemens to check software to ensure all scenarios leading to permanent lockout are appropriate

Which suggests that the wider issues have been considered at least in respect of Siemens. Perhaps they should have included text like RAIB sometimes does, to the effect that this recommendation might also apply to other rolling stock manufacturers (although as the frequency excursion was nationwide it's safe to conclude that no other class had the exact same issue).

The report pretty much confirms the position we got to in the earlier discussions on this thread, with the root of the problem being guidance note in the standard, which effectively made it ambiguous. No doubt several lawyers are making a good living trying to establish who is to blame.

I think in fact the operational consequences would have been well understood by the TOC and drivers if someone had told them that in an unusual but credible situation that remained within the limits set by standards, every 700 and 717 on the AC network would shut down until visited by a technician with a laptop. The text discussing the implementation of the permanent lock-out doesn't mention GTR and seems to suggest this decision was internal to Siemens. Perhaps another of those cases where the people who actually work the trains aren't enough in the loop.

The report also states that the trains that were permanently locked out also lost auxiliary power, other than that available from the batteries. Having that happen to so many trains at once has the potential to create an stranding and self-evacuation similar to the ones RAIB has reported on at Kentish Town and Lewisham but on a larger scale. So I'm surprised RAIB hasn't shown any obvious interest in this incident.

 

[hwl]

There is the recommendation:

Which suggests that the wider issues have been considered at least in respect of Siemens. Perhaps they should have included text like RAIB sometimes does, to the effect that this recommendation might also apply to other rolling stock manufacturers (although as the frequency excursion was nationwide it's safe to conclude that no other class had the exact same issue).

The report pretty much confirms the position we got to in the earlier discussions on this thread, with the root of the problem being guidance note in the standard, which effectively made it ambiguous. No doubt several lawyers are making a good living trying to establish who is to blame.

I think in fact the operational consequences would have been well understood by the TOC and drivers if someone had told them that in an unusual but credible situation that remained within the limits set by standards, every 700 and 717 on the AC network would shut down until visited by a technician with a laptop. The text discussing the implementation of the permanent lock-out doesn't mention GTR and seems to suggest this decision was internal to Siemens. Perhaps another of those cases where the people who actually work the trains aren't enough in the loop.

The report also states that the trains that were permanently locked out also lost auxiliary power, other than that available from the batteries. Having that happen to so many trains at once has the potential to create an stranding and self-evacuation similar to the ones RAIB has reported on at Kentish Town and Lewisham but on a larger scale. So I'm surprised RAIB hasn't shown any obvious interest in this incident.

The standard also says that auxiliaries shouldn't start being disconnected till 48.5Hz...

Hence there should never have been an issue there in this case with loss of lights toilets or HVAC.

This isn't too useful a report the RAIB one is more useful. ORR can't even get what a 4QC is correct in the appendix.

 

[edwin_m]

The standard also says that auxiliaries shouldn't start being disconnected till 48.5Hz...

Hence there should never have been an issue there in this case with loss of lights toilets or HVAC.

This isn't too useful a report the RAIB one is more useful. ORR can't even get what a 4QC is correct in the appendix.

Interesting. Just checked back through the early posts and nobody seems to have said whether the aircon etc stayed on. Can anyone confirm?

I don't think the RAIB has reported on this incident, but reports on others are more detailed even when just a near-miss, presumably because they treat it as a safety issue.

 

[14xxDave]

The ORR can't even get what a 4QC is correct in the appendix.

What's wrong with the description?

 

[edwin_m]

What's wrong with the description?

It's a four quadrant converter, and it converts AC via DC into AC at a different amplitude and frequency. The definition of "Converter" is similarly wrong.

 

[hwl]

What's wrong with the description?

adding to Edwin's comments:
coloured bits = oops
ORR: "Four Quadrant Chopper – for converting a fixed DC input to a variable DC output. It is connected to the train control and communication network for specific operations."
It operates in 2 modes
a) converts the AC output of the transformers secondary @ ~50Hz to DC and feeds the DC link (the other potential external DC link feed is the 3rd rail). [the DC link then feeds the 3 phase variable voltage and frequency drives and the auxiliary static converters producing 415/3phase (HVAC, sockets, lighting etc. and 110V for control systems and battery charging)]
b) in regenerative braking mode it converts DC from the DC link to AC @ ~50Hz to put through the transformer to be fed back into the OHLE including matching the OHLE frequency.

so AC--> DC or DC-->AC but not DC-->DC in this case - one suspects wikipedia fail on ORR's part?

Mode a) is basically a rectifier with improved functionality (fewer problems), the useful bit is adding the ability to create AC wave forms in reverse and hence regenerative braking capability

 

[14xxDave]

It's a four quadrant converter, and it converts AC via DC into AC at a different amplitude and frequency. The definition of "Converter" is similarly wrong.

Ah now I see. As usual the electrical/electronics industry don't care about acronyms, the letters can be assigned to both terms.

 

[Greybeard33]

Largely exonerates NR and explains Siemens rationale for the software change but doesn't discuss the impact of making this change and its potential operational consequences not being understood by the TOC and its drivers. Given the scale of software penetration into modern trains i would have though this would have been an opportunity to explore this event further.

From the description of the rationale for the software change, it appears that the permanent lockout for low frequency was effectively a regression error in the 3.27.x software, albeit at the specification level rather than in the implementation:

In order to prevent the situation arising where a driver is unaware of a safety-related fault or out-of-specification condition and carries out a battery reset with the potential to worsen the situation, Siemens decided to implement a “permanent lock-out” in software version 3.27.x that could not be cleared by a battery reset. Such a permanent lock-out would require a technician to attend the train and perform an analysis of the causes of the lock-out before clearing it. Siemens identified a range of trigger conditions for the permanent lock-out, intended to be those conditions that could be made worse by clearing a lock-out that had been imposed. Among the conditions selected was the detection of a low power supply frequency.

The imposition of a lock-out when the supply frequency is outside of specified limits is understood to be a protection against the generation by the 4QC of electromagnetic interference in a range that can affect signalling circuits. Such protection is only required when the power supply is not in the specified frequency range, and therefore the lock-out could be permitted to automatically reset when the supply frequency returns to its nominal value. This appears likely to result in the minimum of disruption without increasing safety risks. However, power supply frequency excursion of the magnitude experienced are unusual, so the service-disrupting implications of imposing a lock-out requiring a driver or – as in this case – a technician to reset appear not to have been given weight when developing the protection parameters for the on-train software.

Most permanent lock-outs are triggered by events relating to the train itself, which are unlikely to arise simultaneously on multiple trains. Variations in the power supply frequency, however, affect many trains at the same time and result in the same response from all trains that have the same software. It appears therefore that the collective response of the Class 700 and 717 trains to the out-of-specification supply frequency was in accordance with the software design, but was not an explicit intention. Siemens accepts that the temporary reduction in frequency should not have been considered a situation that requires a permanent lock-out.

This does not reflect well on Siemens' change control processes.

Interesting. Just checked back through the early posts and nobody seems to have said whether the aircon etc stayed on. Can anyone confirm?

According to the report, the permanent lockout shut down the 4QC, effectively isolating the train from the AC line. The aircon and other auxiliaries are supplied from the 4QC via the 750V DC link as described by @hwl in post #387, so were inevitably lost:

When the 4QC shuts down, the train is isolated from the traction power supply. Available power is restricted to the limited on-board battery supply, which only supplies certain key on-board systems and does not provide power for traction.

 

[hwl]

From the description of the rationale for the software change, it appears that the permanent lockout for low frequency was effectively a regression error in the 3.27.x software, albeit at the specification level rather than in the implementation:


This does not reflect well on Siemens' change control processes.

According to the report, the permanent lockout shut down the 4QC, effectively isolating the train from the AC line. The aircon and other auxiliaries are supplied from the 4QC via the 750V DC link as described by @hwl in post #387, so were inevitably lost:

The appropriate software solution should have been software control what is happening further downstream by getting the 3phase VVSD traction electronics to shut down at 49.0Hz rather then the 4QC and then having the auxiliary static converters shut down at 48.5Hz. (Effectively what everyone else has implemented)

To trying to be simplistic /"clever" at the design stage. (CAF 4QC software also has issues for other reasons)

ORR are rapidly becoming aware (á la FAA post 737max) that not enough oversight of software was occurring with only Bombardier being fully open with them