ECML/MML major power problems (09/08)

[Robert S]

The graph on page 13 of the Ofgem report indicates that the initial drop in frequency was 0.16 Hz/s for 2.5 seconds. While this was greater than the 0.125 Hz/s of the old standard, it wasn't that much greater and was well below the 1 Hz/s of the new standard. Page 22 of the Ofgem report states that GTR have advised that it was the reduction in frequency below 49 Hz which led to the failure of their trains. The frequency was only below 49 Hz for 33 seconds (from 16:53:38 to 16:54:11) so it seems unlikely that many drivers would have attempted multiple reboots during this period. As it has long been understood that frequencies on the National Grid could drop as low as 47 Hz for short periods and other trainsets were not affected, this suggests that GTR (or the suppliers of their trains) have failed to provide sufficient resilience to rare events which lie within the Grid code.

 

[hwl]

The graph on page 13 of the Ofgem report indicates that the initial drop in frequency was 0.16 Hz/s for 2.5 seconds. While this was greater than the 0.125 Hz/s of the old standard, it wasn't that much greater and was well below the 1 Hz/s of the new standard. Page 22 of the Ofgem report states that GTR have advised that it was the reduction in frequency below 49 Hz which led to the failure of their trains. The frequency was only below 49 Hz for 33 seconds (from 16:53:38 to 16:54:11) so it seems unlikely that many drivers would have attempted multiple reboots during this period. As it has long been understood that frequencies on the National Grid could drop as low as 47 Hz for short periods and other trainsets were not affected, this suggests that GTR (or the suppliers of their trains) have failed to provide sufficient resilience to rare events which lie within the Grid code.

Both the Energy TSI and the older UK group standard also specifies 47Hz...

The contract was signed when the group standard was in force before the TSI.

All questions should be directed to Siemens.

 

[AM9]

Both the Energy TSI and the older UK group standard also specifies 47Hz...

The contract was signed when the group standard was in force before the TSI.

All questions should be directed to Siemens.

Through whoever negotiated and placed the order of course, - presumably the DfT.

The graph on page 13 of the Ofgem report indicates that the initial drop in frequency was 0.16 Hz/s for 2.5 seconds. While this was greater than the 0.125 Hz/s of the old standard, it wasn't that much greater and was well below the 1 Hz/s of the new standard. Page 22 of the Ofgem report states that GTR have advised that it was the reduction in frequency below 49 Hz which led to the failure of their trains. The frequency was only below 49 Hz for 33 seconds (from 16:53:38 to 16:54:11) so it seems unlikely that many drivers would have attempted multiple reboots during this period. As it has long been understood that frequencies on the National Grid could drop as low as 47 Hz for short periods and other trainsets were not affected, this suggests that GTR (or the suppliers of their trains) have failed to provide sufficient resilience to rare events which lie within the Grid code.

If we are to continue to improve the energy efficiency of trains, it is necessay to pursue design obectives to reduce weight and increase transformer and other equipment tuning. Such measures may result in designing the equipment to operate on supplies that remain within the normal operating range, I.e. 49.5Hz to 50.5Hz, with appropriate self-protection against out-of-frequency range excursions. Such excursions are (in the UK) exceptional, and the ruHz figure is the frequency at which the ESO must disconnect supplies to prevent widespread damage. There is then a decision about whether to continue in designing and deploying rolling stock that has protection mechanisms that will kick in very rarely as a price to pay for higher energy efficiency. Is a the shutdown of trains during a greater than 1 in 10 year event a price worth paying? That may be established in the ensuing enquiries, but if that results in a regression to less efficient designs, not only in trains, but on equipmentmacross the board, a different view may be taken in 20 or 30 years time when we are faced with far more grave energy choices. It's unlikely to affect me either way but I fear for many young members of society.
As far as the recovery from the failures goes, I do feel that some remote authorisation may be adopted rather than a travelling technician with local computer equipment. Nevertheless, any such authorisation would have to include enhanced monitoring to ascertain the it would be safe to return operation after an unknown shutdown.

 

[Nicholas Lewis]

Through whoever negotiated and placed the order of course, - presumably the DfT.

If we are to continue to improve the energy efficiency of trains, it is necessay to pursue design obectives to reduce weight and increase transformer and other equipment tuning. Such measures may result in designing the equipment to operate on supplies that remain within the normal operating range, I.e. 49.5Hz to 50.5Hz, with appropriate self-protection against out-of-frequency range excursions. Such excursions are (in the UK) exceptional, and the ruHz figure is the frequency at which the ESO must disconnect supplies to prevent widespread damage. There is then a decision about whether to continue in designing and deploying rolling stock that has protection mechanisms that will kick in very rarely as a price to pay for higher energy efficiency. Is a the shutdown of trains during a greater than 1 in 10 year event a price worth paying? That may be established in the ensuing enquiries, but if that results in a regression to less efficient designs, not only in trains, but on equipmentmacross the board, a different view may be taken in 20 or 30 years time when we are faced with far more grave energy choices. It's unlikely to affect me either way but I fear for many young members of society.
As far as the recovery from the failures goes, I do feel that some remote authorisation may be adopted rather than a travelling technician with local computer equipment. Nevertheless, any such authorisation would have to include enhanced monitoring to ascertain the it would be safe to return operation after an unknown shutdown.

In all likelihood we will never get to hear the outcome as neither DoT nor ORR have referenced any concern to what happened unlike OFGEM and DoBEIS who are ensuring transparency, lessons to be learnt and a dialogue about whether we need to do something different with grid management even if it costs consumers more.
Disappointing as well is new SoS Grant Shapps represents Hatfield/Welwyn commuters so directly in the chaos but says nothing unlike Andrea Leadsom (SoS BEIS) whose in the mix. Hopefully the local rail user groups are on his case to get answers to deal with the wrong kind of Herz

 

[AM9]

In all likelihood we will never get to hear the outcome as neither DoT nor ORR have referenced any concern to what happened unlike OFGEM and DoBEIS who are ensuring transparency, lessons to be learnt and a dialogue about whether we need to do something different with grid management even if it costs consumers more.
Disappointing as well is new SoS Grant Shapps represents Hatfield/Welwyn commuters so directly in the chaos but says nothing unlike Andrea Leadsom (SoS BEIS) whose in the mix. Hopefully the local rail user groups are on his case to get answers to deal with the wrong kind of Herz

I think you are right. The DfT have it seems managed to achieve what they set out to do, i.e. paint GTR as the bogeyman by creating a TOC role that had all the risk and paying the successful bidder to manage the (almost) impossible. So anything that happens to GTR services, however caused is now blamed on the TOC. The media is always keen to have a target for any invective that they create to sell news so, - job done! I've always felt that GTR has plenty of weaknesses, especially communicating with their customers, but they can't really be blamed for the issues on the trains themselves on the 9th August much as meny GTR haters here might like to.
ITYM wrong kind of Hertz.

 

[jellybaby]

GTR ... can't really be blamed for the issues on the trains themselves on the 9th August

Maybe the 700s but did the DfT have anything to do with the 717 order? I thought that was a GTR choice?

 

[AlexNL]

The 717 is a variation on the Class 700 design though, so if the 700 is designed to behave in the way DfT wants it to then it's very likely that the 717 does the exact same thing. If there had been AC versions of the 707, it would've been likely to see those grind to a halt as well.

 

[jellybaby]

The 717 is a variation on the Class 700 design though, so if the 700 is designed to behave in the way DfT wants it to then it's very likely that the 717 does the exact same thing.

That explains why they also failed. GTR could have ordered something entirely different for Moorgate, something like a 378 perhaps? I'm sure that would have been a silly choice since having two fleets of trains which are fairly similar is going to be easier all round but I think it was GTRs choice, not the DfTs.

 

[Greybeard33]

If we are to continue to improve the energy efficiency of trains, it is necessay to pursue design obectives to reduce weight and increase transformer and other equipment tuning. Such measures may result in designing the equipment to operate on supplies that remain within the normal operating range, I.e. 49.5Hz to 50.5Hz, with appropriate self-protection against out-of-frequency range excursions. Such excursions are (in the UK) exceptional, and the ruHz figure is the frequency at which the ESO must disconnect supplies to prevent widespread damage. There is then a decision about whether to continue in designing and deploying rolling stock that has protection mechanisms that will kick in very rarely as a price to pay for higher energy efficiency. Is a the shutdown of trains during a greater than 1 in 10 year event a price worth paying? That may be established in the ensuing enquiries, but if that results in a regression to less efficient designs, not only in trains, but on equipmentmacross the board, a different view may be taken in 20 or 30 years time when we are faced with far more grave energy choices. It's unlikely to affect me either way but I fear for many young members of society.
As far as the recovery from the failures goes, I do feel that some remote authorisation may be adopted rather than a travelling technician with local computer equipment. Nevertheless, any such authorisation would have to include enhanced monitoring to ascertain the it would be safe to return operation after an unknown shutdown.

A multi-megawatt transformer will not catch fire or explode if the frequency dips slightly below the design minimum for a few seconds. It has thermal inertia, just as synchronous generators have mechanical inertia. A well designed underfrequency protection algorithm will have a delay before trip that varies to take account of the magnitude and duration of the frequency excursion, relative to the transformer tolerance characteristic.

Is it established fact that the transformer on the 700/717 is actually significantly more efficient than those on any of the other classes of AC EMU, which took the 9th August incident in their stride, or is this just forum wibble?

 

[717001]

A multi-megawatt transformer will not catch fire or explode if the frequency dips slightly below the design minimum for a few seconds. It has thermal inertia, just as synchronous generators have mechanical inertia. A well designed underfrequency protection algorithm will have a delay before trip that varies to take account of the magnitude and duration of the frequency excursion, relative to the transformer tolerance characteristic.

Is it established fact that the transformer on the 700/717 is actually significantly more efficient than those on any of the other classes of AC EMU, which took the 9th August incident in their stride, or is this just forum wibble?

The same 700s work north and south of London but don't remember hearing of issues to the south, which suggests the electrical outages varied on different lines? Possibly any modern electrically powered train would have been hit on ECML / MML - rather than the 700s / 717s being uniquely impacted?

 

[jellybaby]

The same 700s work north and south of London but don't remember hearing of issues to the south, which suggests the electrical outages varied on different lines? Possibly any modern electrically powered train would have been hit on ECML / MML - rather than the 700s / 717s being uniquely impacted?

South of the river the 700s were powered by dragons^wDC. Meanwhile north of the river the IEPs were fine.

 

[Bald Rick]

There was a big power dip/surge just before 2000 tonight; the data suggests Dinorwic opened the gates quickly. Everything in my house went off and reset quickly. Even Alexa had a moment.

Any issues out there?

 

[w1bbl3]

Through whoever negotiated and placed the order of course, - presumably the DfT.

If we are to continue to improve the energy efficiency of trains, it is necessay to pursue design obectives to reduce weight and increase transformer and other equipment tuning. Such measures may result in designing the equipment to operate on supplies that remain within the normal operating range, I.e. 49.5Hz to 50.5Hz, with appropriate self-protection against out-of-frequency range excursions. Such excursions are (in the UK) exceptional, and the ruHz figure is the frequency at which the ESO must disconnect supplies to prevent widespread damage. There is then a decision about whether to continue in designing and deploying rolling stock that has protection mechanisms that will kick in very rarely as a price to pay for higher energy efficiency. Is a the shutdown of trains during a greater than 1 in 10 year event a price worth paying? That may be established in the ensuing enquiries, but if that results in a regression to less efficient designs, not only in trains, but on equipmentmacross the board, a different view may be taken in 20 or 30 years time when we are faced with far more grave energy choices. It's unlikely to affect me either way but I fear for many young members of society.
As far as the recovery from the failures goes, I do feel that some remote authorisation may be adopted rather than a travelling technician with local computer equipment. Nevertheless, any such authorisation would have to include enhanced monitoring to ascertain the it would be safe to return operation after an unknown shutdown.

I'd expect the issue to have been a controls / sensitivity one which will be down to design decisions taken by the builder. The efficiency of the transformer is a separate matter to that of the protection controls, which are probably software based and how they have been programmed to shutdown the train to protect the electronics then allow or disallow restarting. The fact that IEP's or for that matter 374's being an equally modern designs didn't all shutdown and require an engineer to visit suggests very much that Siemens need to have think about the protection logic as implemented. Equally and assuming Siemens haven't developed a completely bespoke transformer for the UK the shutdown event does raise questions on how the product had been tested as the German grid code is significantly less restrictive on frequency then our own being based on a normal range of 51Hz to 49Hz.

 

[hwl]

Is it established fact that the transformer on the 700/717 is actually significantly more efficient than those on any of the other classes of AC EMU, which took the 9th August incident in their stride, or is this just forum wibble?

It is, but the tolerances are lower and it is very very actively cooled compared to older designs. Ditto the Aventra design being bit different but not as much as leap as the Desiro City one.

 

[hwl]

The same 700s work north and south of London but don't remember hearing of issues to the south, which suggests the electrical outages varied on different lines? Possibly any modern electrically powered train would have been hit on ECML / MML - rather than the 700s / 717s being uniquely impacted?

South of the river is "DC" which theoretically has no frequency but no rectification is perfect so in reality it has a 600Hz asymmetric sawtooth ripple rather than being smooth. 2 DC substations (outside M25) did shut down temporarily with issues. The 700/717 was an AC OHLE issue only.

 

[apk55]

As well as electronic protection I would expect the transformer to still have traditional protection such as
overcurrent (would have a short delay to cope with switch on transients)
current balance - that is the current going into the primary is the same as that coming out as measured by the difference between two current transformers
cooling oil circulating pump proving relay
cooling oil temperature
gas detector relay (Bucholtz relay)

My own thoughts are that the electronic protection should just provide a soft shutdown ie a power supply out of spec warning but not a drop out the transformer unless it detected a potentially damaging situation for the transformer (ie high volts and low frequency). And it would automatically restore operation when the supply was back in spec without needing a reset.

 

[717001]

South of the river is "DC" which theoretically has no frequency but no rectification is perfect so in reality it has a 600Hz asymmetric sawtooth ripple rather than being smooth. 2 DC substations (outside M25) did shut down temporarily with issues. The 700/717 was an AC OHLE issue only.

Thanks. Rarely venture south of the river and not an expert - had forgotten that it had a different set-up.

 

[hwl]

As well as electronic protection I would expect the transformer to still have traditional protection such as
overcurrent (would have a short delay to cope with switch on transients)
current balance - that is the current going into the primary is the same as that coming out as measured by the difference between two current transformers
cooling oil circulating pump proving relay
cooling oil temperature
gas detector relay (Bucholtz relay)

My own thoughts are that the electronic protection should just provide a soft shutdown ie a power supply out of spec warning but not a drop out the transformer unless it detected a potentially damaging situation for the transformer (ie high volts and low frequency). And it would automatically restore operation when the supply was back in spec without needing a reset.

I think they have gone for several sets of belts and braces approach with protection.
With the new stock there are a large number of secondaries (6 in the case to a a 12 car 700 transformer) so plenty of logic needed for current balance. With frequency drop the thermal loading will increase rapidly so it look like frequency monitoring is is being used as a leading indicator rather than monitoring lagging ones.

 

[Greybeard33]

It is, but the tolerances are lower and it is very very actively cooled compared to older designs. Ditto the Aventra design being bit different but not as much as leap as the Desiro City one.

Hmm. The purpose of cooling is to remove waste heat. Higher efficiency should mean there is less heat to be removed! What sort of percentage improvement in efficiency is there over older transformer designs?

 

[edwin_m]

Hmm. The purpose of cooling is to remove waste heat. Higher efficiency should mean there is less heat to be removed! What sort of percentage improvement in efficiency is there over older transformer designs?

I think this means that the transformer is more efficient so needs less cooling, but when the frequency changes the efficiency drops and what cooling there is can't cope with the extra heat.

 

[AM9]

Hmm. The purpose of cooling is to remove waste heat. Higher efficiency should mean there is less heat to be removed! What sort of percentage improvement in efficiency is there over older transformer designs?

It's been suggested that the 700's transformers have minimum iron, thereby reducing weight, (and thermal inertia) to a minimum. At up to c. 2.5MW per transformer, if the transformers are a minimum weight and size design, it wouldn't take long for the internal temperatures to rise above safe working levels if not quickly checked. Whether such a rise would be preventing the reapplication of a supply to the primary because of thermal imbalances once the cooling was removed, I don't know. Repeated attempts to reconnect when the monitoring system might not have a clear picture of whether the transformer was operating withing it safe operating envelope, so under those conditions, a number of retries may have triggered a software lock-out.
With the benefit of hindsight, the inability of a local driver reset can be criticised, but the train's management system can only make decisions based on sensor information, allowing assertions of which scenario existed, and 'it's' historical 'experience' knowledge bas. That TMS knowledge base will surely be updated when the engineering investigation is complete.
In the grand scheme of unpredictable equipment failures, trains that can't be restarted by drivers pale into insignificance when compared to Boeing's recent lack of engineering foresight.

 

[hwl]

It's been suggested that the 700's transformers have minimum iron, thereby reducing weight, (and thermal inertia) to a minimum. At up to c. 2.5MW per transformer, if the transformers are a minimum weight and size design, it wouldn't take long for the internal temperatures to rise above safe working levels if not quickly checked. Whether such a rise would be preventing the reapplication of a supply to the primary because of thermal imbalances once the cooling was removed, I don't know. Repeated attempts to reconnect when the monitoring system might not have a clear picture of whether the transformer was operating withing it safe operating envelope, so under those conditions, a number of retries may have triggered a software lock-out.
With the benefit of hindsight, the inability of a local driver reset can be criticised, but the train's management system can only make decisions based on sensor information, allowing assertions of which scenario existed, and 'it's' historical 'experience' knowledge base. That TMS knowledge base will surely be updated when the engineering investigation is complete.
In the grand scheme of unpredictable equipment failures, trains that can't be restarted by drivers pale into insignificance when compared to Boeing's recent lack of engineering foresight.

One of the reasons to improve efficiency was to get more power through with less weight.
The previous generation of stock mostly used the same ABB-Secheron design of transformer, one of the reasons the previous generation was limited to 5 cars units (electrostar/ desiro) was the max power the transformer could supply. Hence if Siemens (or Bombardier) wanted to do 12x 20m EMUs with the older transformer they would have needed 3 transformers hence Thameslink Rolling Stock spec was wake up call for B & S to have rethink worldwide. Alstom had their own one the right size used in the Pendolino but it is very heavy and bulky and B/S wanted something smaller/lighter (Much better energy efficiency is now expected to win contracts).
With auxilliary loads you need about 2.75MW transformer capacity per 6 car half unit. Siemens opted to design their own new one in house and Bombardier went to a French firm (that is also supply for Talent 3 - UIC gauge equivalent to Aventra) Both with radically different design to previous ones and very configurable e.g. different transformer for 8/12car 700s

SNCF we also unhappy with the ABB-S design and have commission Mitsubishi to supply a prototype to replace them on existing stock.

This provided the impetus for ABB to come up with a new design (not oil cooled), that has been winning some custom from Bombardier (non UK) very recently.

The TMS software was apparently fixed that night (knee jerk fix).

 

[Tim P-H]

Interesting comments about the thermal ratings of the transformers on the Class 700. I do not have a rail background but have many years experience designing and testing electrical protection for electrical system for both the Distribution Operators and National Grid, plus generation.

The last thing you want to do is de-energise an overheating transformer which is fitted with force cooling as you will probably loose the auxiliary supply which powers the cooling systems! What you normally do is disconnect the load, in this case the traction, but keep the auxiliaries energised so you can cool and monitor the transformer. You could possible maintain house supplies as well, lighting and possibly air con for the train.

If overfluxing of the core is a problem due to low iron content, as over fluxing will cause harmonics and heating in the core, they must either by 'on the line' in the case of core design or the protection is far to sensitive or of the wrong type. The overfluxing voltage (knee point voltage) is directly proportional to frequency, so a 1% decrease in frequency lowers the knee point voltage by 1%. So overfluxing may be a problem when the line voltage is high and frequency lower. As low frequency is caused by grid overload this will also force the network voltage to droop as well, so helping the overfluxing problem!. The voltage controls are very slow, 120s to respond.

Someone needs to get their think hats on, plus look globely at how the grid responds to overload conditions, and design a protection system to suit.

 

[AM9]

One of the reasons to improve efficiency was to get more power through with less weight.
The previous generation of stock mostly used the same ABB-Secheron design of transformer, one of the reasons the previous generation was limited to 5 cars units (electrostar/ desiro) was the max power the transformer could supply. Hence if Siemens (or Bombardier) wanted to do 12x 20m EMUs with the older transformer they would have needed 3 transformers hence Thameslink Rolling Stock spec was wake up call for B & S to have rethink worldwide. Alstom had their own one the right size used in the Pendolino but it is very heavy and bulky and B/S wanted something smaller/lighter (Much better energy efficiency is now expected to win contracts).
With auxilliary loads you need about 2.75MW transformer capacity per 6 car half unit. Siemens opted to design their own new one in house and Bombardier went to a French firm (that is also supply for Talent 3 - UIC gauge equivalent to Aventra) Both with radically different design to previous ones and very configurable e.g. different transformer for 8/12car 700s

SNCF we also unhappy with the ABB-S design and have commission Mitsubishi to supply a prototype to replace them on existing stock.

This provided the impetus for ABB to come up with a new design (not oil cooled), that has been winning some custom from Bombardier (non UK) very recently.

The TMS software was apparently fixed that night (knee jerk fix).

Thanks for that, - very interesting.

 

[36270k]

There was a big power dip/surge just before 2000 tonight; the data suggests Dinorwic opened the gates quickly. Everything in my house went off and reset quickly. Even Alexa had a moment.

Any issues out there?

The time coincides with sunset when output from Solar is dropping rapidly and the wind was dropping as well.

 

[edwin_m]

The time coincides with sunset when output from Solar is dropping rapidly and the wind was dropping as well.

Any major televised event ending at around the same time (everyone puts the kettle on)?

 

[ac6000cw]

Out of interest, how many transformers does an 8/12 car class 700 have?

 

[hwl]

Out of interest, how many transformers does an 8/12 car class 700 have?

2 each is two half set with 1 transformer the 8 car with 4 secondary windings (each supplying a traction electronics box/ auxilliares) and 12 car with 6 secondary windings.
Previously a 12 car would have had 3 transformers and hence an extra set of everything (computer...) so lots of savings by just having 2.

 

[ac6000cw]

2 each is two half set with 1 transformer the 8 car with 4 secondary windings (each supplying a traction electronics box/ auxilliares) and 12 car with 6 secondary windings.
Previously a 12 car would have had 3 transformers and hence an extra set of everything (computer...) so lots of savings by just having 2.

Thanks for the info - presumably, as the train is two 'half sets' there is a reasonable amount of 'get you home' provision in the event of a problem with one half-set?

 

[hwl]

Thanks for the info - presumably, as the train is two 'half sets' there is a reasonable amount of 'get you home' provision in the event of a problem with one half-set?

Precisely, half an 8car set can rescue a dead 12car in addition to dragging its other dead half