• Our booking engine at tickets.railforums.co.uk (powered by TrainSplit) helps support the running of the forum with every ticket purchase! Find out more and ask any questions/give us feedback in this thread!

ECML/MML major power problems (09/08)

Status
Not open for further replies.

cactustwirly

Established Member
Joined
10 Apr 2013
Messages
7,447
Location
UK
It is also pertinent to remind ourselves that the inferior Siemens product was foisted upon us by the Department of Transport, at the expense of the Bombardier bid to build the rolling stock which was rejected.

That's because Siemens put forward a better product, power cuts aside, the 700s are far superior to the 710s
 
Sponsor Post - registered members do not see these adverts; click here to register, or click here to log in
R

RailUK Forums

roadie

Member
Joined
31 Dec 2009
Messages
16
Location
Wokingham
It is also pertinent to remind ourselves that the inferior Siemens product was foisted upon us by the Department of Transport, at the expense of the Bombardier bid to build the rolling stock which was rejected.
Bombardier is not necessarily a bed of rose petals, a Canadian company that has supplied its own citizens in Toronto with a trams that 80% have gone back for structural repairs before they had all entered service.
 

hwl

Established Member
Joined
5 Feb 2012
Messages
7,354
Bombardier is not necessarily a bed of rose petals, a Canadian company that has supplied its own citizens in Toronto with a trams that 80% have gone back for structural repairs before they had all entered service.
The quality of everything coming out of their Mexican factory is appalling, including several thousand the EMD SD70s over 2+ decades
 

DerekC

Established Member
Joined
26 Oct 2015
Messages
2,107
Location
Hampshire (nearly a Hog)
I'd better correct this one before anyone accuses me of being unkind to Siemens.

There was probably no requirement for them to make the trains more tolerant. No doubt they designed the software to a rigorous specification - no more, no less - so that if there were a change in requirements, they could make more money out of us.

However, I presume all AC traction is synchronised with the Grid (I'm sure someone will correct me if I'm wrong). If that's the case, it's interesting that the Bombardier 710s, despite all their earlier software problems, carried on running throughout this event.

My guess is that this is a problem with the kit which detects the harmonics generated by the train drive system and shuts everything down if they exceed the limits set for safety of the signalling system. There have been troubles with this before - as I recall, on Electrostars (before anyone orders bouquets for Bombardier). It seems plausible to me that a big excursion of the AC mains frequency somehow caused this kit to trip, although the AC input is transformed down and rectified to DC on board so you might have thought this would provide some isolation. The fact that it affected all the 700s at once points to this sort of issue.
 

Jimini

Established Member
Joined
8 Oct 2006
Messages
1,387
Location
London
I read somewhere else that it was an issue with the reboot sequence taking place before the OHLE was re-energised. If they don't detect any power after 90 seconds then the pans drop again. Do that three times and the on board systems lock themselves out, thus requiring the presence of a of a technician with laptop to physically attend each unit.
 

AM9

Veteran Member
Joined
13 May 2014
Messages
14,191
Location
St Albans
I'd better correct this one before anyone accuses me of being unkind to Siemens.



There was probably no requirement for them to make the trains more tolerant. No doubt they designed the software to a rigorous specification - no more, no less - so that if there were a change in requirements, they could make more money out of us.

However, I presume all AC traction is synchronised with the Grid (I'm sure someone will correct me if I'm wrong). If that's the case, it's interesting that the Bombardier 710s, despite all their earlier software problems, carried on running throughout this event.
Much of your post is just wrong. The trains like any new trains, would have been procured against a published requirement including performance, safety and other parameters. Apart from the obvious performance requirements, (speed, capacity, passenger environment etc.), there would be a whole plethora of legal/safety issues defined such as operating with defined signal systems, power supplies, failure detection and recovery, etc.. Many of these requirements conflict with each other to the extent that one can be achieved at the expense of the other, e.g., power efficiency in contemporary EMU designs tends to require an ac conversion to a DC bus from which a three-phase inverter feeds power efficient ac induction motors. This involves a lot of high-frequency high-power switching electronics that unless carefully designed, would seriously interfere with other systems both on the train and trackside. Sometimes, the careful design techniques required not only increase weight and physical servicing accessibility but also energy efficiency of the traction system. However, interoperability of equipment (including equipment not under the control of the railway is mandated under Electro-Magnetic Compatibility legislation which uniformly applies across the EU specifically and in some form or another all of the developed world.
As has been mentioned in other posts in this thread, this power supply problem was different to any of the previous major failures in that the 50Hz frequency varied outside established norms and varied at greater rates. If as I believe, the relaxation in the supply specification was not included in the specification for the trains, it wouldn't have been tested for during design qualification. If so, then the reviews that have been kicked-off will identify that, and presumably a) recommend a suitable design update to deal with it and b) revise the process of creating technical requirements to prevent a repeat of something similar.

As far as ac traction being synchronised with the grid, no, that is not the case. As I said above contemporary EMU designs tend to have a DC bus, conveniently at or around 750VDC. The 3 phase power to the traction motors is variable with the speed at which they rotate. This has been true since the need for all new EMU designs to be potentially suitable for either OLE and/or 3rd rail networks. Thus (I believe), all Electrostars, all Desiros, all Aventras, the class 395s and probably the new stock currently on order, (e.g. CAF 331s, Stadler etc.), have a DC to 3 phase ac traction inverter fed from a 750VDC bus. During manufacture, they can be fitted with pantographs, transformers and rectifiers for OLE operation and/or pick-up shoes for 3rd rail operation. Some types have both, e.g. class 377/2, /5, /7; class 387, class 350/1, class 700 and class 717.
It's also worth mentioning that 1980's designed class 319s also have a 750VDC bus, because of the Thameslink route's dual power systems although they have DC motors.
 

AM9

Veteran Member
Joined
13 May 2014
Messages
14,191
Location
St Albans
My guess is that this is a problem with the kit which detects the harmonics generated by the train drive system and shuts everything down if they exceed the limits set for safety of the signalling system. There have been troubles with this before - as I recall, on Electrostars (before anyone orders bouquets for Bombardier). It seems plausible to me that a big excursion of the AC mains frequency somehow caused this kit to trip, although the AC input is transformed down and rectified to DC on board so you might have thought this would provide some isolation. The fact that it affected all the 700s at once points to this sort of issue.
Yes the ac from the OLE is transformed and then rectified to DC but that DC would have a 2 x ac frequency ripple. This nominal 100Hz ripple, owing to it's high current, and non-siusoidal waveform create a harmonic spectrum that would be managed on the train by EMC design measures. Those measures would include monitors and protection circuits that would guard against other systems being affected, (signal/comms etc.). As the class 700s have a 'space ship' warble when under power from 3rd rail, I imagine that this is to spread the spectrum of these harmonics to reduce the potential for interference. They do not have the same modulation whilst running under OLE, so I assume that the ripple provides sufficient spreading of the harmonic spectrum without delibarate modulation. As the 750VDC is derived from a three phase National Grid feed, it would have a mjuch lower level of ripple (at 6 x 50Hz i.e. 300Hz, which would considerably reduce the harmonics issues).
 

paul1609

Established Member
Joined
28 Jan 2006
Messages
7,190
Location
Wittersham Kent
Much of your post is just wrong. The trains like any new trains, would have been procured against a published requirement including performance, safety and other parameters. Apart from the obvious performance requirements, (speed, capacity, passenger environment etc.), there would be a whole plethora of legal/safety issues defined such as operating with defined signal systems, power supplies, failure detection and recovery, etc.. Many of these requirements conflict with each other to the extent that one can be achieved at the expense of the other, e.g., power efficiency in contemporary EMU designs tends to require an ac conversion to a DC bus from which a three-phase inverter feeds power efficient ac induction motors. This involves a lot of high-frequency high-power switching electronics that unless carefully designed, would seriously interfere with other systems both on the train and trackside. Sometimes, the careful design techniques required not only increase weight and physical servicing accessibility but also energy efficiency of the traction system. However, interoperability of equipment (including equipment not under the control of the railway is mandated under Electro-Magnetic Compatibility legislation which uniformly applies across the EU specifically and in some form or another all of the developed world.
As has been mentioned in other posts in this thread, this power supply problem was different to any of the previous major failures in that the 50Hz frequency varied outside established norms and varied at greater rates. If as I believe, the relaxation in the supply specification was not included in the specification for the trains, it wouldn't have been tested for during design qualification. If so, then the reviews that have been kicked-off will identify that, and presumably a) recommend a suitable design update to deal with it and b) revise the process of creating technical requirements to prevent a repeat of something similar.

As far as ac traction being synchronised with the grid, no, that is not the case. As I said above contemporary EMU designs tend to have a DC bus, conveniently at or around 750VDC. The 3 phase power to the traction motors is variable with the speed at which they rotate. This has been true since the need for all new EMU designs to be potentially suitable for either OLE and/or 3rd rail networks. Thus (I believe), all Electrostars, all Desiros, all Aventras, the class 395s and probably the new stock currently on order, (e.g. CAF 331s, Stadler etc.), have a DC to 3 phase ac traction inverter fed from a 750VDC bus. During manufacture, they can be fitted with pantographs, transformers and rectifiers for OLE operation and/or pick-up shoes for 3rd rail operation. Some types have both, e.g. class 377/2, /5, /7; class 387, class 350/1, class 700 and class 717.
It's also worth mentioning that 1980's designed class 319s also have a 750VDC bus, because of the Thameslink route's dual power systems although they have DC motors.
and 375/6s
 

hwl

Established Member
Joined
5 Feb 2012
Messages
7,354
Yes the ac from the OLE is transformed and then rectified to DC but that DC would have a 2 x ac frequency ripple. This nominal 100Hz ripple, owing to it's high current, and non-siusoidal waveform create a harmonic spectrum that would be managed on the train by EMC design measures. Those measures would include monitors and protection circuits that would guard against other systems being affected, (signal/comms etc.). As the class 700s have a 'space ship' warble when under power from 3rd rail, I imagine that this is to spread the spectrum of these harmonics to reduce the potential for interference. They do not have the same modulation whilst running under OLE, so I assume that the ripple provides sufficient spreading of the harmonic spectrum without delibarate modulation. As the 750VDC is derived from a three phase National Grid feed, it would have a much lower level of ripple (at 6 x 50Hz i.e. 300Hz, which would considerably reduce the harmonics issues).
Most DC supplies for 3rd rail are now 12 pulse rectifiers to further reduce lower order harmonics - the 12 pulse rectifier consists of 2x 6 pulse rectifiers (you describe above a nominal 6 pulse design above) in parallel but connected to different secondary windings with one star and one delta configuration producing a 30 degree phase shift (i.e. 1/12th) between the 2 sets of 6 pulses so a more complex 300/600/1200Hz picture with much lower harmonics and EMC issues.
 

hwl

Established Member
Joined
5 Feb 2012
Messages
7,354
My guess is that this is a problem with the kit which detects the harmonics generated by the train drive system and shuts everything down if they exceed the limits set for safety of the signalling system. There have been troubles with this before - as I recall, on Electrostars (before anyone orders bouquets for Bombardier). It seems plausible to me that a big excursion of the AC mains frequency somehow caused this kit to trip, although the AC input is transformed down and rectified to DC on board so you might have thought this would provide some isolation. The fact that it affected all the 700s at once points to this sort of issue.
There are 2 sets of harmonic generation / interference / EMC to worry about:
a) rectification of AC in the 4 quadrant converter (where Friday's fun and games seemingly partly originated)
b) DC link to variable frequency AC conversion which is where the late 1990s electrostar issues were (or rather weren't) as the issues mostly lay with new IGBT switching tech at much higher frequencies than Railtrack was used to so the issues were largely to do with it being new and needing a through investigation by the inquisition rather than real problems.
 
Last edited:

AlexNL

Established Member
Joined
19 Dec 2014
Messages
1,683
Isn't it the case that the 700s are paid for on a per-diagram basis (like the 800s) which means that if a train is not available, XLT don't get paid for it?

This could be why Siemens having built-in overly strict protections and lockout mechanisms, which was then inherited by the 707s and 717s as they're based on the same platform.
 

fkofilee

Member
Joined
19 Jul 2015
Messages
164
QQ for the forum - Before Friday (Which i was affected badly by) - When was the last time CSL2 Black was triggered for GTR?

It was triggered half hour after this incident occurred... But ive checked out the CSL Guide and its quite significant to need to be triggered...
 

edwin_m

Veteran Member
Joined
21 Apr 2013
Messages
24,793
Location
Nottingham
I read somewhere else that it was an issue with the reboot sequence taking place before the OHLE was re-energised. If they don't detect any power after 90 seconds then the pans drop again. Do that three times and the on board systems lock themselves out, thus requiring the presence of a of a technician with laptop to physically attend each unit.
I wonder if the problem was that at the same time as the drivers were re-booting the trains, the National Grid was still trying to resolve the problem so the mains frequency might still have been out of spec, causing the reboot to fail. I read somewhere (maybe further back on here) that after a certain number of reboots by the driver the system locks out and has to be attended to by the technician. It sounds like nobody really foresaw that the mains might go that far off its expected frequency, so didn't think through the consequences of the response to that condition.
 

jon0844

Veteran Member
Joined
1 Feb 2009
Messages
28,013
Location
UK
If that turns out to be the case I'm sure it will be quite easy to avoid should we have a repeat incident. Allowing more reboot attempts and waiting longer between reboots.

Also perhaps allowing the cab screens to show more diagnostic information, so if the train knows the power source is incorrect it displays that.

Having a train that relies almost exclusively on computers does at least make it easier to fix certain things without mechanical changes.
 

ashkeba

Established Member
Joined
13 May 2019
Messages
2,171
I wonder if German designers expect the mains supply to stay within spec? That's not entirely a facetious comment, by the way.
I will not be entirely surprised if any investigation concludes that only the Siemens stock reacted to the problem as required in the train specifications while the Bombardier, Hitachi and BREL/ABB units ignored a potentially dangerous problem!
 

AM9

Veteran Member
Joined
13 May 2014
Messages
14,191
Location
St Albans
QQ for the forum - Before Friday (Which i was affected badly by) - When was the last time CSL2 Black was triggered for GTR?

It was triggered half hour after this incident occurred... But ive checked out the CSL Guide and its quite significant to need to be triggered...
Could you please explain what QQ, CSL2 Black and CSL mean?
 

tsr

Established Member
Joined
15 Nov 2011
Messages
7,400
Location
Between the parallel lines
QQ for the forum - Before Friday (Which i was affected badly by) - When was the last time CSL2 Black was triggered for GTR?

It was triggered half hour after this incident occurred... But ive checked out the CSL Guide and its quite significant to need to be triggered...

I believe it was last done by GTR at least some months ago; perhaps more than a year ago. However, Code Black is declared almost every week by some TOCs, particularly LNER and their predecessors, and GTR used it more often at one point - indeed, I remember one day with two GTR Code Black events about 3 years ago.

The CSL/PIDD guidance is nice to have, but in reality instinct takes over. It varies as to whose instinct is relevant. Code Black is generally authorised at a higher level than Red and Yellow, and implies that people really shouldn’t even consider travelling on the affected network, as nothing will be moving for some time. As well as the impact on travel, this does have fairly far-reaching commercial and PR implications as well.

Some TOCs, such as Virgin, tend to have more nuanced colour bands reflecting grades of severity for PIDD messages, and the associated levels of advice have almost superseded the whole “it’s all fallen over... don’t go anywhere” thing.
 

AM9

Veteran Member
Joined
13 May 2014
Messages
14,191
Location
St Albans
Most DC supplies for 3rd rail are now 12 pulse rectifiers to further reduce lower order harmonics - the 12 pulse rectifier consists of 2x 6 pulse rectifiers (you describe above a nominal 6 pulse design above) in parallel but connected to different secondary windings with one star and one delta configuration producing a 30 degree phase shift (i.e. 1/12th) between the 2 sets of 6 pulses so a more complex 300/600/1200Hz picture with much lower harmonics and EMC issues.
Ah yes I'd forgotten that they would have a star and delta transformer/rectifier configurationgiving a 600Hz component. That would be high enough to be almost smooth. In addition, the presence of trains with regen feeding back DC from the 3 phase motors asychronously, would further remove any 50Hz related components.
 

AM9

Veteran Member
Joined
13 May 2014
Messages
14,191
Location
St Albans
I wonder if German designers expect the mains supply to stay within spec? That's not entirely a facetious comment, by the way.
Most electric trains in Germany are 15kV 16.7Hz fed by a DB-only distribution system. They are not dependant on the exact 50Hz grid frequency.
 

Bayum

Established Member
Joined
21 Mar 2008
Messages
2,902
Location
Leeds
I will not be entirely surprised if any investigation concludes that only the Siemens stock reacted to the problem as required in the train specifications while the Bombardier, Hitachi and BREL/ABB units ignored a potentially dangerous problem!
Were the 800s not stopped BY the problem, or were they stopped as a result of the congestion?
 

yorkie

Forum Staff
Staff Member
Administrator
Joined
6 Jun 2005
Messages
67,439
Location
Yorkshire
Were the 800s not stopped BY the problem, or were they stopped as a result of the congestion?
A member of GTR staff on my train told me that Azumas were affected by the problem. However, if that was the case, it was presumably straightforward to reset them; my next train was an Azuma and it was pretty much on time.
 

HSTEd

Veteran Member
Joined
14 Jul 2011
Messages
16,636
Ah yes I'd forgotten that they would have a star and delta transformer/rectifier configurationgiving a 600Hz component. That would be high enough to be almost smooth. In addition, the presence of trains with regen feeding back DC from the 3 phase motors asychronously, would further remove any 50Hz related components.

Its also likely that the grid frequency spec is tighter on the modern trains because they have fewer tolerances to optimise the weight and size of the transformer.
If its a static substation you can use equipment with far bigger margins than anything you have to be able to move.
 

Fred26

Member
Joined
5 Mar 2010
Messages
1,107
Could you please explain what QQ, CSL2 Black and CSL mean?

QQ - Quick Question.

CSL2 - It's the level of disruption. I'm not entirely sure what it stands for though. Customer Service Level 2, anyone?! (Tongue in cheek).

Black Status - Service is so bad you shouldn't even consider travelling. For frontline staff it usually means all hands on deck. Indeed, there were several 'call-to-arms' made on Friday for any staff available to attend their local station. It wasn't compulsory, but many staff did go in.
 
Joined
7 Jan 2009
Messages
859
So we should certainly thank all staff that turned out to help resolve what proved in practice a most difficult situation, even by GTR standards.

Separately, it seems that the 2nd grid failure (Hornsea) was at a bank of inverters converting wind power-derived DC to AC and there is some discussion that this also was due to underfrequency (specifically, the rate of drop of frequency) following the first fault (what is the chance of two separate large-scale faults happening within 5 mins of each other...). (Underfrequency protection at Hornsea is not something that National Grid directly controls because this power source is connected to the local network (Northern Powergrid) rather than the NGC network.) In other words, Hornsea potentially experienced a similar problem to the C700s...
 

youngpete

Member
Joined
14 Jun 2018
Messages
36
When the generation at Little Barford was lost the network frequency fell and at a number of major bulk supply points between National Grid and the Regional Electricity companies (RECs) the Low Frequency protection correctly kicked in and disconnected supplies to preserve the stability of the National Grid. In the former SEEBOARD area it was the practice that the traction supplies to the former Southern Region were NOT disconnected in such circumstances. Unfortunately what has not become clear is if supplies were actually lost to the ECML and MML catenary systems and if so why. Or was it a case that the trains could just not deal with the frequency blip and shut down.
 
Status
Not open for further replies.

Top