• Our booking engine at tickets.railforums.co.uk (powered by TrainSplit) helps support the running of the forum with every ticket purchase! Find out more and ask any questions/give us feedback in this thread!

ECML/MML major power problems (09/08)

Status
Not open for further replies.

43096

On Moderation
Joined
23 Nov 2015
Messages
15,160
So we should certainly thank all staff that turned out to help resolve what proved in practice a most difficult situation, even by GTR standards.
Not just GTR: I know of an EMT staff member who came in off annual leave to assist service recovery; they were there until gone 3am.

Full credit to EMT: they ran a full service on Saturday.
 
Sponsor Post - registered members do not see these adverts; click here to register, or click here to log in
R

RailUK Forums

Nicholas Lewis

Established Member
Joined
9 Aug 2019
Messages
5,996
Location
Surrey
I wonder if the problem was that at the same time as the drivers were re-booting the trains, the National Grid was still trying to resolve the problem so the mains frequency might still have been out of spec, causing the reboot to fail. I read somewhere (maybe further back on here) that after a certain number of reboots by the driver the system locks out and has to be attended to by the technician. It sounds like nobody really foresaw that the mains might go that far off its expected frequency, so didn't think through the consequences of the response to that condition.
Stystem frequency was back within spec within 3 mins of start of incident.
When the generation at Little Barford was lost the network frequency fell and at a number of major bulk supply points between National Grid and the Regional Electricity companies (RECs) the Low Frequency protection correctly kicked in and disconnected supplies to preserve the stability of the National Grid. In the former SEEBOARD area it was the practice that the traction supplies to the former Southern Region were NOT disconnected in such circumstances. Unfortunately what has not become clear is if supplies were actually lost to the ECML and MML catenary systems and if so why. Or was it a case that the trains could just not deal with the frequency blip and shut down.
According to NR news alert at the time A Network Rail spokeswoman said: “There was a power surge on the national grid this evening which means we lost power to all our signalling over a wide area, including the Newport, Gloucester, Ashford, Bristol, Eastbourne, Hastings, Three Bridges and Exeter areas". I can testify to Brighton ML losing power as on a train at the time but came back within a couple of minutes and rest of journey was uneventful.
Yes this specifically refers to signalling supplies which tend to be connected at local distribution level so are more likely to be affected by load shedding in discrete geographic areas however traction power is supplied at grid level so if its off it generally would be be associated with a much bigger geographic area with loss of power. So as ECML is fed from Wood Green and Welywn Feeder Stations at the south end you would expect to see associated reports of power out in these geographic areas but they don't crop up yet Southern England does . Appreciate this doesnt mean MML or ECML were affected but i believe not.
What is fact is the low system frequency would have been seen by every train with its pan up on UK network but only the 700 & 717's have sat down.
 

AndrewE

Established Member
Joined
9 Nov 2015
Messages
5,063
Stystem frequency was back within spec within 3 mins of start of incident.

According to NR news alert at the time A Network Rail spokeswoman said: “There was a power surge on the national grid this evening which means we lost power to all our signalling over a wide area, including the Newport, Gloucester, Ashford, Bristol, Eastbourne, Hastings, Three Bridges and Exeter areas". I can testify to Brighton ML losing power as on a train at the time but came back within a couple of minutes and rest of journey was uneventful.
Yes this specifically refers to signalling supplies which tend to be connected at local distribution level so are more likely to be affected by load shedding in discrete geographic areas however traction power is supplied at grid level so if its off it generally would be be associated with a much bigger geographic area with loss of power. So as ECML is fed from Wood Green and Welywn Feeder Stations at the south end you would expect to see associated reports of power out in these geographic areas but they don't crop up yet Southern England does . Appreciate this doesnt mean MML or ECML were affected but i believe not.
What is fact is the low system frequency would have been seen by every train with its pan up on UK network but only the 700 & 717's have sat down.
I wonder if "power surge" is used in every eventuality by NR? It's something that anyone can understand: them not wanting to risk blowing both the distribution network and trains from over-voltage.
In fact the frequency drop seems to have triggered disconnections by local distribution networks, but it seems that both traction supplies off the grid and S&T supplies were OK after a very short time (the latter after the railway's stand-by generators kicked in.)
It's just a pity that the trains reacted the way they did...
 

edwin_m

Veteran Member
Joined
21 Apr 2013
Messages
24,793
Location
Nottingham
From the Energy TSI: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32014R1301&from=EN
The values and limits of the voltage and frequency shall comply with EN 50163:2004, clause 4 for the selected system.
This is likely to be what all modern trains have to tolerate. Anyone have access to EN50163?

Stystem frequency was back within spec within 3 mins of start of incident.
Might a driver in a hurry have completed several re-boots in that time?

I presume static converters at feeder stations, as to be installed at two places on the ECML, would de-couple those areas from the frequency of the grid.
 

Greybeard33

Established Member
Joined
18 Feb 2012
Messages
4,229
Location
Greater Manchester
From the Energy TSI: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32014R1301&from=EN

This is likely to be what all modern trains have to tolerate. Anyone have access to EN50163?


Might a driver in a hurry have completed several re-boots in that time?

I presume static converters at feeder stations, as to be installed at two places on the ECML, would de-couple those areas from the frequency of the grid.
For 25kV systems with a synchronous connection to an interconnected system, as in Britain, the EN gives the frequency limits as +/-1% (49.5Hz - 50.5Hz) for 95% of a year and +4%/-6% (47Hz - 52Hz) for 100% of the time. However, there is a note that, in practice, frequency is more closely controlled than this in Europe and vehicles will operate only within the range 49Hz - 51Hz. If the frequency is out of this range, the vehicle performance may be reduced or the drives disconnected.

There is no limit on the rate of change of frequency.
 

edwin_m

Veteran Member
Joined
21 Apr 2013
Messages
24,793
Location
Nottingham
For 25kV systems with a synchronous connection to an interconnected system, as in Britain, the EN gives the frequency limits as +/-1% (49.5Hz - 50.5Hz) for 95% of a year and +4%/-6% (47Hz - 52Hz) for 100% of the time. However, there is a note that, in practice, frequency is more closely controlled than this in Europe and vehicles will operate only within the range 49Hz - 51Hz. If the frequency is out of this range, the vehicle performance may be reduced or the drives disconnected.

There is no limit on the rate of change of frequency.
Gridwatch suggests grid frequency was as low as 48.889Hz at 15:55
And that wasn't the extreme value but just the one that happened to be taken then.

Licence requirement is above 49.5Hz
So if I'm understanding it correctly, the supply went outside the frequency range that it's supposed to stay within for 95% of the time but didn't go beyond the absolute limits. And the fact this hasn't happened in the several years since introduction of the 700s does indeed confirm that the frequency control is a lot better than the minimum required by the TSI, because if it was more than 0.5Hz off for anywhere near 5% of the time then this event would have happened dozens of times already.

The standard allows a range of frequencies which it is permitted for the infrastructure to supply but where the trains may not work. Why not just say the infrastructure controller must keep the frequency between 49Hz and 51Hz and trains have to operate normally within this range, and if it goes outside this range they may not be able to operate but should restore once the frequency goes back within limits?
 

HSTEd

Veteran Member
Joined
14 Jul 2011
Messages
16,629
So if I'm understanding it correctly, the supply went outside the frequency range that it's supposed to stay within for 95% of the time but didn't go beyond the absolute limits. And the fact this hasn't happened in the several years since introduction of the 700s does indeed confirm that the frequency control is a lot better than the minimum required by the TSI, because if it was more than 0.5Hz off for anywhere near 5% of the time then this event would have happened dozens of times already.

The standard allows a range of frequencies which it is permitted for the infrastructure to supply but where the trains may not work. Why not just say the infrastructure controller must keep the frequency between 49Hz and 51Hz and trains have to operate normally within this range, and if it goes outside this range they may not be able to operate but should restore once the frequency goes back within limits?

Remember the TSI is a minimum standard that must be met by vehicles operating internationally.
National Grid specifies a much tighter standard for the grid system which inherently constrains the standards for NetworK Rail.

Tightening the frequency limits that the traction system must be able to tolerate significantly simplifies and lightens the traction package.

I presume static converters at feeder stations, as to be installed at two places on the ECML, would de-couple those areas from the frequency of the grid.
Unlikely because one of the points of having static converters is it would allow you to eliminate neutral sections.
So it is likely the system will be configured to have those feeders contiguous with conventional units.

It is also worth noting that decoupling the supply completely would require static converters capable of supplying 100% of demand.

If you accept phase coupling to the grid, you can get away with static converters that only meet 50% of demand.
 

Greybeard33

Established Member
Joined
18 Feb 2012
Messages
4,229
Location
Greater Manchester
Why not just say the infrastructure controller must keep the frequency between 49Hz and 51Hz and trains have to operate normally within this range, and if it goes outside this range they may not be able to operate but should restore once the frequency goes back within limits?
For the frequency limits EN50163 refers out to EN50160, "Voltage characteristics of electricity supplied by public electricity networks". This is because the railway has no control over the frequency, when it is synchronised to the public network.

In tripping out below 49Hz the 700/717 were, I suppose, compliant with the letter of the TSI. But surely it is implicit that a simple reset action should be sufficient to bring the traction system back on line, once the power supply is back within limits?
 

edwin_m

Veteran Member
Joined
21 Apr 2013
Messages
24,793
Location
Nottingham
Remember the TSI is a minimum standard that must be met by vehicles operating internationally.
National Grid specifies a much tighter standard for the grid system which inherently constrains the standards for NetworK Rail.

Tightening the frequency limits that the traction system must be able to tolerate significantly simplifies and lightens the traction package.
With current technology (sorry!) it makes sense for a 50Hz electrification to be synchronised to the grid, but that might not be so with some future technological development. It's not so today with the 15kV countries. So an infrastructure controller could provide a TSI-compliant supply and the train builder could provide a TSI-compliant train, and they wouldn't work reliably together which is the whole point of interoperability!

For the frequency limits EN50163 refers out to EN50160, "Voltage characteristics of electricity supplied by public electricity networks". This is because the railway has no control over the frequency, when it is synchronised to the public network.

In tripping out below 49Hz the 700/717 were, I suppose, compliant with the letter of the TSI. But surely it is implicit that a simple reset action should be sufficient to bring the traction system back on line, once the power supply is back within limits?
The wording of the ENs is fairly sloppy, but I agree the common sense interpretation is that the train may have to shut down to protect itself if the supply goes below 49Hz, but it should be easily recoverable. Hopefully this can and will be fixed by a software upgrade.
 
Joined
7 Jan 2009
Messages
859
Hopefully this can and will be fixed by a software upgrade.

Indeed, noting that most (all?) other electric fleets were not affected by the frequency dip. The frequency was back within statutory limits within 3 mins, so it feels like that the C700s may have tripped out (one way!) on electromagnetic interference grounds (done to protect the advanced power electronics in the drive systems as well as for safety).
 

AndrewE

Established Member
Joined
9 Nov 2015
Messages
5,063
A bit more background:https://www.theguardian.com/busines...l&utm_campaign=GuardianTodayUK&CMP=GTUK_email
...industry sources claim National Grid has been aware of the growing potential for a wide-scale blackout “for years”, and has suffered a spate of near-misses in recent weeks.

The Guardian understands that in every month since May there has been a severe dip in the grid’s frequency from its normal range around 50Hz. Industry sources have confirmed that the grid’s frequency has fallen below 49.6Hz on three different occasions in recent months, the deepest falls seen on the UK grid since 2015. On Friday the blackout was triggered when the frequency slumped to 48.88Hz.

In June, the frequency of the grid plummeted to within a whisker of National Grid’s legal limit of 49.5Hz after all three units of EDF Energy’s West Burton gas-fired power plant in Nottinghamshire tripped offline without warning.

The unexpected outage triggered an emergency call for backup electricity supplies which stabilised the energy grid’s frequency before a blackout was triggered.

In addition, the grid’s frequency fell to 49.55Hz on 9 May, and 49.58Hz of 11 July.
It seems that that the grid has been running on empty for a while!
 

apk55

Member
Joined
7 Jul 2011
Messages
438
Location
Altrincham
Problem is that wind turbines are very slow to respond to changes in power demand, it typically takes takes seconds to adjust the blade angle to adjust power output. In contrast coal fired power stations can respond to changes load in milliseconds with gas fired power stations being almost as good. Up to now the grid has been using fast response power stations to stabilize the frequency although this has meant that many of them have been running inefficiently on part load with spare capacity on hand (spinning reserve) to cope with sudden increases in demand.
If we are going to use a lot more renewable then we will need to look at fast response generators. Pumped storage is one but then there is only a limited expansion potential and they are located well away from loads in mountainous areas.
Grid level batteries are good possibility but expensive and have the ability to go from being a load (charging) to full load in milliseconds. Perth in Australia is already trying out this system.
 

hwl

Established Member
Joined
5 Feb 2012
Messages
7,354
Problem is that wind turbines are very slow to respond to changes in power demand, it typically takes takes seconds to adjust the blade angle to adjust power output. In contrast coal fired power stations can respond to changes load in milliseconds with gas fired power stations being almost as good. Up to now the grid has been using fast response power stations to stabilize the frequency although this has meant that many of them have been running inefficiently on part load with spare capacity on hand (spinning reserve) to cope with sudden increases in demand.
If we are going to use a lot more renewable then we will need to look at fast response generators. Pumped storage is one but then there is only a limited expansion potential and they are located well away from loads in mountainous areas.
Grid level batteries are good possibility but expensive and have the ability to go from being a load (charging) to full load in milliseconds. Perth in Australia is already trying out this system.
Pumped storage being in the "mountains" isn't a big issue given the Grid Architecture in the UK, the time taken to bring on line is and that is where batteries can help.
 

Greybeard33

Established Member
Joined
18 Feb 2012
Messages
4,229
Location
Greater Manchester
With current technology (sorry!) it makes sense for a 50Hz electrification to be synchronised to the grid, but that might not be so with some future technological development. It's not so today with the 15kV countries. So an infrastructure controller could provide a TSI-compliant supply and the train builder could provide a TSI-compliant train, and they wouldn't work reliably together which is the whole point of interoperability!
EN50163 actually has different frequency limits for systems with no synchronous connection to an interconnected system (e.g. supply systems on certain islands). These have much wider tolerances, +/-2% (49Hz - 51Hz) for 95% of a week and +/-15% (42.5Hz - 57.5Hz) for 100% of the time. I think this reflects the increased difficulty of maintaining a stable frequency in a small network of synchronous generators with rapid load variations.

(There are also three different sets of frequency limits for 16.7Hz systems, but I think that is OT for this thread).

To me the wording of the Frequency section of the EN bears the hallmarks of a typical standardisation committee compromise/fudge between the opposing views of different interest groups. The supply industry, in their CENELEC committee, would have insisted on keeping 47Hz as the extreme limit for the grid. But the train makers, in the railway committee, would have argued that 47Hz was unrealistically conservative and would unnecessarily penalise the design of transformers and traction converters. So they got the note added to say that they only really had to design for 49Hz.

The chickens are now coming home to roost. Wind and solar farms, and the continental interconnectors, use induction generators or inverters that just follow the frequency of the reduced number of synchronous generators in thermal and nuclear power stations. So there is less spinning mechanical inertia to smooth out short term frequency fluctuations when there is a sudden change in the balance between supply and load. At the same time train transformers and traction electronics have become more finely tuned to the supply frequency, to maximise efficiency and minimise weight, and so are less tolerant of fluctuations.
 

edwin_m

Veteran Member
Joined
21 Apr 2013
Messages
24,793
Location
Nottingham
EN50163 actually has different frequency limits for systems with no synchronous connection to an interconnected system (e.g. supply systems on certain islands). These have much wider tolerances, +/-2% (49Hz - 51Hz) for 95% of a week and +/-15% (42.5Hz - 57.5Hz) for 100% of the time. I think this reflects the increased difficulty of maintaining a stable frequency in a small network of synchronous generators with rapid load variations.

(There are also three different sets of frequency limits for 16.7Hz systems, but I think that is OT for this thread).

To me the wording of the Frequency section of the EN bears the hallmarks of a typical standardisation committee compromise/fudge between the opposing views of different interest groups. The supply industry, in their CENELEC committee, would have insisted on keeping 47Hz as the extreme limit for the grid. But the train makers, in the railway committee, would have argued that 47Hz was unrealistically conservative and would unnecessarily penalise the design of transformers and traction converters. So they got the note added to say that they only really had to design for 49Hz.

The chickens are now coming home to roost. Wind and solar farms, and the continental interconnectors, use induction generators or inverters that just follow the frequency of the reduced number of synchronous generators in thermal and nuclear power stations. So there is less spinning mechanical inertia to smooth out short term frequency fluctuations when there is a sudden change in the balance between supply and load. At the same time train transformers and traction electronics have become more finely tuned to the supply frequency, to maximise efficiency and minimise weight, and so are less tolerant of fluctuations.
That all sounds plausible. I'm not sure if any of the islands with no sychronous connection has railways, but if so they'd probably better think twice about buying any Desiros.

For the rest of us, it seems that excursions below 49Hz are very rare, so probably OK if that causes some trains to shut down for their own protection - provided they can easily be restarted once the frequency is restored.
 

yorkie

Forum Staff
Staff Member
Administrator
Joined
6 Jun 2005
Messages
67,429
Location
Yorkshire

apk55

Member
Joined
7 Jul 2011
Messages
438
Location
Altrincham
I am sure the problem must lie in the control electronics for the 4 quadrant converter.

If there problems with the transformer then it would affect every other transformer connected to the grid. As long as maximum voltage is not applied at the same time as minimum frequency which would cause the flux density in the core to rise possibly to to saturation level, but normally there is a good safety margin in design to avoid generating harmonics in the supply.

There is no particular reason why a 4 quadrant converter can not work over a wide frequency range. Therefore the problem must lie with the controlling electronics which generates the timing signals for the power switching devices. This will probably contain a phase locked clock to the supply from which the timing signals would be derived and could lose lock if the frequency drifts outside a narrow range or even changes too quickly and so trip out.
I could well see some software or hardware mods being introduced
 

HSTEd

Veteran Member
Joined
14 Jul 2011
Messages
16,629
If there problems with the transformer then it would affect every other transformer connected to the grid. As long as maximum voltage is not applied at the same time as minimum frequency which would cause the flux density in the core to rise possibly to to saturation level, but normally there is a good safety margin in design to avoid generating harmonics in the supply.

I think the problem might be that this safety margin has been cut to be wafer thin to reduce the weight of the transformer.
 

hwl

Established Member
Joined
5 Feb 2012
Messages
7,354
I think the problem might be that this safety margin has been cut to be wafer thin to reduce the weight of the transformer.
Agreed new transformer design that has significant cooling and is pretty light compared to older designs.
 

Greybeard33

Established Member
Joined
18 Feb 2012
Messages
4,229
Location
Greater Manchester
Agreed new transformer design that has significant cooling and is pretty light compared to older designs.
So the transformer is presumably designed with just enough iron to tolerate an overvoltage at 49Hz. And I would guess that the thresholds of the underfrequency and overvoltage protection are independent of each other, so the HV circuit breaker will trip, to protect the transformer, if the frequency dips below 49Hz regardless of the voltage.

But in itself this should not have prevented a reboot. The monitoring software presumably "thought" it had detected a more serious, irrecoverable, fault.
 

edwin_m

Veteran Member
Joined
21 Apr 2013
Messages
24,793
Location
Nottingham
So the transformer is presumably designed with just enough iron to tolerate an overvoltage at 49Hz. And I would guess that the thresholds of the underfrequency and overvoltage protection are independent of each other, so the HV circuit breaker will trip, to protect the transformer, if the frequency dips below 49Hz regardless of the voltage.

But in itself this should not have prevented a reboot. The monitoring software presumably "thought" it had detected a more serious, irrecoverable, fault.
It may be using the line frequency as a cross-check on its own master clock, which might also be used to ensure that the traction inverters don't stray into signalling frequencies. If so then clock drift might be considered a safety-critical failure and the irrevocable shutdown may be a means of reducing the probability of that failure going undetected. Fixing that isn't just a question of changing the software, it involves re-doing the RAM analysis and possibly finding some other mitigation of that hazard. All of which will require some sort of independent check and approval.
 

43096

On Moderation
Joined
23 Nov 2015
Messages
15,160
It has been suggested that the issue was not actually with the trains themselves and the frequency of the supply in the overhead line, but actually a failure with the Siemens control system at Three Bridges, which when it faulted sent a message to all Class 700 and 717 sets in AC mode to shut down. Re-booting the trains was not universally successful and when the maximum number of goes was exceeded the units shut down (much like your bank card will do if you get the PIN wrong too many times) and need a fitter with a laptop to get them going again.
 

hwl

Established Member
Joined
5 Feb 2012
Messages
7,354
Why would that only take out units on AC though?
The fault /fault messages (doesn't particularly matter which for this question) only related to the 4 quadrant converter (4QC) which rectifies the stepped down AC coming out of the transformer or in the regenerative braking case turns the current from the DC link into AC to be stepped up in the transformer, hence no DC running impact.
 

Greg Read

Member
Joined
3 Feb 2019
Messages
53
According the the BBC News this morning, it was the National Grid that had to decide whether to take many households off the grid, or shut down the Power to Network Rail, they took the decision to cut the power to the rail network, they were or are syaing this decision will be looked at in the inquiry.
 

hwl

Established Member
Joined
5 Feb 2012
Messages
7,354
According the the BBC News this morning, it was the National Grid that had to decide whether to take many households off the grid, or shut down the Power to Network Rail, they took the decision to cut the power to the rail network, they were or are saying this decision will be looked at in the inquiry.
There wasn't much deciding as it was automatic and happened so quickly after the 2nd trip incident. Everything in entire areas was cut nothing selective. Only in very few locations could NR alone but cut off for example the NR 33Kv networks that feed 3rd rail supplies from the Wimbledon, New Cross and Croydon National Grid 400/275kV supplies with seperate NR only transformers.
 

Class 170101

Established Member
Joined
1 Mar 2014
Messages
7,908
According the the BBC News this morning, it was the National Grid that had to decide whether to take many households off the grid, or shut down the Power to Network Rail, they took the decision to cut the power to the rail network, they were or are syaing this decision will be looked at in the inquiry.

Politically though NR doesn't vote but householders do. We haven't yet got to the state where we can tell where individual householders are otherwise you may be able to cut off sufficient householders for a short period of time and they wouldn't notice.
 

yorksrob

Veteran Member
Joined
6 Aug 2009
Messages
38,818
Location
Yorks
It's accepted practice that at times of shortage, energy suppliers will ration supply to industrial customers in preference to cutting off domestic households. It's down to those industrial customers to have equipment that doesn't take several hours to reboot.
 
Status
Not open for further replies.

Top