Amsterdam: complete outage of traffic control

AlexNL · 21 Aug 2018

At around 18:45 Dutch time, the Amsterdam signalling box went down. This brought train traffic on the Schiphol - Amsterdam - Weesp axis to a standstill, as controlling points and signals became impossible. Around 20:00 (Dutch time) ProRail announced that the issue had been resolved, but around 20:30 the entire box went down again.

As train traffic came to a standstill, passengers have searched for alternatives. Either through carpooling (#treinpoolen on Twitter) or by pouring themselves onto bus lines, which quickly suffered from severe overcrowding. At some bus stops, police have had to step in to perform crowd control measures.

As of 22:30 it looks like temporary block working has been introduced, some trains have been able to depart from Amsterdam since then. However, NS and ProRail do not expect the disruption to be cleared before end of service.

This issue is also likely to have some knock-on effects tomorrow morning, it's likely that large amounts of rolling stock (for domestic as well as international services) is displaced so I expect some cancellations and short forms on all routes which go from, to or via Amsterdam.

RailUK Forums

Gerald Fiennes · 21 Aug 2018

Wow, do let us know what the cause turns out to be. Sounds very similar to the incident in June where everything into Victoria was caped following the power supply at Streatham playing up.

deep south · 21 Aug 2018

I flew into Schiphol this afternoon and tried to get a train to Centraal at about 17:45, just as all trains stopped - they said it was due to persons on the track. After 25 minutes they advised passengers to use the bus instead, so we had a very slow journey to Sloterdijk. Sounds like it got a lot worse after that!

MisterT · 22 Aug 2018

That was another disruption that happened just before the signal box failure. Someone had stolen a few things at a shop and didn't want to get caught, so he went onto the tracks into the tunnel, causing all trains to be held to avoid an accident.

The signal box failure is still ongoing at the moment (3 am local time), but some trains are able to run by manual control.

Maybe nice to know: in case of a real 'melt down' of a signal box, there are two main back-up locations in Utrecht and Amersfoort. Every signalling box can be controlled from those locations. In addition to that, every signalling box contains one or a few back-up workplaces. Even in case the back-up locations can not be used because of an emergency, there is even a back-up for the back-up at all signalling posts.
But it takes some time to set up, and an order to go to the back-up location is not easily given.

Edit: as of 4:15 am, the problems have been solved.

Groningen · 22 Aug 2018

There was a second thing going on. Less serious, but it had an impact on the running of the trains. A shoplifter decided to escape Schiphol using the traintracks. Bringing everything to a standstill. / In case of no trains busses are used, but in the case of Amsterdam and Schiphol a lot of busses (and drivers) are needed.

CMS · 22 Aug 2018

Whenever I have travelled by train in The Netherlands, I have been seriously impressed by the multi-modal connectivity, service frequency, customer info and generally good station environments - but the service disruption is massive. There aren't often big disruptions but when they are, traffic stops for hours, there are massive diversions and a good chunk of routes completely shut down, such as this example or the storm where traffic stopped across the whole country a few moths ago or when there was an incident at Schiphol and the entire Intercity Direct service stopped and HSL traffic went via Gouda. The info is generally still quite good and the density of the network massively helps as there are alternatives but I find it so odd, this doesn't seem to happen in the UK/France/Switzerland - Belgium is random.

Groningen · 22 Aug 2018

A few days ago we had Prorail and NS coming with a statement that in 2030 the Dutch Railway is full. We/they are searching for solutions. They are looking at light rail. I suggest that we leave 1500 Volts overhead wires.

CMS · 22 Aug 2018

Groningen said:
A few days ago we had Prorail and NS coming with a statement that in 2030 the Dutch Railway is full. We/they are searching for solutions. They are looking at light rail. I suggest that we leave 1500 Volts overhead wires.

I can believe that! I recently had to wait 30 mins for a Thalys at Rotterdam and the amount of conflicting movements, trains already double-deck and still busy, full bi-directional signalling and trains around every 10 mins on each route out of the station, I wonder how the network is to cope.

One thing they must do is either get new stock for Intercity Direct or get those trains off the HSL - I don't know how they will, but mixed speed on this route really limits capacity. I'm surprised the current situation even exists.

Groningen · 22 Aug 2018

What the Thalys concerns in Rotterdam there are no real conficts with tracks. On Sporenplan.nl you can search every trainline. Thalys to Belgium is track 2 and to Amsterdam 11. Any track crossed is not at the same level (of height).

43096 · 22 Aug 2018

AlexNL said:
At around 18:45 Dutch time, the Amsterdam signalling box went down. This brought train traffic on the Schiphol - Amsterdam - Weesp axis to a standstill, as controlling points and signals became impossible. Around 20:00 (Dutch time) ProRail announced that the issue had been resolved, but around 20:30 the entire box went down again.

As train traffic came to a standstill, passengers have searched for alternatives. Either through carpooling (#treinpoolen on Twitter) or by pouring themselves onto bus lines, which quickly suffered from severe overcrowding. At some bus stops, police have had to step in to perform crowd control measures.

As of 22:30 it looks like temporary block working has been introduced, some trains have been able to depart from Amsterdam since then. However, NS and ProRail do not expect the disruption to be cleared before end of service.

This issue is also likely to have some knock-on effects tomorrow morning, it's likely that large amounts of rolling stock (for domestic as well as international services) is displaced so I expect some cancellations and short forms on all routes which go from, to or via Amsterdam.

Having been caught up in this at Schiphol, I have to say NS were a total disgrace. There was no information (other than blatant lies about person on the track), departure boards were blank and the platform screens said “Listen for announcements”. There were no announcements. None whatsoever. Having done the standard of go for food and beer, it was quickly apparent that nothing was moving and that NS had given up for the night. With a several hour taxi queue, we got a bus to Haarlem, where eventually a Vlissingen IC turned up to get us back to Rotterdam.

Goes to show that it is not just the likes of SWR and GTR that can do a shambles that goes up to 11 on the riot-o-meter (scale 1 to 10): no announcements, info screens useless, staff gone into hiding and closing the station at a major global airport.

AlexNL · 22 Aug 2018

ProRail have explained what caused the issue: a software bug.

At Schiphol Airport, platforms 1-2 and 5-6 are allocated dynamically on a first come first serve basis. There is a trigger point in the infrastructure which is activated when a train passes over it.

Yesterday, when the shoplifter decided to attempt an escape through the tunnel, a train came to a stop right on top of the trigger point, which is normally passed at speed. Due to a bug the trigger was activated repeatedly until a database table got full, which caused the entire system to go down. The backup system works on a recent copy of the data, and thus also crashed.

CMS · 22 Aug 2018

Groningen said:
What the Thalys concerns in Rotterdam there are no real conficts with tracks. On Sporenplan.nl you can search every trainline. Thalys to Belgium is track 2 and to Amsterdam 11. Any track crossed is not at the same level (of height).

I definitely saw some conflicting movements with local trains, perhaps they were unscheduled though. My Thalys went from platform 4, I remember thinking, how odd there is no platform 5 but then on the platform I realised hat there was a through line between 4 & 6.

43096 said:
With a several hour taxi queue, we got a bus to Haarlem, where eventually a Vlissingen IC turned up to get us back to Rotterdam.

I think this happens a lot with NS disruptions, it goes from almost perfect to total chaos. If the network was not as dense, there really would be no alternatives. It's moments like this which boost my image of SNCF and the UK. It's a same you didn't get a bus straight towards Amsterdam - they are extremely expensive though!

MisterT · 22 Aug 2018

I'm wondering what people would expect on information when all control is lost in a busy area like around Amsterdam/Schiphol.
There simply was no more information to give than that, and with that amount of people, it was impossible to get any alternative means of transportation. According to NS, more than 52000 passengers were stranded.

NSEFAN · 22 Aug 2018

AlexNL said:
ProRail have explained what caused the issue: a software bug.

At Schiphol Airport, platforms 1-2 and 5-6 are allocated dynamically on a first come first serve basis. There is a trigger point in the infrastructure which is activated when a train passes over it.

Yesterday, when the shoplifter decided to attempt an escape through the tunnel, a train came to a stop right on top of the trigger point, which is normally passed at speed. Due to a bug the trigger was activated repeatedly until a database table got full, which caused the entire system to go down. The backup system works on a recent copy of the data, and thus also crashed.

Oops! That's a believably awful bug! It's the little things like that that can bring down entire systems.

43096 · 22 Aug 2018

MisterT said:
I'm wondering what people would expect on information when all control is lost in a busy area like around Amsterdam/Schiphol.
There simply was no more information to give than that, and with that amount of people, it was impossible to get any alternative means of transportation. According to NS, more than 52000 passengers were stranded.

Some form of communication: screens and announcements saying there are no trains due to xyz and that there is currently no known estimate for resumption would be a good start. For nothing to be said and staff to go and hide is frankly appalling customer service and utterly inexcusable.

Put yourself in the position of someone arriving at Schiphol for the first time. What would you think?

43096 · 22 Aug 2018

CMS said:
I think this happens a lot with NS disruptions, it goes from almost perfect to total chaos. If the network was not as dense, there really would be no alternatives. It's moments like this which boost my image of SNCF and the UK. It's a same you didn't get a bus straight towards Amsterdam - they are extremely expensive though!

Hotel was in Rotterdam anyway...

Bus was wedged so wasn’t going to bother “tapping in”. I’ll venture 99% of those on the bus didn’t bother paying, either!

43096 · 22 Aug 2018

NSEFAN said:
Oops! That's a believably awful bug! It's the little things like that that can bring down entire systems.

So that bug causes the whole of Amsterdam panel to be wiped out? That is an unbelievably cr@p bit of software then, especially for a safety critical system. It smacks of inadequate testing - what other horrors are lurking in it?

NSEFAN · 22 Aug 2018

43096 said:
So that bug causes the whole of Amsterdam panel to be wiped out? That is an unbelievably cr@p bit of software then.

It is indeed crap, although I can see how it could have happened. If the two bits have been made independently then there's a higher chance of a glaring error like that being missed, even more so if would never show up during normal operation, until just by chance a train stops at exactly the wrong point.

AlexNL · 22 Aug 2018

The system has been running for 12 years and this hasn't happened before. While it is something that definitely should be fixed, I wouldn't call that unbelievably crappy software.

Also, the safety critical system did exactly what it should do in case of unforseen circumstances: it stopped working.

43096 · 22 Aug 2018

AlexNL said:
The system has been running for 12 years and this hasn't happened before. While it is something that definitely should be fixed, I wouldn't call that unbelievably crappy software.

Also, the safety critical system did exactly what it should do in case of unforseen circumstances: it stopped working.

Sorry, but something like that should not be causing half of Amsterdam to shut down and not be fixable for hours - local Schiphol area will obviously be affected, but not on that scale. That incident and the inability to recover from it affected tens of thousands of people, not just on the railway, and had the potential to cause a serious public order incident because of the way it was handled (or not).

CMS · 22 Aug 2018

MisterT said:
I'm wondering what people would expect on information when all control is lost in a busy area like around Amsterdam/Schiphol.
There simply was no more information to give than that, and with that amount of people, it was impossible to get any alternative means of transportation. According to NS, more than 52000 passengers were stranded.

That is exactly what NS say! 'Het is nog niet bekend hoe lang dit zal duren' - that information is useless to customers! They need to give customers an action i.e. get a bus, go to Haarlem/Amsterdam Zuid etc. 52000 stranded people is inexcusable unless there is some sort of national emergency.

To give credit to NS, at least they have announcements at major stations in English. I find it shameful that in the UK we don't offer travel information in other languages bar Eurostar, yet I am very surprised that they have NO live travel information in English on their app and website though. You have to use a translator app or Rijden de Treinen.

duesselmartin · 22 Aug 2018

That could be the major disadvantage of large electronic signalling centres.
If Duisburg / Germany fails millions would be affected.

NSEFAN · 22 Aug 2018

43096 said:
Sorry, but something like that should not be causing half of Amsterdam to shut down and not be fixable for hours - local Schiphol area will obviously be affected, but not on that scale. That incident and the inability to recover from it affected tens of thousands of people, not just on the railway, and had the potential to cause a serious public order incident because of the way it was handled (or not).

Of course it shouldn't have stopped all of Amsterdam, but if the system has been in use for 12 years without a problem then it's no surprise that the bug wasn't found, because nobody was looking for it. Once the whole lot has packed up it'll be a manual process of getting things running again, working out which trains are where and liaising with control to recover some kind of service. The problem started at 18:45 so it's not surprising that nothing much could run for the rest of the day.

It's bad enough if a fairly local signal box has a total failure of some kind, but if a ROC had a similar meltdown to that in Amsterdam then I would expect a similar level of chaos, both for recovering the service and also the TOCs getting people to where they need to be. There's only so many taxis out there and coach hire can't always be obtained at immediate notice.

43096 · 22 Aug 2018

NSEFAN said:
It's bad enough if a fairly local signal box has a total failure of some kind, but if a ROC had a similar meltdown to that in Amsterdam then I would expect a similar level of chaos, both for recovering the service and also the TOCs getting people to where they need to be. There's only so many taxis out there and coach hire can't always be obtained at immediate notice.

I get that there aren't going to be buses/coaches/taxis available to get everyone moving - you're talking tens of thousands of people. What I cannot accept - and it applies in the UK; this was a very British-style incident - is the complete lack of information. Just be honest - if they don't know when service will resume, then say so. To say nothing is just unacceptable on every level. That is what NS did last night, and they failed badly as an organisation.

AlexNL · 22 Aug 2018

NSEFAN said:
The problem started at 18:45 so it's not surprising that nothing much could run for the rest of the day.

What did not help was that ProRail originally announced the incident as having been resolved around 20:00. Between 18:45 en 20:00, NS and ProRail will most likely have been busy with guiding stranded trains to the next station so that passengers can disembark.

After 20:00 the focus must've shifted to service recovery, but when the problem resurfaced 30 minutes later they had to go back into "oh **** nothing is working anymore"-mode and find new ways to get people moving again (introducing temporary block working?).

MisterT · 23 Aug 2018

43096 said:
Some form of communication: screens and announcements saying there are no trains due to xyz and that there is currently no known estimate for resumption would be a good start.

They said exactly that 7 minutes after the problem became known (18:52) and 4 minutes earlier (18:48) there was already a message for Amsterdam Centraal only
At 19:12, NS realized that the problems would last for the rest of the day and updated the message accordingly.
Then, when the manual control became possible at 22:35, the message was changed from 'no trains' to 'a few trains are running'.

paddington · 23 Aug 2018

CMS said:
To give credit to NS, at least they have announcements at major stations in English. I find it shameful that in the UK we don't offer travel information in other languages bar Eurostar, yet I am very surprised that they have NO live travel information in English on their app and website though. You have to use a translator app or Rijden de Treinen.

If you don't know Dutch and are in the Netherlands, you probably know some English. In other EU countries the same is true, perhaps with the addition of Russian in some places. If a staff member knows Dutch and English, they'd probably be able to communicate with over 99% of passengers/customers. Which languages should be used in the UK? Each additional language would only cater to a small amount of people, yet if you want to cover 99% of your passengers you'd need lots of staff who each know a different second language, or several polyglots!

Groningen · 24 Aug 2018

That train that caused all the trouble in the Schiphol tunnel was contacted an amazing 32.000 times for the system crashed. If that shoplifter had to pay for all trouble he caused, he would be poor for the rest of his life. To call this disaster a disgrace is a little bit low by the ground, but maybe things can be learned from it.

AndrewE · 24 Aug 2018

AlexNL said:
ProRail have explained what caused the issue: a software bug.

At Schiphol Airport, platforms 1-2 and 5-6 are allocated dynamically on a first come first serve basis. There is a trigger point in the infrastructure which is activated when a train passes over it.

Yesterday, when the shoplifter decided to attempt an escape through the tunnel, a train came to a stop right on top of the trigger point, which is normally passed at speed. Due to a bug the trigger was activated repeatedly until a database table got full, which caused the entire system to go down. The backup system works on a recent copy of the data, and thus also crashed.

I am impressed that they are prepared to release this info to the public so promptly and in such detail. Here we would have evasions and white lies to protect the guilty party, and probably nothing would emerge unless there was a court case if NR decided to go after a supplier for compensation.

Groningen · 24 Aug 2018

Between 8 and 9 hours in the morning at Schiphol Airport there are (if i counted correctly) 44 departures.

Amsterdam: complete outage of traffic control

Established Member

RailUK Forums

Member

Member

Member

Established Member

Member

Established Member

Member

Established Member

On Moderation

Established Member

Member

Member

Established Member

On Moderation

On Moderation

On Moderation

Established Member

Established Member

On Moderation

Member

Established Member

Established Member

On Moderation

Established Member

Member

Member

Established Member

Established Member

Established Member