Network Rail Open Data

[Poggs]

Out of interest, you say 'I'. Does that mean you had some influence in the choice of platform? (And no problem about the neutral standard. I totally understand that you need something that can be accessed from any platform).

I've had access to the feed from TD.net for the past year, and my goal has always to be to lower the barrier of entry to anyone who wants to consume the data. Stomp was the natural choice - had I wanted a full feed directly from Network Rail, it's an IBM WebSphere MQ server and several thousand pounds! I fed this back up the food chain.

On the schedules, do you need to have a your own Amazon S3 account set up to download them? I don't and I had assumed that was fine since the URL to download schedules is a networkrail one, not an amazon one, but now wondering if that's the problem. I see one of my download attempts has caused a page redirect to https://nr-datafeed-cif.s3.amazonaws.com/.

You don't need an Amazon account to download the schedules - S3 is the content delivery network. You will, however, need to call https://datafeeds.networkrail.co.uk/ntrod /CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=tocupdate-
mon and authenticate with your username/password to be redirected to a temporary URL where you can download the schedule files.

 

[Zoe]

I have been watching the TD feed for one area in the last few days and I have noticed that at times, I don't get a message for each berth step. For example I may get some berth step messages for a specific train but then no messages of any berth steps for that train for a time and then when I do get a mesasage, I notice that some berth steps have not been reported. When it's very quiet there doesn't seem to be any issue though as earlier I observed 1A40 and 1C99 through the Bristol area and every berth step was reported.

 

[ralphchadkirk]

I have been watching the TD feed for one area in the last few days and I have noticed that at times, I don't get a message for each berth step. For example I may get some berth step messages for a specific train but then no messages of any berth steps for that train for a time and then when I do get a mesasage, I notice that some berth steps have not been reported. When it's very quiet there doesn't seem to be any issue though as earlier I observed 1A40 and 1C99 through the Bristol area and every berth step was reported.

Manual input area?

 

[Zoe]

Manual input area?

No, these are PSB areas. It's also not specific berths, it's different with each train, sometimes I can be very lucky like earlier when I every berth step on 1A40 and 1C99 was reported but other times quite a few berth steps are not getting reported although exactly which ones seems to change with each train.

 

[Paul Kelly]

I've been following the train movements feed and noticed there was an outage this morning, actually one shortage from 09:36-09:37 and then a much longer one from 09:40-10:38, where there were way fewer reports than normal coming through. Since 10:39 it's been back to normal though.

I saw the same behaviour yesterday afternoon for a while too, but don't have the exact times. Just wondering if those times this morning correspond with when anybody else is seeing strangeness?

 

[Poggs]

I've been following the train movements feed and noticed there was an outage this morning, actually one shortage from 09:36-09:37 and then a much longer one from 09:40-10:38, where there were way fewer reports than normal coming through. Since 10:39 it's been back to normal though.

I saw the same behaviour yesterday afternoon for a while too, but don't have the exact times. Just wondering if those times this morning correspond with when anybody else is seeing strangeness?

I've noticed this too - let me get some feedback sent... several of the things I'm working on simply won't work very will with missing berth steps!

 

[Zoe]

I've been following the train movements feed and noticed there was an outage this morning, actually one shortage from 09:36-09:37 and then a much longer one from 09:40-10:38, where there were way fewer reports than normal coming through. Since 10:39 it's been back to normal though.

I saw the same behaviour yesterday afternoon for a while too, but don't have the exact times. Just wondering if those times this morning correspond with when anybody else is seeing strangeness?

Yes, I noticed that. Strangely after things went back to normal I had about 30 minutes or so of getting a lot more TD berth step reports before it went back to the current situation where quite a few do not seem to be reported.

 

[DarloRich]

I am sorry are you people even speaking English???????????

I think this is a very bad idea. I mean if the spotters have this data how will i look connected and important? Mere normals shouldnt know how things work! This is the thin end of the wedge! Next you will work out that most of it is just pushing buttons, pulling levers, drinking tea and reading the paper FFS!

Where is my union?

 

[Aictos]

Ooh ooh ooh! I would say centred around Stevenage or around Finsbury Park.. well I would wouldn't I?

Hmm, I would be interested in either Edinburgh/Newcastle to Doncaster and Leeds to Doncaster

 

[b0b]

Can someone explain what the "berth" data is?

 

[ralphchadkirk]

Can someone explain what the "berth" data is?

They are train description berths; basically a small dot matrix screen by signals (not all though) on the signaller's panel (or computer if a lever frame, or even just as part of the display at IECCs) which 'describes' the train - it shows the head code. These can either be interposed, where the signaller changes or enters a description, cancelled, when a description is removed, or stepped when the train moves to the next signal section so its associated description moves to the next berth along.

 

[b0b]

These can either be interposed, where the signaller changes or enters a description, cancelled, when a description is removed, or stepped when the train moves to the next signal section so its associated description moves to the next berth along.

so can i assume its a bit like a signal block section, or does that simplify it too much?

 

[ralphchadkirk]

so can i assume its a bit like a signal block section, or does that simplify it too much?

A simplification would be that a berth is a screen that tells the signaller what is in each signal section.

 

[b0b]

so how do I get a map of where the berths are located?

 

[DynamicSpirit]

Originally Posted by DynamicSpirit
Has anyone had any luck downloading the train schedules? I don't seem to be able to.

I have just tried: https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full

It works for me, I am subscribed to the schedule "All Full Daily"

On more investigation it seems that the one you tried - the ALL_FULL one - is working, but is almost the only file that is.

The ALL_UPDATE files for Thursday-Sunday are also there, but Mon-Wed aren't. I'm guessing that may be an artifact of this being the very first week of data, and the remaining ones will be there by Friday.

However, none of the files for individual TOCs are working - whenever I try any of them I get the 401 not subscribed error. On more experimenting, this error happens either if you download something that you're not subscribed to, or if you mis-type something and so try to download a file that doesn't exist. So I would hypothesize that either there's a bug on the system - it's not recognizing individual TOC subscriptions, or the sample URL in the developer toolkit (https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ED_FULL_DAILY&day=toc-full) is wrong

 

[ian13]

However, none of the files for individual TOCs are working - whenever I try any of them I get the 401 not subscribed error. On more experimenting, this error happens either if you download something that

They're not currently available - they will be, but there are other issues with the schedule data that are being worked on first.

 

[DAH]

Having the same problem, just will not connect even though my status is 'Active'.

Getting:

Fatal error: Uncaught exception 'StompException' with message 'Could not connect to datafeeds.networkrail.co.uk:61618 (10/10)' in /<path>/Stomp.php:168
Stack trace: 
#0 <path>/Stomp.php(190): Stomp->_makeConnection() 
#1 <path>/stomp_test.php(7): Stomp->connect() 
#2 {main} thrown in <path>/Stomp.php on line 168

stomp_test.php is using what is suggested in the Developer Pack V4.0.

OK,

Out of curiosity, I just tried running this from the localhost on my PC and got instant results on train movements. Uploaded the exact same files to my web server and received the same error messages again(unable to connect etc).

Is there a logical reason for this?

 

[dosxuk]

OK,

Out of curiosity, I just tried running this from the localhost on my PC and got instant results on train movements. Uploaded the exact same files to my web server and received the same error messages again(unable to connect etc).

Is there a logical reason for this?

The port could be firewalled on the web server, blocking the connection attempt before it even leaves the machine.

 

[DAH]

The port could be firewalled on the web server, blocking the connection attempt before it even leaves the machine.

Cheers for that suggestion, 5 minutes live chat and the port has been openned.

Thank you

 

[Poggs]

They're not currently available - they will be, but there are other issues with the schedule data that are being worked on first.

What are they, just out of interest?

 

[DynamicSpirit]

They're not currently available - they will be, but there are other issues with the schedule data that are being worked on first.

Thanks ian13, good to know that the data is on its way.

 

[jamesjefferies]

Well, I've managed to get a simple Ruby listener up and running - I've written about it on my blog and have it set up on my github account

Let me know if it proves useful... Cheers!

 

[headshot119]

I've set myself up a program in Ruby just to download the raw TD step data. I'm having trouble accessing anything other than the M&C East Midlands feed. I can't seem to find a way to work out the URLs for all the different feeds.

For example I know that M&C East Midlands is /topic/TD_MC_EM_SIG_AREA.

But what do I use for Western Wales & Marches?

Would have helped if I'd have read the developer pack all the way from top to bottom.

Has anyone got a list, or knows where I can find a list to interpret the area IDs?

 

[Poggs]

I've set myself up a program in Ruby just to download the raw TD step data. I'm having trouble accessing anything other than the M&C East Midlands feed. I can't seem to find a way to work out the URLs for all the different feeds.

For example I know that M&C East Midlands is /topic/TD_MC_EM_SIG_AREA.

But what do I use for Western Wales & Marches?

Would have helped if I'd have read the developer pack all the way from top to bottom.

Has anyone got a list, or knows where I can find a list to interpret the area IDs?

I have a list here - drop me a private message with your email address and I'll forward 'em on.

 

[Aictos]

I have a list here - drop me a private message with your email address and I'll forward 'em on.

Could you do the same for me too please?

 

[robh]

Anyone developing a Java client I've come across ActiveMQ Apollo which is a Stomp message broker but the binary packages on the website contains some useful examples. If you extract the archive and browse to examples/stomp/java you can see an implementation of a stomp client that uses the fusesource stompjms library.

Here is a modified version of the Listener example to work with the network rail datafeeds. You will probably want to make use of a JSON library to get anything useful out of the messages but this shows the basic idea:

import org.fusesource.stomp.jms.*;
import javax.jms.*;

class Listener {

    public static void main(String[] args) throws JMSException {

        String user = "your_username";
        String password = "your_password";
        String host = "datafeeds.networkrail.co.uk";
        int port = 61618;
        String destination = "/topic/TRAIN_MVT_ALL_TOC"; // A topic to which you are subscribed

        StompJmsConnectionFactory factory = new StompJmsConnectionFactory();
        factory.setBrokerURI("tcp://" + host + ":" + port);

        Connection connection = factory.createConnection(user, password);
        connection.start();
        Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
        Destination dest = new StompJmsDestination(destination);

        MessageConsumer consumer = session.createConsumer(dest);
        
        System.out.println("Waiting for messages...");
        
        while(true) {
            Message msg = consumer.receive();
            // Process message here
        }

    }
}

 

[DAH]

Would anyone know where i could find a reference file which will help with TOC_ID and cancellation codes within the train movement feed??

ie:
[toc_id] => 71
[canx_reason_code] => X2

Many thanks

 

[Poggs]

Would anyone know where i could find a reference file which will help with TOC_ID and cancellation codes within the train movement feed??

ie:
[toc_id] => 71
[canx_reason_code] => X2

Many thanks

I don't have a list of TOC IDs, but the train_id of the previously activated train that's been cancelled will have the schedule UID in it, and from there you can find out the TOC.

The cancellation reason codes are as per the Delay Attribution Guide at http://www.delayattributionboard.co.uk/documents/dag_pdac/APRIL 1ST 2011 DELAY ATTRIBUTION GUIDE.pdf.

 

[DAH]

I don't have a list of TOC IDs, but the train_id of the previously activated train that's been cancelled will have the schedule UID in it, and from there you can find out the TOC.

The cancellation reason codes are as per the Delay Attribution Guide at http://www.delayattributionboard.co.uk/documents/dag_pdac/APRIL 1ST 2011 DELAY ATTRIBUTION GUIDE.pdf.

Thank you

 

[Skimble19]

Anyone else notice RES listed on the list of TOCs?