Network Rail Open Data

Status
Not open for further replies.

Metroland

Established Member
Joined
20 Jul 2005
Messages
3,212
Location
Midlands
It includes:

Budget and costs of high profile projects
Assessments for 50 high risk level crossings, scored on the rail industry's level crossing risk model
Agendas and minutes of meetings of the board
Executive directors’ expenses
Number of high earners by £25k band

David Higgins, chief executive said: “I believe in transparency. It helps improve decision making and can drive culture change in an organisation. Given the amount of money that taxpayers put into the rail industry, it is only correct they are given the right to scrutinise what their money is paying for. I believe Network Rail has to take the lead in opening up the industry to new levels of transparency and I hope others will soon follow that lead.”

We’re committed to increasing the amount of information we make available, and want the choices on what we publish in the future to be informed by what people would find helpful. So as well as proactively releasing this information, we’re inviting suggestions on what information should be added to this initial offering in the future.


Additionally, software developers can access real time railway data feeds, including:

Train Movements - train positioning and movement event data
TD - train positioning data, at signal berth level
TSR (Temporary Speed Restrictions) – details of temporary reductions in line speeds across the rail network
VSTP (Very Short Term Plan) – schedule records created via this process (and thus not available via CIF)
RTPPM (Real Time Public Performance Measure) - this shows the performance of trains against the timetable, measured as the percentage of trains arriving at destination on time
SCHEDULE – extract of train schedules from the Integrated Train Planning System

http://www.networkrail.co.uk/news/2012/JUN/our-transparency-programme-launches/
 
Sponsor Post - registered members do not see these adverts; click here to register, or click here to log in
R

RailUK Forums

GB

Established Member
Joined
16 Nov 2008
Messages
5,771
Location
Somewhere
Will be a shame if true that freight information will not be released. Hope that is not the case.
 

dosxuk

Member
Joined
2 Jan 2011
Messages
787
Will be a shame if true that freight information will not be released. Hope that is not the case.

The documentation states that all freight TOC data is filtered out, and that only TD data for class 1, 2 and 9 trains will be transmitted, so basically there will be no freight data.
 

sonic2009

Established Member
Joined
19 Jan 2010
Messages
4,175
Location
Moulton nr Winsford
The documentation states that all freight TOC data is filtered out, and that only TD data for class 1, 2 and 9 trains will be transmitted, so basically there will be no freight data.

I heard it's due to a certain Freight Operator not wishing to give out the information.
 

Poggs

Member
Joined
28 Aug 2008
Messages
281
Location
London
I'm looking forwards to seeing what Poggs does with all that :)

Already working on it.

Anyone got a five-mile diagram they can chuck my way for their favourite area of the country? I'll come up with a P2-esque map.
 

ralphchadkirk

Established Member
Joined
20 Oct 2008
Messages
5,764
Location
Essex
Just got access to the live feeds. Unfortunately the documentation for PHP just seems to be a copy and paste from the extension producers website. And it's flatly refusing my username and password despite saying that the security token isn't needed yet...


Sent from my iPad using Tapatalk HD
 

whhistle

On Moderation
Joined
30 Dec 2010
Messages
2,636
Already working on it.

Anyone got a five-mile diagram they can chuck my way for their favourite area of the country? I'll come up with a P2-esque map.

Try something busy but simple.
Leicester, Rugby, Coventry maybe?

Does this mean we'll see a few more apps which plot the train as it passes timing points rather than just at stations (like the NRE one does now)?
 

Poggs

Member
Joined
28 Aug 2008
Messages
281
Location
London
Just got access to the live feeds. Unfortunately the documentation for PHP just seems to be a copy and paste from the extension producers website. And it's flatly refusing my username and password despite saying that the security token isn't needed yet...

Is your account 'active', or still in 'pending'?
 

DennisDeo

New Member
Joined
29 Jun 2012
Messages
1
You have a leading empty space in your username when accessing the data feeds. The authentication would fail if the username/password won't match up.
 

dooton

Member
Joined
18 Nov 2010
Messages
64
Working on a Java implementation. There documentation could be a little more helpful I do agree!
 

ralphchadkirk

Established Member
Joined
20 Oct 2008
Messages
5,764
Location
Essex
You have a leading empty space in your username when accessing the data feeds. The authentication would fail if the username/password won't match up.

Thanks - have just spotted that leading space. Question is, where is it coming from as I'm not adding it and as far as I can see neither is the connect or make connection methods in Stomp!
--- old post above --- --- new post below ---
Have fixed it.

If anyone else is having the same problem, open Stomp/Frame.php and change line 68 to read: $data .= $name . ":" . $value . "\n";

The leading space was where the colon was.

So far that edit doesn't seem to have affected anything else, but as with any changes make sure you back up!
 
Last edited:

b0b

Established Member
Joined
25 Jan 2010
Messages
1,277
Java version

working great for me; here's some simple java code using the gozirra stomp library at http://www.germane-software.com/software/Java/Gozirra/


Code:
    private static void go() throws Exception
    {
        System.out.println("Connecting");
        final Client c = new Client(Constants.SERVER, Constants.PORT, Constants.USERNAME, Constants.PASSWORD);
        System.out.println("Connected");

        System.out.println("Subscribing");
        for (final String topic : Constants.TOPICS)
        {
            System.out.println("  " + topic);
            c.subscribe("/topic/" + topic,
                    new Listener()
                    {
                        public void message(Map map, String s)
                        {
                            processMessage(map, s);
                        }
                    });
        }

        System.out.println("done subscribing");

    }
the Constants are obvious but Constants.TOPICS is

Code:
    static String[] TOPICS = {
            "TRAIN_MVT_ALL_TOC",
            "TD_ALL_SIG_AREA",
            "VSTP_ALL",
            "TSR_ALL_ROUTE" }

and processMessage is up to you to implement, the String s has the json. easy stuff!
 
Joined
29 Jun 2012
Messages
8
Location
Sheffield, South Yorkshire
Thanks B0b for posting your java, I was attempting to get it working using the Stomp rubygem with the ruby example, but it seemed to keep dropping the connection... (and thanks to poggs for clearing up to use /topic and not /queue)

however, all works well with Gozirra, so I'm making progress.

Typical timing though, off to North Wales for hols for a couple of weeks, just when I want to do some JSON parsing!
 

ralphchadkirk

Established Member
Joined
20 Oct 2008
Messages
5,764
Location
Essex
Still having trouble with it - though I suspect it is me, not Stomp this time.

Basically trying to get the TSRs on the LNE(S) route, the script connects, subscribes with no Stomp errors but stalls when running the readFrame() method, sending the CPU usage up to 100% and crashing out when reaching the maximum time.
 

DynamicSpirit

Established Member
Joined
12 Apr 2012
Messages
4,907
Thanks for posting that, metroland - and so quickly after the announcement too! I've got my subscription in.

Has anyone had any luck downloading the train schedules? I don't seem to be able to. For example, the developer pack gives the example of using the URL https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ED_FULL_DAILY&day=toc-full to download the Northern Rail daily full schedule (Actually the PDF had an extra space after 'ntrod' which I've removed). But whenever I try that URL in the browser, I just get back the error User not subscribed to CIF_ED_FULL_DAILY. But a quick look at my account shows that's rubbish - it's clearly showing my account as active and subscribed to that schedule. Same thing with all the other schedules I've tried.
--- old post above --- --- new post below ---
Working on a Java implementation. There documentation could be a little more helpful I do agree!

Agreed. And it's even worse for people like me who inhabit the .NET world, since they don't even provide samples in those languages!
 

robh

Member
Joined
30 Jun 2012
Messages
7
Has anyone had any luck downloading the train schedules? I don't seem to be able to.

I have just tried: https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full

It works for me, I am subscribed to the schedule "All Full Daily"

Working on a Java implementation. There documentation could be a little more helpful I do agree!

I agree, the network rail doc is just a copy of the "Java API" section of the ActiveMQ documentation: https://activemq.apache.org/stomp.html with the key sentence "Note that this API is provided purely for testing purposes and you should always consider using standard JMS API from Java instead of this one." removed.

Can anyone tell me the port that their stomp service is running on? It's something like ***16 if I remember correctly. I was playing around with it at work on Friday and had it working but I can't seem to download the developer PDF here at home as it's asking me for a login and the one for the datafeeds isn't being accepted.

EDIT: Found a cached version (slightly older revision than I was reading on Friday though): https://docs.google.com/viewer?a=v&...U9D1B0&sig=AHIEtbSe7pJ6kervc15m6ST6CMetUZeMKQ

It's port 61618
 
Last edited:

DAH

Member
Joined
17 Apr 2008
Messages
19
Just got access to the live feeds. Unfortunately the documentation for PHP just seems to be a copy and paste from the extension producers website. And it's flatly refusing my username and password despite saying that the security token isn't needed yet...

Having the same problem, just will not connect even though my status is 'Active'.

Getting:

Code:
Fatal error: Uncaught exception 'StompException' with message 'Could not connect to datafeeds.networkrail.co.uk:61618 (10/10)' in /<path>/Stomp.php:168
Stack trace: 
#0 <path>/Stomp.php(190): Stomp->_makeConnection() 
#1 <path>/stomp_test.php(7): Stomp->connect() 
#2 {main} thrown in <path>/Stomp.php on line 168

stomp_test.php is using what is suggested in the Developer Pack V4.0.
 

ralphchadkirk

Established Member
Joined
20 Oct 2008
Messages
5,764
Location
Essex
Still haven't got to actually parsing a JSON message yet! This is my first experience of ActiveMQ and Stomp, but I'm having more trouble than I expected. Still got the crashing problem...
--- old post above --- --- new post below ---
Having the same problem, just will not connect even though my status is 'Active'.

Getting:

Code:
Fatal error: Uncaught exception 'StompException' with message 'Could not connect to datafeeds.networkrail.co.uk:61618 (10/10)' in /<path>/Stomp.php:168
Stack trace: 
#0 <path>/Stomp.php(190): Stomp->_makeConnection() 
#1 <path>/stomp_test.php(7): Stomp->connect() 
#2 {main} thrown in <path>/Stomp.php on line 168

stomp_test.php is using what is suggested in the Developer Pack V4.0.

Are you adding the tcp:// prefix to datafeeds.n...?
 

dooton

Member
Joined
18 Nov 2010
Messages
64
Ok so i've just got some JSON back. Tried putting the train ID and schedule id that it returned into open train times and it wasn't found? It was just for fun to see what I had gotten back..

The programming I am ok with, understanding the railways inner workings could be trickier than i thought.
 

ralphchadkirk

Established Member
Joined
20 Oct 2008
Messages
5,764
Location
Essex
Yep, I sure am.

That's me out of suggestions then!

For reference, this is the quick 'n' dirty code I'm using just to try and extract a JSON message to see if it all works (unfortunately, it doesn't yet).

PHP:
<?php
set_time_limit(120);

    // include a library
    require_once("Stomp.php");
    // make a connection
    $con = new Stomp("tcp://datafeeds.networkrail.co.uk:61618");
    // connect
    $con->connect($user,$pass);

$con->subscribe("/topic/TSR_LNE_S_ROUTE",array('transformation' => 'jms-map-json'));
$msg = $con->readFrame(); // Code stalls on this line
if ( $msg != null) {
        echo "Received array: "; 
        print_r($msg->map);
        // mark the message as received in the queue
        $con->ack($msg);
    } else {
        echo "Failed to receive a message\n";
    }
?>
 

ian13

Member
Joined
17 Jul 2008
Messages
166
My advice in PHP is to completely ignore the developer pack.

First, install the PECL STOMP package (just type "pecl install stomp" in the command line).

Then, use code such at the following:

PHP:
<?php
// Network Rail Stomp Handler example by ian13
$server = "tcp://datafeeds.networkrail.co.uk:61618";
$user = "username";
$password = "password";
$channel = "TRAIN_MVT_ALL_TOC";

$con = new Stomp($server, $user, $password);
if (!$con) {
    die('Connection failed: ' . stomp_connect_error());
}
$con->subscribe("/topic/" . $channel);
while($con){
	if ($con->hasFrame()){
		$msg = $con->readFrame();
		foreach (json_decode($msg->body) as $event) {
		  // do stuff with $event here
		}
		$con->ack($msg);
	}
}
die('Connection lost: ' . time());
?>
 
Last edited:

ralphchadkirk

Established Member
Joined
20 Oct 2008
Messages
5,764
Location
Essex
Ah, interesting, I can get the RTTPM feed with my code, but not the TSR feed. Haven't tried anything else yet.

I'll try your code in a minute Ian.
 

DynamicSpirit

Established Member
Joined
12 Apr 2012
Messages
4,907
I deliberately steered the platform down the Stomp route because it's a simple, neutral standard.

Have you looked at http://activemq.apache.org/nms/ for your .NET requirements?

Thanks for the link. That looks potentially useful. I haven't looked at that side yet because I'm still trying to figure out how to download the schedules (getting what I'd assume is the easy bit out of the way first).

Out of interest, you say 'I'. Does that mean you had some influence in the choice of platform? (And no problem about the neutral standard. I totally understand that you need something that can be accessed from any platform).

On the schedules, do you need to have a your own Amazon S3 account set up to download them? I don't and I had assumed that was fine since the URL to download schedules is a networkrail one, not an amazon one, but now wondering if that's the problem. I see one of my download attempts has caused a page redirect to https://nr-datafeed-cif.s3.amazonaws.com/.
 

Paul Kelly

Verified Rep - BR Fares
Joined
16 Apr 2010
Messages
3,696
Location
Reading
Ok so i've just got some JSON back. Tried putting the train ID and schedule id that it returned into open train times and it wasn't found?

Timetable sites such as OpenTrainTimes etc. use codes from the train planning systems, i.e. TIPLOC codes for locations (e.g. HTHRGRN for Hither Green, OXFD for Oxford) and UID codes for individual trains (e.g. C01238, G26384, etc.).

The real-time running data is derived from TRUST/TOPS and uses a completely different set of codes, i.e. STANOX codes for locations (e.g. 88451 for Hither Green, 74446 for Oxford) and TOPS train codes for individual trains, made up of part of the STANOX code for the origin station, the train headcode, and a couple of codes to indicate the rough time of day and date the train started its journey (e.g. 861M58MP01, 121O32MV01 etc.).

Translating between these two sets of codes is a whole big job in itself!

Some good information on codes here and here.
 

DAH

Member
Joined
17 Apr 2008
Messages
19
Have tried Ian's way, gone is the 'cannot connect' message and appears to connect but then throws another exception :

Socket connection hasn't been established

Confused :cry:

Update, just altered my password and get the same 'socket' message, so guess it isnt connecting afterall.
 
Last edited:
Status
Not open for further replies.

Top