• Our booking engine at tickets.railforums.co.uk (powered by TrainSplit) helps support the running of the forum with every ticket purchase! Find out more and ask any questions/give us feedback in this thread!

Network rail data feeds

Status
Not open for further replies.

najaB

Veteran Member
Joined
28 Aug 2011
Messages
30,691
Location
Scotland
Has anyone tried playing around with NR data feeds? If so, do you have any code you can share to get the schedule data (including daily updates) into a database? It looks like quite the job between association records and schedule records so no point reinventing the wheel if there's anyone with shareable code.
 
Sponsor Post - registered members do not see these adverts; click here to register, or click here to log in
R

RailUK Forums

trainmania100

Established Member
Joined
8 Nov 2015
Messages
2,566
Location
Newhaven
I have access to the data feeds but haven't had time to work out how to use it
Maybe I'll use php never used Ruby
 

najaB

Veteran Member
Joined
28 Aug 2011
Messages
30,691
Location
Scotland
I have access to the data feeds but haven't had time to work out how to use it
Maybe I'll use php never used Ruby
I've managed to decode the movement data and encode them into a KML file, but that only lets me see what train is where. I'd need the schedule to figure out where it's going to/from.
--- old post above --- --- new post below ---
Oh, yes, I've been using PHP. The examples here are a good place to start.
 

najaB

Veteran Member
Joined
28 Aug 2011
Messages
30,691
Location
Scotland
Have you been integrating it with HTML?
Yeah. I've a small server that I play around with. The script is running just now so you should be able to see Scotrail's trains here. A couple of things I've noticed, some locations don't have lat/long data so trains appear to materialise somewhere in the Gulf of Guinea(!) and without schedule data they don't always disappear when they reach their destinations.

It's very much just a proof of concept at this stage.
 

trainmania100

Established Member
Joined
8 Nov 2015
Messages
2,566
Location
Newhaven
Yeah. I've a small server that I play around with. The script is running just now so you should be able to see Scotrail's trains here. A couple of things I've noticed, some locations don't have lat/long data so trains appear to materialise somewhere in the Gulf of Guinea(!) and without schedule data they don't always disappear when they reach their destinations.

It's very much just a proof of concept at this stage.
It's looking great , i like how it uses the maps and shows the trust ID
 

Kristofferson

Member
Joined
23 Nov 2012
Messages
1,132
I used the RTT API instead of the NR feed to build a personalised departure board with Angular & Bootstrap. Much easier to understand and does the job!


Sent from my iPhone using Tapatalk
 

najaB

Veteran Member
Joined
28 Aug 2011
Messages
30,691
Location
Scotland
I used the RTT API instead of the NR feed to build a personalised departure board with Angular & Bootstrap. Much easier to understand and does the job!
I did consider the RTT API, but I doubt they'd appreciate a full download of the entire schedule daily! :)
--- old post above --- --- new post below ---
It's looking great , i like how it uses the maps and shows the trust ID
If I can get the schedule data figured out, my next task is to show the calling points when you click on a train. It's completely a spare-time project but I'm in hotels for the next month or so - which means lots of spare time!
 
Last edited:

Peter Mugridge

Veteran Member
Joined
8 Apr 2010
Messages
14,751
Location
Epsom
Why is 2T73 currently showing in the Gulf of Guinea off Africa? Ah - I know... it's 156 478 gone for another swim... ;)







Sorry - couldn't resist it; I did see the actual reason above.
 
Last edited:

najaB

Veteran Member
Joined
28 Aug 2011
Messages
30,691
Location
Scotland
Why is 2T73 currently showing in the Gulf of Guinea off Africa? Ah - I know... it's 156 478 gone for another swim... ;)
I got to the bottom of it - not the Atlantic! - there's 1800 or so locations in the TIPLOC file that don't exist in the list of geographic coordinates. I've managed to find a few of them (quite a few are on the Borders Railway), but it'll take a while to find them all since they have useful, meaningful and easily identifiable names like "PEAK FOREST CRIPPLE SIDING" "PLYMOUTH SURPLUS" and "COAL CONCENTRATION DEPOT".
 

HA25322

Member
Joined
18 Aug 2014
Messages
344
Location
The Moon
Now if only someone could get Trust to download as well so we knew what was allocated to the services. Maybe one day. I will keep my fingers crossed that it may happen. However good job so far.
 

najaB

Veteran Member
Joined
28 Aug 2011
Messages
30,691
Location
Scotland
Now if only someone could get Trust to download as well so we knew what was allocated to the services. Maybe one day. I will keep my fingers crossed that it may happen. However good job so far.
Unfortunately, it isn't listed as an available feed. One can hope though. My plan for this week is to try and get to the bottom of the schedule feed, but I may need to spin up a new cloud compute instance as just one TOC's schedule looks too big for my poor little Raspberry Pi to handle. :eek:
 

HA25322

Member
Joined
18 Aug 2014
Messages
344
Location
The Moon
Unfortunately, it isn't listed as an available feed. One can hope though. My plan for this week is to try and get to the bottom of the schedule feed, but I may need to spin up a new cloud compute instance as just one TOC's schedule looks too big for my poor little Raspberry Pi to handle. :eek:

I didn't think it would be as it would be well known by now I would have thought. Shame though as it would make the hobby so much easier. :lol:
 

najaB

Veteran Member
Joined
28 Aug 2011
Messages
30,691
Location
Scotland
So, I've managed to get the basics of the schedule feeds worked out and into a database, but I've a couple of questions.

VSTP schedules - am I correct that they don't overwrite the permanent schedule so I don't need to delete anything from the database if/when I figure out how to read the VSTP feed (for some reason it's in a different format)?

Also, in the associations feed there seem to be a lot of almost duplicate records, does anyone have a simple explanation of how association records work? I have an idea in my head but want to see if I've got it right.

Thanks all.
--- old post above --- --- new post below ---
And another one for anyone who has played around with this stuff...

I've loaded the main JSON formatted snapshot into a database and I've got double entries for some schedules - is this my code or does the data contain duplicated entries? There's a few tens of thousands of lines in the file so don't fancy going through it manually.
 
Last edited:

HA25322

Member
Joined
18 Aug 2014
Messages
344
Location
The Moon
You can have several items under the 1 headcode. It may be WTT, the STP & VSTP. And VAR. VAR would be a slight difference to the original WTT. Eg timing issue due to engineering works that week. Or a slight tweak due to pathing constraints. I don't envy you at all. But good luck and best wishes.


Sent from my iPhone using Tapatalk
 

najaB

Veteran Member
Joined
28 Aug 2011
Messages
30,691
Location
Scotland
You can have several items under the 1 headcode. It may be WTT, the STP & VSTP. And VAR. VAR would be a slight difference to the original WTT. Eg timing issue due to engineering works that week. Or a slight tweak due to pathing constraints. I don't envy you at all. But good luck and best wishes.
It was partially my code, truncated the databases and started again and most of the duplicates are gone.
 

gsnedders

Established Member
Joined
6 Sep 2015
Messages
1,472
It was partially my code, truncated the databases and started again and most of the duplicates are gone.

FWIW, I'd be more than interested if you threw up the code somewhere and made it available. Maybe even help. :)
 

najaB

Veteran Member
Joined
28 Aug 2011
Messages
30,691
Location
Scotland
FWIW, I'd be more than interested if you threw up the code somewhere and made it available. Maybe even help. :)
You might not say that if you saw how I code! :) I'll see about setting up a repository somewhere when I get a chance.
--- old post above --- --- new post below ---
I got to the bottom of it - not the Atlantic! - there's 1800 or so locations in the TIPLOC file that don't exist in the list of geographic coordinates. I've managed to find a few of them (quite a few are on the Borders Railway), but it'll take a while to find them all since they have useful, meaningful and easily identifiable names like "PEAK FOREST CRIPPLE SIDING" "PLYMOUTH SURPLUS" and "COAL CONCENTRATION DEPOT".
I sent a list of missing locations off to NR few days ago, and they said they've requested a data update. Wonder how long it will take to get the file updated?
 

najaB

Veteran Member
Joined
28 Aug 2011
Messages
30,691
Location
Scotland
[RANT]Not surprised that NR has so many problems with maintaining their infrastructure, given that they don't know where it is! Pulling locations directly from the most recent planning data file and you get such oddities as Bounds Green being in Norway!!![/RANT]
 

The Planner

Veteran Member
Joined
15 Apr 2008
Messages
15,836
[RANT]Not surprised that NR has so many problems with maintaining their infrastructure, given that they don't know where it is! Pulling locations directly from the most recent planning data file and you get such oddities as Bounds Green being in Norway!!![/RANT]

In terms of train planning, it is pretty irrelevant to be fair. Anything that looks odd is probably just something that has been put in to fill a mandatory data field.
 

najaB

Veteran Member
Joined
28 Aug 2011
Messages
30,691
Location
Scotland
In terms of train planning, it is pretty irrelevant to be fair. Anything that looks odd is probably just something that has been put in to fill a mandatory data field.
I know it is, that was just a rant. For planning purposes it is the network links that matter - I've not taken that close a look at them (well over a million records!) but there wasn't anything that jumped out as obviously wrong.

What's odd is that some of the geographic errors are *so* close to being right - take a look at any service passing through Dundee for example, the points are correct for longitude, but about 1.5km too far south.
 

Attachments

  • Dundee.JPG
    Dundee.JPG
    41.8 KB · Views: 37

greatkingrat

Established Member
Joined
20 Jan 2011
Messages
2,750
It's not that odd, probably was just miskeyed during data entry, a difference of 0.01 degree in latitude would be about 1.5km.
 

The Planner

Veteran Member
Joined
15 Apr 2008
Messages
15,836
For planning purposes it is the network links that matter - I've not taken that close a look at them (well over a million records!) but there wasn't anything that jumped out as obviously wrong.

Oh there are plenty that are wrong believe me and that leads to crap schedules that miss timing points or put those in that aren't required.
 

najaB

Veteran Member
Joined
28 Aug 2011
Messages
30,691
Location
Scotland
It's not that odd, probably was just miskeyed during data entry, a difference of 0.01 degree in latitude would be about 1.5km.
Hmm... I wonder if it's an Eastings/Northings to Longitude/Latitude rounding error - I did try a couple different ways to do the conversion but the results were the same, but they might all be using the same underlying algorithm.
 

RailAleFan

Member
Joined
2 Jul 2014
Messages
314
Location
Midlands
I know it is, that was just a rant. For planning purposes it is the network links that matter - I've not taken that close a look at them (well over a million records!) but there wasn't anything that jumped out as obviously wrong.

What's odd is that some of the geographic errors are *so* close to being right - take a look at any service passing through Dundee for example, the points are correct for longitude, but about 1.5km too far south.

What do you have as Easting / Northing for Dundee? I took location data from NapTAN

https://data.gov.uk/dataset/naptan

Which for Dundee has;

Easting 340242

Northing 729793

...and converted to Lat / Lon as

Lat 56.45648450

Lon -2.97120452

...and seems to be pretty close...

http://live-departures.info/rail/station/Dundee/
 

najaB

Veteran Member
Joined
28 Aug 2011
Messages
30,691
Location
Scotland
What do you have as Easting / Northing for Dundee?
From the data set:
Code:
+------------------------+---------+----------+------------------+-----------------+
| LocationName           | Easting | Northing | Longitude        | Latitude        |
+------------------------+---------+----------+------------------+-----------------+
| Dundee Central Jn      |  340699 |   727969 | -2.9619003219444 | 56.440270689776 |
| Dundee Central Jn C.E. |  340500 |   728320 | -2.9652076338357 | 56.443398958257 |
| Dundee H.S.            |  340900 |   728450 | -2.9587492784205 | 56.444617140695 |
| Dundee                 |  341262 |   728157 | -2.9528117904044 | 56.442030086038 |
| Dundee Head Shunt      |       0 |        0 |  -7.556448482059 | 49.766185796754 |
| Dundee West C.E.       |  340250 |   728320 | -2.9692624737147 | 56.443367359206 |
| Dundee West T.C.       |  341200 |   728450 | -2.9538832949216 |  56.44465463208 |
+------------------------+---------+----------+------------------+-----------------+

So there's definitely something dodgy about the source data.

Edit: Forgot to mention that the Eastings/Northings are from the data source, the Lat/Long is my conversion.
 
Last edited:

najaB

Veteran Member
Joined
28 Aug 2011
Messages
30,691
Location
Scotland
Another VSTP question - how do you get the TOC? The atoc_code field is blank?!
 

najaB

Veteran Member
Joined
28 Aug 2011
Messages
30,691
Location
Scotland
Cross referenced from the train service code. May not always work.
I shall have a look at this - I'm not in front my PC at the moment but from your comment I guess there's a pattern to be derived. Thanks.
 
Status
Not open for further replies.

Top