Firstly, if they're buying advance tickets, they'll save extra thanks to their railcards (a 50% rather than 33% discount) by buying through
http://www.tpexpress.co.uk/train-tickets/student-discounts/ - they need to go through that link to get the discount.
Doing that, then, I'd have though the cheapest AND fastest option would be to get a Transpennine express advance ticket to Scarborough: there are a number of direct trains a day. Picking a random Friday in July (13th) there are currently advance fares at
£6.45 each way. They can travel for £15.20 each way first class too.
Simply follow the
National Rail booking horizons page and check when the desired dates of travel come up. Reinforce to them that they
must get the booked train - otherwise their tickets are invalid!
After that, the
Arriva 93 bus is fairly regular between Scarborough and Whitby. Not sure of the cost but I can't imagine it being too much. Even if it costs them a fiver each way, that's a total return price of around £22.45 if they can get those cheap advance tickets.
Edit -
This thread suggests that the bus will be £5 each way. Given that there's four of them, a taxi might prove better value - and it would have the advantage of getting them to Scarborough station reliably on time for the return journey! Splitting with advances at Middlesborough is actually a little cheaper, but the Scarborough option should be faster. There's also a half hourly bus service, compared to 4 trains a day!