Thanks for all your replies. I'm not sure I was clear enough in my original post. I want to go out via Hull to access Bempton and then return direct from Scarborough. From the replies it looks like I need a via Hull for the way out but then can't come back direct (via York) and I'd have to return (very slowly) via Hull.
If you're willing to limit yourself to trains that call at Gilberdyke on the first part of your outwards journey from Doncaster to Hull (e.g. 1005,1205 Mon-Sat) then I think this might work:-
(1) £8.30 Doncaster-Gilberdyke Offpeak Day Single (route: not via Leeds/York )
(2) £21.40 Gilberdyke-Malton Offpeak Day Return (route: Any Permitted)
Total £29.70
Stop short at Gilberdyke on ticket (1) and immediately start to use (2). Break outbound at Bempton, Resume outbound at Scarborough, switch to return half at Malton and stop short at Doncaster. ( This ticket is valid via Hull/Scarborough and also via York/Doncaster. )