From recent experience, I can recommend the following lines:
- Hakone Tozan Tetsudō-sen (Hakone-Yumoto to Gora) - literally meaning 'mountain-climbing line', this goes up a valley with 3 reversals and gradients of up to 1-in-13, so it's one of the steepest non-rack railways in the world. There's a variety of older rolling stock, including a Swiss set that dates back to the opening of the line in 1919. At the top of the line there's a funicular (confusingly called a "cable car") which connects onto a cable car (confusingly called a "ropeway") which takes you to the shore of Lake Ashi, with great views of Mount Fuji on clear days.
- Oigawa Railway (Kanaya to Kawane Onsen Sasamodo) - this acts as both a heritage railway as well as public transport. Most trains are, I believe, operated by nice old single-car EMUs - but some of the 'express' workings are simultaneously steam hauled and electric propelled - a first for me! The timetable is quite variable, so worth checking in advance. The railway was split in two after storm damage a few years ago, so you have to take a local bus to connect between the two halves. You can take the Shinkansen either from Tokyo to Shizuoka, or Kyoto to Hamamatsu, to connect onto the local JR train which takes you to the interchange station of Kanaya. From there it's a 20 minute walk to Shin-Kanaya, the railway's HQ - but some of their trains run to/from Kanaya.
- San'in line (Kyoto to Ayabe and beyond) - this runs along a river valley from Kyoto towards Ayabe and is surprisingly scenic. It's served by Hashidate express services, which have 2+1 seating in Green Car (i.e. first class), and split and join at Ayabe to serve both Higashi-Maizuru and Fukuyichima/beyond. Some of the branch lines in the Ayabe and Fukuyichima areas are operated using old class 115 EMUs, which have largely already been withdrawn; we happened to come across one on a stopper from Ayabe to Fukuyichima.
There are no doubt many others that are worth doing too. There's a great site here:
https://bateman.monologue.jp/en/index.html which lists the types of trains used by each railway company across Japan, ranking them by their 'endangered' status
We used a Japan Rail Pass in Green Class - despite us not making optimal use of it (e.g. we used it for local travel around Tokyo on two of our 7 days) and the substantial price increases a few years back, it ended up working out about 20% cheaper than buying tickets separately for each journey.
Even if it had worked out more expensive, I wouldn't have minded it as the flexibility it offers is highly valuable, and you can make reservations online with a seat selector. If you do this, I highly suggest only collecting each reservation just before you board the train, as it's a pain to get them changed once they're issued. On one occasion we asked the conductor on the platform whether we could board her train (since it was too close to departure to get a reservation online), and she gladly issued reservations on the spot, so that's a backup option if you have a last-minute change of plans.
You probably already know this, but the easiest way of paying for local transport around cities, as well as small purchases in the ubiquitous convenience stores and vending machines, is using an IC card such as Suica, Pasmo etc. These are effectively like an Oyster card - you load them with money and then each journey or purchase is deducted from the balance. Unfortunately, topping up the card is only possible with cash - although most stations and convenience stores have 7-Bank ATMs that don't charge a fee. Don't worry about looking up the fare in advance (it'll be affordable) or having the correct balance when you start your journey, as you can always just top up your card at the end of your journey using an 'adjustment machine' (located inside the barriers).
IC cards were very hard to get hold of when we visited, although it should now be a bit easier. You can get a Welcome Suica card at the JR exchange office which they have in the landside area at each of the major airports (Haneda/Narita etc.). You can do this when you pick up your Japan Rail Pass. The card itself is free but the balance and card expires after 30 days, and can't be refunded, so make sure you use it up before you leave!