Exactly right. Rail is very much a niche market across swathes of the US, whereas in the UK it's a realistic option for many journeys, due to population density, shorter distances, and crowded roads.
I think the more relevant issue in the US is that rail is never seen as part of the solution, except in Mega-Cities and areas known for their green credentials. California HSR has been through the works and is now finally getting built, but prior to that San Francisco to Los Angeles was car or flight only and California is one of the greenest states (despite the never-ending traffic jam that is Los Angeles). The distance between the two cities is 383 miles, less than London - Edinburgh and in the UK and much of Europe, a rail line between a metropolitan area of 13 million people and another of 4 million people is a no brainer.
No-one's realistically suggesting that people take the train from New York to Los Angeles, but for the smaller "intermediate distance" journeys, there's a case to be made for rail usage that isn't being made (with the notable exception of the North East Corridor).
Looking at the top 5 US cities by population* and their respective rail networks (I'm ignoring the Amtrak 1-2 tpd services as effectively useless at providing a service, they're basically railtours):
New York
Significant metro network (NY Subway and PATH), but realistically only served by 3 commuter railway lines (Metro-North, LIRR and NJ Transit) and one intercity mainline (NEC). While I appreciate that Metro-North, LIRR and NJ Transit have branches, that's also the same for the 12 or so lines into London. (GW, Chiltern, WC, Midland, GN, WA, GE, LTS, SE, Chatham, LBSC, SW). Especially when you consider distances.
Los Angeles
Metro Rail is about the size of Manchester Metrolink and Metro-Link (the LA commuter rail) has about 5 lines in total. There are no significant intercity services.
Chicago
Metra and the L are a decent sized commuter network and metro system. Similar to New York, relative to the size of the Metropolitan area. Chicago, being Amtrak's "hub" is probably the only place with a non-trivial amount of intercity rail outside of the North-East Corridor. The interesting thing is that there were plans to create a "Chicago Hub Network" of High Speed Rail since 1991 (the latest an SNCF proposal in 2009), which have since been shelved, because politician decided that returning money to the federal reserve was more important.
Houston
METRORail is a Tram system, otherwise, the Sunset Limited was the only rail in the Houston area. This has been suspended since Hurricane Katrina. There's a proposal for a high-speed-rail link to Dallas.
Philadelphia
SEPTA has a decent commuter railway service, a 2 line metro system and a "Trolley" system about the size of Rome's, Philadelphia is also on the NEC.
The thing is, it's not like there aren't journey opportunities there, either to other nearby major cities, or within the cities themselves. There's just a sense in the US (and you certainly get this talking to people), that public transport is what you take if you absolutely have to, not because it's ever the best option. The reason New York has an actually decent rail network is because it's basically impossible to drive into New York in the morning peak or out again in the evening. Even things like the LIRR and Metro-North have awful off-peak services, because it's expected people will just drive instead. Just look at Houston, the 4th biggest US city by population, has practically no public transport at all.
*Yes, I'm aware the metropolitan area is probably a better indicator, but it's too late now, the results come out similar..........