Pretty much in agreement with that. Anecdotally it always seems to me that it appears to be fairly common that a driver who actually does 30mph in an urban area (even when appropriate) has fingers metaphorically pointed at them by people on the pavement who think they're going too fast. The fact is that 30 is actually sometimes not a common speed in certain towns and cities, and even if it is legal and technically safe to go at that speed, even that relatively low limit can seem pretty fast to vulnerable road users. This is with good reason, as an impact with a pedestrian at a speed of around 30 is going to be at the very least extremely painful.
Country lanes, wet roads and such like have all been pointed out by yourself and others - quite correctly, it is the case that a "one size fits all" approach to these roads is inappropriate. Regrettably it often seems that "derestricted" 60mph rural roads are so often left at that speed simply because the infrastructure needed to signpost and enforce a lower speed limit, which would actually be safe, is not deemed to be a sensible use of money. The trouble is that it is entirely legal to drive at a stupid speed down many twisting, single-track, muddy, dark country lanes, when an equivalent road with a few more houses may have a speed restriction running 20 or 30mph less. In this way it seems that the law almost tempts those stupid enough to try to use it as a target. If the limit was 40mph by default for rural roads, of course there would be many who would think it's perfectly safe to nudge a bit past that, but there would be the opportunity for a much firmer line to be taken against those who can now use it as some sort of cross-country demolition derby speed target.
The only issue with this is how to legislate some roads which could genuinely have higher safe speeds than, say 40mph, but would otherwise fit into the single carriageway bracket. If 40mph was the default limit for rural roads then raising that to 50 or 60 would need to be easy on a case-by-case basis once evidence was available that the road would be capable for such traffic. There could also be a need for new signage and so on, which wouldn't necessarily be useful.
The point, though, still stands that for many locations there is nothing by law to state that somebody cannot drive past a concealed bridleway on a country lane doing 60mph, approach a major hazard such as a busy junction, etc. etc. and this means that there is one fewer way to dissuade people from doing stupid things. The only alternative is a large grant to every highways authority in the land, allowing them to monitor safety and signpost new limits on every stretch of road where the, ahem, "derestricted" speed limit of 60mph is too high, but the cost of redoing this is so far prohibitive. And that's not going to happen!