For a time, there was a disconnected freight railway between Bristol Docks (Hotwells) and Avonmouth. This was when larger ships started unloading at Avonmouth rather than come up the Avon, but the warehouses etc were still at the original docks. The trackbed through the gorge later became the southern part of the Portway road, after Avonmouth was instead connected via Clifton Down and tunnel to the main line at Narroways Junction. This new link picked up the old route just south of Sea Mills and today is the Severn Beach branch.
As someone else said above, the early railways were often built in working isolated pieces (often by small independent companies) bought up and joined up later. The Bristol and Exeter railway was originally separate from the GWR (and anything else) with the two Bristol terminii yards apart at right-angles, hence the present day curvature of Temple Meads which joined them. Often the joining up of pieces was the intention from the start, but I don't think that was the case for the the Hotwells-Avonmouth railway, as connecting it meant some serious tunneling not originally contemplated at all.