Most people using London taxis are not disabled or of limited mobility. Exceptions could be made for such cases, perhaps controlled by showing an appropriate Freedom Pass or ENCTS card if a Police Officer or parking attendant questioned the matter.
The problem is, if you ban taxis and PHVs for all but disabled/limited mobility people the result will be a massive decrease in the number of vehicles in service. This results in increased waiting time for a taxi/PHV to arrive, and realistically you'd need to forget hailing and have a booking only system. At that point you effectively have an expanded Dial-a-ride scheme, the cost of which falls mainly on the public, and provides a service that many disabled/limited mobility people would object to using. In a sense it would be like going back to the days of the "invalid carriage" where disabled people weren't meant to expect their travel needs to be provided as part of the mainstream system.
I don't know the stats for the percentage of taxi/PHV trips made by disabled people. But London does have the Taxicard scheme that provides people with certain mobility issues a discount on taxi journeys. The details vary by borough but it amounts (IIRC) to a discount of £8.50 per trip (one way) for 104 trips per year (so one two-way journey per week). £8.50 takes you about 2.5 miles. The budget for London Taxicard is about £11 million per year - so allowing for some admin costs it is somewhere in the region of 600,000 (two-way) subsidised journeys. Obviously one journey per week per person (and only for those eligible for Taxicard) is only a small fraction of the total number of taxi/PHV trips made by disabled and limited mobility people.
Briefly - and only to illustrate how a simple idea can be more complex in reality - the enforcement of bus lanes has nothing to do with the police. Parking attendants can only deal with parking contraventions in bus lanes (dropping a passenger doesn't count). Bus lanes are usually enforced by camera with a PCN sent in the post. This means a taxi/PHV driver would not have the passenger's documents to hand when the PCN arrives. So drivers would need to take copies (possibly a photo) of the passenger's entitlement for every trip, just in case they get a PCN. That could be a data protection/privacy minefield.
This is a problem even under the current arrangements. London Travelwatch did some work a few years ago on the problem of people not being able to get a PHV to pick them up because there was a bus lane in front of their home. The reason being was that if the passenger didn't have proof of disability, or was a 'no show', then the driver would have no proof they entered the bus lane and stopped only to pick up a disabled passenger. It is a very difficult and complex problem to resolve.
However, even then there is a need for common sense in selecting where to stop which is inevitably not displayed, e.g. stopping such that a bus will get stuck the wrong side of a traffic island. People with disabilities and those driving them around are not absolved from acting reasonably to others - that's why, for example, parking using a Blue Badge in a way that is causing an obstruction can still result in a ticket.
Obviously it would help if common sense were applied, but if walking more than 10m is extremely difficult for someone then dropping them 20m from where they need to go is not a good idea.
Usually a drop off/pick up is only going to take a few minutes. Parking with a blue badge might involve a long-stay, possibly with the vehicle left unattended. Hence parking would be dealt with as an obstruction, whereas the drop off/pick up is more like delaying a train by a few minutes in order to help a wheelchair user off with a ramp. Very few rail passengers would begrudge this delay - road users should be willing to be equally patient.
I should perhaps have said "nobody without a disability needs to use a taxi in London".
Fair enough, obviously including all forms of disability including invisible ones.
I would add to the list people who need to use a taxi because the nature of the trip is not suitable for public transport. For example a group of business people travelling together for a meeting and needing to have a confidential conversation on the way (I've done just that with a barrister on several occasions, not conversations you can have on the tube). Alternatively, people travelling with high-value items (e.g. jewellery, cash). Also there are circumstances where bulky or heavy items need to be delivered very quickly and a taxi/PHV provides a more efficient way of doing so than hiring a courier. I'm sure there are many more good reasons other people could think of.