I'm a daily commuter on the ELLX (ex-South London Line), so I'll wade into this discussion..
Presumably due to slow line speeds in places and having to slot in between other services and freight. I know what you mean though - the slowest-feeling part I've experienced is Clapham Junction to Wandsworth Road which usually takes me 7 minutes, sometimes 10 minutes for what is a little over a mile!
[*]Why are the trains so poor & short?
I don't think that they are particularly short, given the stations they serve. They're built with one thing in mind: capacity. The greatest capacity is achieved by making people stand. I don't know about the NLL/WLL but on the ELL, the northbound trains going into Canada Water are totally wedged and they just wouldn't be able to fit all those people on with a conventional layout.
The side-effect is that on the edge of peak there are not enough seats, whereas on a conventional layout there would have been. Also, the layout doesn't feel as "cosy" as a conventionally-laid out train. The seats themselves however are amongst the most uncomfortable I've ever sat on - very low back with little support and virtually no padding. They are very similar to the Victoria LU seats (made at the same factory maybe?).
The trains accelerate very slowly out of stations, and it takes a very long time for the doors to be unlocked after entering any station. I'm sure this contributes to the feeling of a slow journey compared to a London Underground train which would brake and accelerate much sharper and have much quicker door unlocking.
I personally think the stock has some deficiencies, given that it was built specifically for the LO metro service, but I can see why the decisions were made the way they were.
[*]Why is the frequency so poor?
I don't think there really is demand for a more regular service. Also, I suspect at the pinchpoints, such as Surrey Quays on the ELL, the line is pretty much near or at capacity. On other parts of the network, there are other TOCs services to fit in too.
And of course a lot of it is Boris and politics...