I've walked round a depot in boots with fairly thick soles and it hurts, I hate to think what it's be like in trainers etc.
I'd be interested to have a go after hearing the comments about how hard it is. Ballast doesn't look particularly hostile for walking on, no worse than some of the ground when hiking in the Scottsh mountains, but as I have never walked on ballast I can't conclude anything. The only issue I can think of without trying for myself is the ballast will shift under my feet, potentially throwing me off balance.