When I worked in a petrol station, we occasionally had instances where we needed to swipe cards. This was most commonly men over the age of 70 who couldn't work out which way to put their card into the machine (if it can't read the chip three times it defaults to swiping), but there were instances where the chip couldn't be read or, of course, the connection to the bank went down. We had a procedure which was very simple.
I always amazed how many people had not signed their card (rendering it invalid in the first place, even for chip&pin transactions), and how many people had written the PIN on the back (secure). More amazingly, lots of people were using cards with a name belonging to someone of the opposite gender. They'd invariably claim it was their wife's card and/or that they had a joint account (and it probably was) but I still had to decline it. I have no evidence to prove this. When you see a man with a beard signing his name as "Lucy", you have to be concerned of fraud. And it is fraud. Entering the PIN or signing your name is verifying your identity, and using someone else's PIN or signature is a form of fraud.
I'd be very concerned if my card was authorised without any signature check and would complain. It puts me at risk of fraud. It's not a perfect system, but in the fallback scenario it is the best thing we have to ensure that the cardholder's account is safe.