I've had a full read of the PF regulations and there is nothing that mentions being allowed to charge one in circumstances whereby the ticket is otherwise valid, except for the time of day the journey is taking place. It only mentions things like being in the wrong class of accommodation, not having a railcard, not being entitled to a discount. Therefore it would be arguable that a PF is not valid if issued in these circumstances as the provisions in the NRCoT (i.e. the contract) should apply, i.e. they you merely may an excess fare.
Regulation 3(2)(a) defines what a "valid travel ticket" is. Subparagraph (i) thereof states that the ticket must be valid for "the day and time of the journey that person is making, has made or intends to make".
It's not clear whether this is merely intended to apply to tickets which can't be excessed - such as certain local authority concessionary passes or rangers/rover, or where no suitable fare exists to excess to* - or whether it's intended to apply to all tickets, including those that are valid subject to an excess being paid. The fact that a particular Off-Peak ticket isn't valid for travel at a particular time doesn't mean that a passenger is committing an offence under Byelaw 18, for example, so I would have thought that this is intended to mean the former - that only those tickets which can't be excessed are subject to a Penalty Fare.
*For example, if, for the journey made, there is an Anytime Day Return and an Off-Peak Return defined, but no Anytime Return - e.g.
Oxford to London - and the passenger uses the return portion of an Off-Peak Return on the second day of validity during a time where it is not permitted.