Diesel engines do not inherently provide engine braking, unlike petrol engines. In petrol engines the braking effect is provided by the closed throttle butterfly valve restricting airflow into the engine. Diesel engines do not need a throttle valve. It is incorrect that compression of air in the cylinders provides much braking effect - the energy absorbed in the compression stroke is mostly released in the expansion stroke, like a bouncing spring. Therefore some automotive diesels are equipped with artificial engine braking (e.g. "Jake brake") for safety reasons.
If a DMU engine is kept in gear during coasting, there will be some decelerating effect from friction and pumping losses in the engine, plus the hotel loads on the engine (aircon compressor and alternator). But the engine will consume no fuel at all unless the speed drops sufficiently that the driver needs to reapply a short burst of power. On the other hand, if the transmission allows freewheel coasting, the deceleration rate will be a bit less, but the idling engine is burning fuel all the time to overcome losses and provide the hotel power. If power needs to be reapplied, more fuel is wasted accelerating the engine back up to speed.
I should think that it would depend on the route profile (gradients, speed restrictions, distance between stops etc.) which strategy is more economical, but my guess is that coasting in gear would win overall.