The two stations are in fact one, but the platforms are separated by an enormous building that used to be for passport and customs checks.
I travelled on a through ticket so can't say - would have thought it probable rather than only a machine.
According to the Rodalies website there are penalty fares, if I understand its rather poor English - puzzling, because I recall there being unstaffed halts near Port Bou, so there must be a conductor.
http://www20.gencat.cat/portal/site/rodalies?newLang=en_GB
Others on here are better informed about Spain....wait for them to reply!
I've no idea how punctual the sleeper is, but if you miss the connection you'll have a 2-hour wait and Port Bou isn't a hive of activity!
If it's not too late to suggest it, you could consider travelling on the La Tour de Carol portion of the same train, which has a more comfortable connection (and definitely pay on train, according to Seat61) and is spectacular scenery (though the Cote Vermeille bit is pretty good too).