For anyone keen to minimise bus travel (at the expense of spending time waiting for connections), you can go by train as far as Elblag, where the bus station is just outside the railway station. I think Braniewo also has a train service, but not from the Gdansk direction.
On the Russian side, the German version of Wikipedia has a
link to a timetable page indicating two trains per day (on working days, I think, although I'm open to correction by people who can read Russian properly) from Mamonovo to Kaliningrad, but only one in the opposite direction (too late in the evening to get the bus to Poland). (Even if you don't read Cyrillic, you should be able to recognise Kaliningrad and Mamonovo as K and M are the same as in our alphabet).