I was always taught that racism is ascribing traits or characteristics to an individual based on preconceived notions of the traits or characteristics associated with their ethnicity. So, for example, assuming that a black man will do well in long-distance running. So it doesn't necessarily mean thinking that someone is inferior (though it usually does).
I would say that your idea is somewhat correct since it is making assumptions based on race, but it's quite funny actually, because there are certain traits you can apply to someone based on their skin colour because of genetics. Lighter skin tends to preserve Vitamin D better thanks to the evolution of living in a climate with less sunlight than our ancestors, but darker skin tends to fare better in high sunlight environments, so I don't think you would be wrong to guess a black man would be able to live in a high sunlight environment better than a white man and such. But since it still technically is ascribing traits and characteristics to someone based on skin colour, is it still racist despite being scientific? I think that's an interesting question personally.
Not every applied trait is genetic of course. If black people are actually more athletic for example, then I'd put that down more to cultural practices than genetic. Raise two people of a different race in the same conditions of living where lots of walking and running is required, I doubt skin colour will play any part of it. It's not just them two either, there is actually a posture in the third world known as the Asian Squat, Slav Squat or Third World Squat etc, and some have been lead to believe it could be a genetic thing due to the fact most white people cannot do it. However, this is most certainly cultural, because western nations (which are mostly white) use chairs and sit toilets, and so the necessary muscles tighten to the point we can't squat without raising our heels (though as a martial artist, I can do the posture), while Asian and Slavic nations use squat toilets and keep the habit.
I personally think my question raises a decent point though because as a liberal, I wouldn't usually prescribe someone a trait or characteristic just because of skin pigmentation, but if you had me guessing certain things such as who would handle high or low sunlight conditions better, then I know who I'd put my money on. But they are both still prescribing traits to someone, only in one case it's got a scientific basis, so again would that count as racism despite no malicious intent? I do think it certainly is an interesting question.