Being tossed around due to weight of the bullet has nothing to do with it.
It is all based on time of flight.
Heavier bullets tend to have a higher BC, so maintain their velocity, therefore shorter time of flight.
Not true. If you calculate based on TOF, a heavier bullet has less wind drift than a lighter bullet. It is less about ballistic coefficient than it is about projectile mass for drift.
A projectile that has twice the mass has about 1.42x the surface area for wind to act upon than the lighter projectile. Increasing the wind force on it. Yes, time of flight has some impact also, but mass absolutely has a large role to play in wind drift.