Physical fitness and body weight are key correlates of health. Nevertheless, an increasing number of children display poor physical fitness and high body weight. The aim of this study was to examine the prospective association of physical fitness with body weight throughout the elementary school years with a special emphasis on children with high body weight or poor physical fitness at baseline. A total of 303 Austrian children (55.1% male) completed the German motor test up to eight times over a 4-year time span (between the ages 6 and 10 years). Physical fitness did not differ across quartiles of body weight at baseline. A more pronounced weight gain, however, was associated with an impaired development of physical fitness and this association was more pronounced in children with higher baseline body weight. In addition, the detrimental effects of an impaired development of physical fitness on subsequent body weight were more pronounced in children with higher baseline body weight. No differences in the longitudinal association between body weight and physical fitness, on the other hand, were observed across quartiles of baseline fitness. These results emphasize the importance of the promotion of physical fitness, particularly in children with increased body weight, to ensure future health.