Socio-economic status (SES) and mathematical performance seem to be risk factors of mathematics anxiety (MA) in both children and adults. However, there is little evidence about how exactly these three constructs are related, especially during early stages of mathematical learning. In the present study, we assessed longitudinal performance in symbolic and non-symbolic basic numerical skills in pre-school and second grade students, as well as MA in second grade students. Participants were 451 children (average pre-school age = 5 years, 6 months) from 12 schools in Chile, which differed in school vulnerability index (SVI), an indicator of SES. We tested an explanatory model of MA that included SES and longitudinal performance in basic numerical skills as predictors. The results showed a direct effect of SES on MA and a mediating effect of performance in symbolic and non-symbolic comparison tasks in pre-school. However, in second grade, only performance in symbolic comparison significantly mediated the SES-MA relationship. These findings suggest that performance in non-symbolic comparison plays an important role in explaining MA at initial stages, but that its influence is no longer significant by the time children reach formal instruction in second grade. By contrast, as children's formal educational experience in mathematics increases, MA becomes linked primarily to symbolic numerical tasks. In sum, SES affects MA and this is due in part to the effect of SES on the development of numerical learning in pre-school, which in turn has an impact on subsequent, more complex learning, ultimately leading to differences in MA. We discuss the implications of these findings for preventing and acting upon the emergence of MA.