A positive relationship between handlers and animals in farm systems is essential because the human-animal relationship has implications for welfare and productivity. For this reason, on-farm animal welfare assessment protocols often include the behavioral response of animals to humans to measure the quality of the human-animal relationship. The existing literature has described this relationship as being multifactorial in nature. In the current study, we aimed to investigate the potential influence of farm management and infrastructure characteristics, calf manager traits, and intrinsic features of dairy calves on the human-animal relationship. To this end, an escape test was conducted with 698 calves on 30 dairy farms in Chile. This test measured the calf's response to the active approach of an unfamiliar human (and was scored from 0 [fearful] to 4 [friendly]). The explanatory variables used to predict calves' response in the escape test were grouped according to the following categories: (1) farm management and infrastructure (e.g., calf-dam separation age, space allowance); (2) calf manager (e.g., attitudes, behavior, and background); and (3) calf (e.g., breed, sex, age). We concluded that calf managers with additional jobs on the farm, no training, low job satisfaction, a greater proportion of negative contacts, and more negative attitudes were predictive of fearfulness in the escape test. Holstein breed (compared with Holstein and Jersey crossbreeds) was associated with greater odds of fearful calves. Our study confirms the association between animals' fear and handlers' features, which can potentially be used to select employees on a farm. Understanding the factors that influence fear responses in calves may highlight ways to improve the relationship between animals and humans.