This study aimed to examine the association between the neighborhood environment and domain-specific physical activity and sitting time in Brazilian adults. This cross-sectional study included 1803 adults (53.7% women) from Brazil’s five regions (North, Northeast, Midwest, Southeast and South). The perception of the environment was evaluated via a questionnaire. We considered seven indicators of the neighborhood environment: land use mix-diversity, land use mix-access, street connectivity, walking/cycling facilities, aesthetics, safety from traffic and safety from crime. Using the IPAQ, we evaluated physical activity (transport and leisure) and sitting time. Overall, land use mix-diversity (β = 3.22; 95% CI = 0.26, 6.19), land use mix-access (β = 2.27; 95% CI = 0.76, 3.79), and walking/cycling facilities (β = 2.42; 95% CI = 0.35, 4.49) were positively associated with leisure-time physical activity (min/week). On the other hand, only land use mix-diversity (β = 3.65; 95% CI = 0.63, 5.49) was positively associated with transport physical activity (min/week). No neighborhood environment indicator was associated with sitting time (min/day). Perception of the neighborhood environment was associated with physical activity (transport and leisure), while no significant associations occurred with sitting time among the five regions of Brazil. The neighborhood environment can be a viable component in the promotion of physical activity, but geographic diversity must be considered.