This paper analyzes the impact of colonialism on the ways in which subaltern populations establish relations of sameness and otherness with their ancestors and with archaeological vestiges in their surroundings. From a perspective that questions the dominant dichotomous and structural interpretation that conceived the logic of otherness as a product of intrinsic community dynamics, this study reveals the central role of historically specific power relations in the articulation of identity configurations. This is revealed in the comparative analysis of the northwest region of the Iberian Peninsula, the Bolivian highlands, and the Chilean Atacama region. The populations of the three regions present animistic beliefs in mouros, chullpas, gentiles and abuelos, which they associate with archaeological remains under a common logic. Starting with the Christian struggle against paganism in Europe and the extirpation of idolatry in America, this article demonstrates how enlightened modernity, nationalism and contemporary multiculturalism have led to different configurations of patterns of identity and alterity, as well as alternative notions of ancestry in each case. Archaeology has played a prominent role in these processes, subordinating local conceptions of otherness and partaking in their heritagisation.