Activities developed under hazardous conditions, such as mining, require an active investment from the industry to avoid serious injuries and even loss of lives. One of the safety measures taken to address this problem is to implement a video surveillance system able to track vehicles and workers to enforce safety protocols preventing dangerous events that could lead to accidents. In this paper, we propose an intelligent video surveillance system able to detect and predict possible dangerous interactions between workers and vehicles operating on a monitored environment. Our system is able to analyze video information from one or several cameras, detects and tracks objects and emits an alarm that warns the operator on possible safety hazards. A multiclass object detection is trained to detect vehicles and people. The system object detection stage was implemented in CUDA for parallel processing to achieve real time operation. A multitracking system was implemented to track detected objects that move from one camera field of view to a neighbor camera field of view. Experiments on a monitored parking space show that our proposed system yields good performance in tracking people and vehicles, detecting close interaction and generating relevant alarms.