Object tracking in video sequences has been extensively studied in computer vision. Although promising results have been achieved, often the proposed solutions are tailored for particular objects, structured to specific conditions or constrained by tight guidelines. In real cases it is difficult to recognize these situations automatically because a large number of parameters must be tuned. Factors such as these make it necessary to develop a method robust to various environments, situations and occlusions. This paper proposes a new simple appearance model, with only one parameter, which is robust to prolonged partial occlusions and drastic appearance changes. The proposed strategy is based on covariance descriptors (which represent the tracked object) and an on-line nearest neighbor classifier (to track the object in the sequence). The proposed method performs exceptionally well and reduces the average error (in pixels) by 47% compared with tracking methods based on on-line boosting. © 2010 IEEE.