Quality control of fruits to satisfy increasingly competitive food markets requires the implementation of automatic visual servo systems in fruit processing operations to cope with market challenges. A new and fast method for identifying and classifying moving fruits by processing single color images from a static camera in real time was developed and tested. Two algorithms were combined to classify and track moving fruits on image plane using representative color features. The method allows classifying the fruit by color segmentation and estimating its position on the image plane, which provides a reliable algorithm to be implemented in robotic manipulation of fruits. To evaluate the methodology an experimental real time system simulating a conveyor belt and real fruit was used. Testing of the system indicates that with natural lighting conditions and proper calibration of the system a minimum error of 2% in classification of fruits is feasible. The methodology allows for very simple implementation, and although operational results are promising, even higher accuracy may be possible if structured illumination is used.