Ear disorders are among the most common diseases treated in primary care, with a high percentage of non-relevant referrals. The conventional diagnostic procedure is done by a visual examination of the ear canal and tympanic membrane. Consequently, the accuracy of the diagnosis is affected by observer-observer variation, depending on the technical skill and experiences of the physician as well as on the subjective bias of the observer. This situation impacts the proper implementation of treatments, increases health costs, and can lead to serious health complications. To eliminate subjectivity and enhance diagnostic accuracy, we present a diagnostic tool for nine ear conditions in a computer-aided diagnosis scheme. We propose a hybrid learning framework based on convolutional and recurrent neural networks for video otoscopy analysis. The proposed method first extracts the deep features of each relevant frame from the video. Then, a Long Short-term Memory network is introduced to learn spatial sequential data by analyzing deep features for a certain time interval. We carried out the study in collaboration with the Clinical Hospital of the University of Chile and included 875 subjects in a period of 12 months (continuous). The experiments were conducted on a new video otoscopy dataset and showed high performance in terms of accuracy (98.15%), precision (91.94%), sensitivity (91.67%), specificity (98.96%), and F1-score (91.51%). To the best of our knowledge, the proposed system is capable of predicting more diagnoses of ear conditions known to date with high performance. Our system is designed to assist in a real otoscopy examination by analyzing a sequence of images instead of a still image as previous state-of-the-art works. This advantage allows it to provide a comprehensive diagnosis of both eardrum and ear canal diseases.