It has been shown that saliency maps can be used to improve the performance of object recognition systems, especially on datasets that have only limited training data. However, a drawback of such an approach is that it requires a pre-trained saliency network. In the current paper, we propose an approach which does not require explicit saliency maps to improve image classification, but they are learned implicitely, during the training of an end-to-end image classification task. We show that our approach obtains similar results as the case when the saliency maps are provided explicitely. We validate our method on several datasets for fine-grained classification tasks (Flowers, Birds and Cars), and show that especially for domains with limited data the proposed method significantly improves the results.