Ensemble methods have gained considerable attention from machine learning and soft computing communities in the last years. There are several practical and theoretical reasons, mainly statistical reasons, why an ensemble may be preferred. A set of learners with similar training performance may have different generalization performance, when exposed to sparse data, large volume of data or data fusion. The basic idea of an ensemble learning algorithm is to build a predictive model by combining a set of learning hypotheses instead of laboriously designing the complete map between inputs and responses in a single step. Ensemble based systems have shown to produce favorable results compared to those of single-expert system for a broad range of applications such as financial, medical and social models, network security, web mining or bioinformatics, to name a few. The focus of this chapter will be on ensembles for classification and regression estimation. Design, implementation and application will be the main topics of the chapter. Specifically, conditions under which ensemble based systems may be more beneficial than their single machine, algorithms for generating individual components of ensemble systems and various procedures through which the individual classifiers can be combined. Finally, future research directions will be included, like incremental learning, machine fusion, and other areas in which ensemble of machines have shown great promise.