A key problem in the application of kriging is the definition of a local neighborhood in which to search for the most relevant data. A usual practice consists in selecting data close to the location targeted for prediction and, at the same time, distributed as uniformly as possible around this location, in order to discard data conveying redundant information. This approach may however not be optimal, insofar as it does not account for the data spatial correlation. To improve the kriging neighborhood definition, we first examine the effect of including one or more data and present equations in order to quickly update the kriging weights and kriging variances. These equations are then applied to design a stepwise selection algorithm that progressively incorporates the most relevant data, i.e., the data that make the kriging variance decrease more. The proposed algorithm is illustrated on a soil contamination dataset.