We search for evidence of diffuse Ly alpha emission from extended neutral hydrogen surrounding Ly alpha emitting galaxies (LAEs) using deep narrow-band images of the Extended Chandra Deep Field South. By stacking the profiles of 187 LAEs at z = 2.06, 241 LAEs at z = 3.10, and 179 LAEs at z = 3.12, and carefully performing low-surface brightness photometry, we obtain mean surface brightness maps that reach 9.9, 8.7, and 6.2 x 10(-19) erg cm(-2) s(-1) arcsec(-2) in the emission line. We undertake a thorough investigation of systematic uncertainties in our surface brightness measurements and find that our limits are 5-10 times larger than would be expected from Poisson background fluctuations; these uncertainties are often underestimated in the literature. At z similar to 3.1, we find evidence for extended halos with small-scale lengths of 5-8 kpc in some but not all of our sub-samples. We demonstrate that sub-samples of LAEs with low equivalent widths and brighter continuum magnitudes are more likely to possess such halos. At z similar to 2.1, we find no evidence of extended Ly alpha emission down to our detection limits. Through Monte-Carlo simulations, we also show that we would have detected large diffuse LAE halos if they were present in our data sets. We compare these findings to other measurements in the literature and discuss possible instrumental and astrophysical reasons for the discrepancies.