We use stacked spectra of the host galaxies of photometrically identified Type Ia supernovae (SNe Ia) from the Dark Energy Survey (DES) to search for correlations between Hubble diagram residuals and the spectral properties of the host galaxies. Utilizing full spectrum fitting techniques on stacked spectra binned by Hubble residual, we find no evidence for trends between Hubble residuals and properties of the host galaxies that rely on spectral absorption features (<1.3 sigma), such as stellar population age, metallicity, and mass-to-light ratio. However, we find significant trends between the Hubble residuals and the strengths of [O ii] (4.4 sigma) and the Balmer emission lines (3 sigma). These trends are weaker than the well-known trend between Hubble residuals and host galaxy stellar mass (7.2 sigma) that is derived from broad-band photometry. After light-curve corrections, we see fainter SNe Ia residing in galaxies with larger line strengths. We also find a trend (3 sigma) between Hubble residual and the Balmer decrement (a measure of reddening by dust) using H beta and H gamma. The trend, quantified by correlation coefficients, is slightly more significant in the redder SNe Ia, suggesting that the bluer SNe Ia are relatively unaffected by dust in the interstellar medium of the host and that dust contributes to current Hubble diagram scatter impacting the measurement of cosmological parameters.