We set conservative, robust constraints on the annihilation and decay of dark matter into various Standard Model final states under various assumptions about the distribution of the dark matter in the Milky Way halo. We use the inclusive photon spectrum observed by the Fermi Gamma-ray Space Telescope through its main instrument, the Large Area Telescope. We use simulated data to first find the "optimal" regions of interest in the.-ray sky, where the expected dark matter signal is largest compared with the expected astrophysical foregrounds. We then require the predicted dark matter signal to be less than the observed photon counts in the a priori optimal regions. This yields a very conservative constraint as we do not attempt to model or subtract astrophysical foregrounds. The resulting limits are competitive with other existing limits and, for some final states with cuspy dark-matter distributions in the Galactic Center region, disfavor the typical cross section required during freeze-out for a weakly interacting massive particle to obtain the observed relic abundance.