We discuss the additional perturbation introduced during inflation by quantum stress tensor fluctuations of a conformally invariant field such as the photon. We consider both a kinematical model, which deals only with the expansion fluctuations of geodesics, and a dynamical model which treats the coupling of the stress tensor fluctuations to a scalar inflaton. In neither model do we find any growth at late times, in accordance with a theorem due to Weinberg. What we find instead is a correction which becomes larger the earlier one starts inflation. This correction is non-Gaussian and highly scale dependent, so the absence of such effects from the observed power spectra may imply a constraint on the total duration of inflation. We discuss different views about the validity of perturbation theory at very early times during which currently observable modes are trans-Planckian.