We measure the cross-correlation between weak lensing of galaxy images and of the cosmic microwave background (CMB). The effects of gravitational lensing on different sources will be correlated if the lensing is caused by the same mass fluctuations. We use galaxy shape measurements from 139 deg(2) of the Dark Energy Survey (DES) Science Verification data and overlapping CMB lensing from the South Pole Telescope (SPT) and Planck. The DES source galaxies have a median redshift of z(med) similar to 0.7, while the CMB lensing kernel is broad and peaks at z similar to 2. The resulting cross-correlation is maximally sensitive to mass fluctuations at z similar to 0.44. Assuming the Planck 2015 best-fitting cosmology, the amplitude of the DESxSPT cross-power is found to be A(SPT) = 0.88 +/- 0.30 and that from DESxPlanck to be A(Planck) = 0.86 +/- 0.39, where A = 1 corresponds to the theoretical prediction. These are consistent with the expected signal and correspond to significances of 2.9 sigma and 2.2 sigma, respectively. We demonstrate that our results are robust to a number of important systematic effects including the shear measurement method, estimator choice, photo-z uncertainty and CMB lensing systematics. We calculate a value of A = 1.08 +/- 0.36 for DESxSPT when we correct the observations with a simple intrinsic alignment model. With three measurements of this cross-correlation now existing in the literature, there is not yet reliable evidence for any deviation from the expected LCDM level of cross-correlation. We provide forecasts for the expected signal-to-noise ratio of the combination of the five-year DES survey and SPT-3G.