We report our multi-band infrared (IR) imaging of the transitional millisecond pulsar system J1023+0038, a rare pulsar binary known to have an accretion disk in 2000-2001. The observations were carried out with ground-based and space telescopes from near-IR to far-IRwavelengths. We detected the source in near-IR JH bands and Spitzer 3.6 and 4.5 mu m mid-IR channels. Combined with the previously reported optical spectrum of the source, the IR emission is found to arise from the companion star, with no excess emission detected in the wavelength range. Because our near-IR fluxes are nearly equal to those obtained by the 2MASS all-sky survey in 2000 February, the result indicates that the binary did not contain the accretion disk at the time, whose existence would have raised the near-IR fluxes to twice larger values. Our observations have thus established the short-term nature of the interacting phase seen in 2000-2001: the accretion disk existed for at most 2.5 yr. The binary was not detected by the WISE all-sky survey carried out in 2010 at its 12 and 22 mu m bands and our Herschel far-IR imaging at 70 and 160 mu m. Depending on the assumed properties of the dust, the resulting flux upper limits provide a constraint of <3 x 10(22)-3 x 10(25) g on the mass of the dust grains that possibly exist as the remnants of the previously seen accretion disk.