The major computation cost of pseudo-spectral method comes from the evaluation of differentiation matrix multiplication. In the past, uniform or Chebyshev collocation points are used for sampling. The differentiation matrix multiplication was evaluated by fast Fourier transform (FFT) or fast cosine transform (FCT), in order to reduce the computation complexity from O (N-2) to O (N log (N)). However, the intrinsic properties of FFT or FCT may cause the wraparound effect and Gibbs phenomenon. Moreover, FFT or FCT is not applicable to other collocation points such as Legendre and Hermite. In order to improve the accuracy and applicability of the pseudo-spectral method, the fast multipole method (FMM) is exploited to substitute the FFT or FCT. By making use of the similarity of the N-body problem and the collocation problem, a new FMM-based pseudo-spectral time domain method is developed in this paper.