The temporal variations in a vertical reference frame depend on the variations in the reference surface (e.g., the geoid), and in the surface where the benchmarks are located. The mass redistribution on and beneath the Earth's surface generates changes in the geoid, which is the reference surface for orthometric heights. To model the phenomena and processes that affect Earth's gravity field, the maintenance of a high-precision global physical reference frame that considers its temporal variations is necessary. In this contribution, we estimate and analyze the impact of three geophysical signals, namely mega-earthquakes (Mw > 7.5), hydrological loadings, and glacial isostatic adjustment over the South American vertical reference frames based on the Gravity Recovery and Climate Experiment monthly solutions, glacial isostatic adjustment, and hydrological models. Our results reveal that long-term trends range from -0.5 to 0.3 mm/year, which implies possible height variations up to 10 mm over 20 years. These results point out the possibilities for maintaining modern vertical reference frames if variations in the trends (e.g., due to accelerations) are disregarded. The seasonal variations (from peak to peak), co-seismic deformation, and post-seismic deformation reach 31 mm, 3.8 mm, and 0.8 mm, respectively. According to wavelet analysis based on the independent components (IC) obtained from independent component analysis, it was observed that the predominant signal has one cycle per year. The first four ICs represent 94.4% of the geoid variability. The spatial patterns of the four modes indicated that geoid variations are concentrated in the Amazon, Tocantins, Parana, and Sao Francisco river basins. Temporal variations of the geoid in South America reach magnitudes that should be considered in the evolution of height systems to accomplish the requirements of a modern height system (accuracy on the order of 10 mm or better).