We investigate how star-forming galaxies typically assemble their masses at high redshift. Taking advantage of the deep multi-wavelength coverage of the GOODS data set, we select two of the largest samples of high-redshift star-forming galaxies based on their UV colors and measure stellar mass of individual galaxies. We use template-fitting photometry to obtain optimal estimates of the fluxes in lower-resolution ground-based and Spitzer images using prior information about galaxy positions, shapes, and orientations. By combining the data and realistic simulations to understand measurement errors and biases, we make a statistically robust determination of stellar mass function (SMF) of the UV-selected star-forming galaxies at z similar to 4 and 5. We report a broad correlation between stellar mass and UV luminosity, such that more UV-luminous galaxies are, on average, also more massive. However, we show that the correlation has a substantial intrinsic scatter, particularly for UV-faint galaxies, evidenced by the fact there is a non-negligible number of UV faint but massive galaxies. Furthermore, we find that the low-mass end of the SMF does not rise as steeply as the UV luminosity function (alpha(UVLF) approximate to -(1.7-1.8) while alpha(SMF) approximate to -(1.3-1.4)) of the same galaxies. In a smooth and continuous formation scenario where star formation rates (SFRs) are sustained at the observed rates for a long time, these galaxies would have accumulated more stellar mass (by a factor of approximate to 3) than observed and therefore the SMF would mirror more closely that of the UV luminosity function. The relatively shallow slope of the SMF is due to the fact that many of the UV-selected galaxies are not massive enough, and therefore are too faint in their rest-frame optical bands, to be detected in the current observations. Our results favor a more episodic formation history in which SFRs of low-mass galaxies vary significantly over cosmic time, a scenario currently favored by galaxy clustering. Our findings for the UV-faint galaxies at high redshift are in contrast with previous studies on more UV-luminous galaxies, which exhibit a tighter SFR-M-star correlation. The discrepancy may suggest that galaxies at different luminosities may have different evolutionary paths. Such a scenario presents a nontrivial test to theoretical models of galaxy formation.