We exploit the 7 Ms Chandra observations in the Chandra Deep Field-South (CDF-S), the deepest X-ray survey to date, coupled with CANDELS/GOODS-S data, to measure the total X-ray emission arising from 2076 galaxies at 3.5 <= z < 6.5. This aim is achieved by stacking the Chandra data at the positions of optically selected galaxies, reaching effective exposure times of >= 10(9)s. We detect significant (> 3.7 sigma) X-ray emission from massive galaxies at z >= 4. We also report the detection of massive galaxies at z >= 5 at a 99.7 per cent confidence level (2.7 sigma), the highest significance ever obtained for X-ray emission from galaxies at such high redshifts. No significant signal is detected from galaxies at even higher redshifts. The stacking results place constraints on the BHAD associated with the known high-redshift galaxy samples, as well as on the SFRD at high redshift, assuming a range of prescriptions for X-ray emission due to X- ray binaries. We find that the X-ray emission from our sample is likely dominated by processes related to star formation. Our results show that low-rate mass accretion on to SMBHs in individually X-ray-undetected galaxies is negligible, compared with the BHAD measured for samples of X-ray detected AGN, for cosmic SMBH mass assembly at high redshift. We also place, for the first time, constraints on the faint-end of the AGN X-ray luminosity function (logL(X) 42) at z > 4, with evidence for fairly flat slopes. The implications of all of these findings are discussed in the context of the evolution of the AGN population at high redshift.