Context. Recent observations suggest a significant and rapid buildup of dust in galaxies at high redshift (z > 4); this presents new challenges to our understanding of galaxy formation in the early Universe. Although our understanding of the physics of dust production and destruction in a galaxy's interstellar medium (ISM) is improving, investigating the baryonic processes in the early universe remains a complex task owing to the inherent degeneracies in cosmological simulations and chemical evolution models. Aims. In this work we characterized the evolution of 98 z similar to 5 star-forming galaxies observed as part of the ALMA Large Program ALPINE by constraining the physical processes underpinning the gas and dust production, consumption, and destruction in their ISM. Methods. We made use of chemical evolution models to simultaneously reproduce the observed dust and gas content of our galaxies, obtained respectively from spectral energy distribution (SED) fitting and ionized carbon measurements. For each galaxy we constrained the initial gas mass, gas inflows and outflows, and efficiencies of dust growth and destruction. We tested these models with both the canonical Chabrier and a top-heavy initial mass function (IMF); the latter allowed rapid dust production on shorter timescales. Results. We successfully reproduced the gas and dust content in most of the older galaxies (greater than or similar to 600 Myr) regardless of the assumed IMF, predicting dust production primarily through Type II supernovae (SNe) and no dust growth in the ISM, as well as moderate inflow of primordial gas. In the case of intermediate-age galaxies (300-600 Myr), we reproduced the gas and dust content through Type II SNe and dust growth in ISM, though we observed an overprediction of dust mass in older galaxies, potentially indicating an unaccounted dust destruction mechanism and/or an overestimation of the observed dust masses. The number of young galaxies (less than or similar to 300 Myr) reproduced, increases for models assuming top-heavy IMF but with maximal prescriptions of dust production. Galactic outflows are required (up to a mass-loading factor of 2) to reproduce the observed gas and dust mass, and to recover the decreasing trend of gas and dust over stellar mass with age. Assuming the Chabrier IMF, models are able to reproduce similar to 65% of the total sample, while with top-heavy IMF the fraction increases to similar to 93%, alleviating the tension between the observations and the models. Observations from the James Webb Space Telescope (JWST) will allow us to remove degeneracies in the diverse intrinsic properties of these galaxies (e.g., star formation histories and metallicity), thereby refining our models.