Recent evidence based independently on spectral line strengths and dynamical modelling point towards a non-universal stellar Initial Mass Function (IMF), probably implying an excess of low-mass stars in elliptical galaxies with a high velocity dispersion. Here we show that a time-independent bottom-heavy IMF is compatible neither with the observed metal-rich populations found in giant ellipticals nor with the number of stellar remnants observed within these systems. We suggest a two-stage formation scenario involving a time-dependent IMF to reconcile these observational constraints. In this model, an early strong star-bursting stage with a top-heavy IMF is followed by a more prolonged stage with a bottom-heavy IMF. Such model is physically motivated by the fact that a sustained high star formation will bring the interstellar medium to a state of pressure, temperature and turbulence that can drastically alter the fragmentation of the gaseous component into small clumps, promoting the formation of low-mass stars. This toy model is in good agreement with the different observational constrains on massive elliptical galaxies, such as age, metallicity, alpha-enhancement, M/L, or the mass fraction of the stellar component in low-mass stars.