At fixed stellar mass, the size of low redshift early-type galaxies is found to be a factor of two larger than their counterparts at z~1, a result with important implications to galaxy formation models. In this paper, we have explored the buildup of the local mass-size relation of elliptical galaxies using two visually classified samples. At low redshift we compiled a subsample of 2,656 elliptical galaxies from the Sloan Digital Sky Survey (SDSS), whereas at higher redshift (up to z~1) we extracted a sample of 228 objects from the HST/ACS images of the Great Observatories Origins Deep Survey (GOODS). All the galaxies in our study have spectroscopic data, allowing us to determine the age and mass of the stellar component. Contrary to previous claims in the literature, using the fossil record information contained in the stellar populations of our local sample, we do not find any evidence for an age segregation at a given stellar mass depending on the size of the galaxies. At a fixed dynamical mass there is only a <9% size difference in the two extreme age quartiles of our sample. Consequently, the local evidence does not support a scenario whereby the present-day mass-size relation has been progressively established via a bottom-up sequence, where older galaxies occupy the lower part this relation, remaining in place since their formation. We do not find any age segregation difference in our high-z sample either. Therefore, we find a trend in size that is insensitive to the age of the stellar populations, at least since z~1. This result supports the idea that the stellar mass-size relation is formed at z~1, with all galaxies populating a region which roughly corresponds to 1/2 of the present size distribution. We have explored two possible scenarios for size growth: puffing up or minor merging. The fact that the evolution in size is independent of stellar age, together with the absence of an increase in the scatter of the relationship with redshift does not support the puffing up mechanism. The observational evidence, however, can not reject at this stage the minor merging hypothesis. We have made an estimation of the number of minor merger events necessary to bring the high-z galaxies into the local relation compatible with the observed size evolution. Since z=0.8, if the merger mass ratio is 1:3 we estimate ~3+-1 minor mergers and if the ratio is 1:10 we obtain ~8+-2 events.