Earthquakes introduce transient mechanical damage in the subsurface, which causes postseismic hazards and can take years to recover. This observation has been linked to relaxation, a phenomenon observed in a wide range of materials after straining perturbations, but systematic controls on the recovery duration in the shallow subsurface after earthquake ground shaking have not been determined. Here, we analyse the effects of two successive large earthquakes and their aftershocks on ground properties using estimates of seismic velocity from ambient noise interferometry. We show that the relaxation time scale is a constant that is an intrinsic property of the substrate, independent of the intensity of ground shaking. Our study highlights the predictability of earthquake damage dynamics in the shallow subsurface and also in other materials. This finding may be reconciled with existing state variable frameworks by considering the superposition of different populations of damaged contacts.