Severe damage to cement asphalt mortar (CA mortar) can compromise the stability and safety of high-speed railway operations due to various complex factors during service. The loads from high-speed trains and temperature gradients within the ballastless track structure are significant contributors to this damage. However, most previous studies have focused on laboratory tests or numerical simulations under simple loading conditions, while few have investigated the damage evolution of CA mortar when both train loads and temperature gradients are considered simultaneously. In this paper, a finite element model of the CRTS II ballast track and a high-speed railway train dynamics model based on the damage constitutive model of CA mortar was established. The damage evolution of CA mortar through long-term cyclic numerical simulations under the combined effects of train load and temperature gradient load were investigated. By integrating the maintenance criteria for high-speed railways, the lifetime of CA mortar using the criteria of crack length and off-seam width was predicted. In addition, the material and structural properties of CA mortar were also optimized, considering the relationship between its elastic modulus and density, to enhance its lifetime. The conclusions reached are more realistic. The results indicate that the combined load causes deformation in the ballast track structure, leading to gradual damage progression from the edge to the interior of the CA mortar layer. The lifetime of CA mortar is determined by the number of days it takes for the crack length to reach the maintenance criteria. The lifetime of CA mortar under different temperature gradients ranges from 1 to 2 years. Increasing the elastic modulus and thickness of the CA mortar layer improves its lifespan. An elastic modulus of 9000 MPa and a thickness of 50 mm for the CA mortar were recommended.