While some aspects of pragmatic competence are known to decline with age, for metaphor skills the evidence is inconclusive, possibly due to heterogeneity in the assessment tools. Furthermore, the previous literature on age-related changes in pragmatic skills has rarely considered the role of Theory of Mind (ToM), which is described as one of the main factors affecting metaphor across theoretical and experimental studies in children and clinical populations. This study aimed at elucidating age-related differences in metaphor understanding and the interplay between metaphor skills and ToM in middle-aged and older adults with a fine-grained approach. Participants (n = 201, age range 54-93) completed tasks assessing ToM and metaphor understanding. On the one side, we used the Physical and Mental Metaphors task, to distinguish between different types of metaphor (physical, such as "Lifeguards are lizards," meaning that they lie in the sun, vs. mental, such as "Adolescents are pendulums," meaning that they are emotionally unstable) and different aspects of metaphor understanding, namely, accuracy in finding a link between topic and vehicle versus type of interpretation (from physical to psychological). On the other side, we analyzed two aspects of ToM skills: the accuracy in mental state understanding and the intentionality, defined as the degree of mental state attribution, assessed with the Strange Stories and the Animated Social Triangles task, respectively. Structural equation models indicated a decline in metaphor skills with advancing age. Furthermore, we found that ToM is involved in metaphor understanding in a specific fashion. While higher ToM accuracy explained better metaphor accuracy, higher ToM intentionality explained better performance in the interpretation of mental, but not physical, metaphors. These findings suggest that age-related differences in pragmatics extend to metaphor skills and that ToM plays a role in metaphor comprehension in older age, with a division of labor where the ability to understand what others think is key to spotting a metaphorical link, but the greater tendency to attribute mental states is what specifically helps when it comes to grasping the psychological nuances of a metaphor.