INTRODUCTION: We investigated the moderating effects of midlife and late-life cognitive activity (CA) on the relationship between tau pathology and both cognition and cognitive decline. METHODS: Eighty-nine non-demented older adults from a Korean cohort underwent comprehensive evaluations, including CA assessments and tau neuroimaging at baseline, and Mini-Mental State Examination (MMSE) at baseline and the 2-year follow-up. RESULTS: Greater midlife CA was associated with higher MMSE scores in a given amount of tau pathology, whereas higher levels of midlife CA were associated with faster tau-related decline in MMSE scores, particularly in individuals with mild cognitive impairment. Late-life CA did not exhibit any interaction with tau on either MMSE scores or their 2-year change. DISCUSSION: Greater midlife CA is generally associated with better cognitive performance despite the presence of tau pathology. However, paradoxically, increased midlife CA appears to be linked to a more rapid tau-related cognitive decline in already cognitively impaired individuals. HIGHLIGHTS: Greater midlife cognitive activity (CA) was generally associated with better cognitive performance in a given amount of tau pathology. Paradoxically, higher levels of midlife CA were related to a more rapid tau-related cognitive decline in already cognitively impaired individuals. Late-life CA did not exhibit any moderation effect on the association between tau and cognitive performance or decline.