CD8+ T cells are critical for immune protection against severe COVID-19 during acute infection with SARS-CoV-2. However, the induction of antiviral CD8+ T cell responses varies substantially among infected people, and a better understanding of the mechanisms that underlie such immune heterogeneity is required for pandemic preparedness and risk stratification. In this study, we analyzed SARS-CoV-2-specific CD4+ and CD8+ T cell responses in relation to age, clinical status, and inflammation among patients infected primarily during the initial wave of the pandemic in France or Japan. We found that age-related contraction of the naive lymphocyte pool and systemic inflammation were associated with suboptimal SARS-CoV-2-specific CD4+ and, even more evidently, CD8+ T cell immunity in patients with acute COVID-19. No such differences were observed for humoral immune responses targeting the spike protein of SARS-CoV-2. We also found that the proinflammatory cytokine IL-18, concentrations of which were significantly elevated among patients with severe disease, suppressed the de novo induction and memory recall of antigen-specific CD8+ T cells, including those directed against SARS-CoV-2. These results potentially explain the vulnerability of older adults to infections that elicit a profound inflammatory response, exemplified by acute COVID-19.