Sex-based differences have been observed in the incidence and prognosis of various cancers, as well as in the response to immune check point inhibitors (ICIs). These disparities are partially attributed to sex-based differences in the molecular characteristics of the anticancer immune response, which are largely influenced by sex hormones. Here, we provide a comprehensive overview on how sex hormones affect innate and adaptive immunity and contribute to shaping the features of tumor immune microenvironment and response to anticancer immunotherapy. We also discuss the promising potential and challenges of combining sex hormone manipulation with anticancer immunotherapy as new therapeutic strategy. We surmise that a sex-based perspective should be part of precision medicine approaches, and sex hormones manipulation provides opportunities for innovative immune therapeutic approaches.