While numerous studies strive to exploit the complementary potential of MRI and PET using learning-based methods, the effective fusion of the two modalities remains a tricky problem due to their inherently distinctive properties. In addition, current studies often face the problem of small sample sizes and missing PET data due to factors such as patient withdrawal or low image quality. To this end, we propose a hybrid multi-modality multi-task learning (HM