Whispering is a common adverse hearing condition. However, it is still unclear whether older adults have more difficulty perceiving emotions in whispered speech, and whether hearing loss contributes to these difficulties. To fill this research gap, we compared emotional prosody perception under phonated and whispered conditions in three groups of participants (younger adults, and older adults with and without hearing loss). The results revealed that both older adult groups were less accurate when processing emotions in whispered speech. Moreover, older adults with hearing loss performed worse than normal-hearing peers and younger adults in both phonated and whispered conditions. This study presented the first empirical data on the ability of older adults with hearing loss to recognize emotions in whispered speech. The findings highlighted the negative impact of whispering on emotional prosody recognition among older adults, with hearing loss exacerbating these difficulties.