The form of political polarization where citizens develop strongly negative attitudes towards out-party policies and members has become increasingly prominent across many democracies. Economic hardship and social inequality, as well as inter-group and racial conflict, have been identified as important contributing factors to this phenomenon known as "affective polarization." Such partisan animosities are exacerbated when these interests and identities become aligned with existing party cleavages. In this paper we use a model of cultural evolution to study how these forces combine to generate and maintain affective political polarization. We show that economic events can drive both affective polarization and sorting of group identities along party lines, which in turn can magnify the effects of underlying inequality between those groups. But on a more optimistic note, we show that sufficiently high levels of wealth redistribution through the provision of public goods can counteract this feedback and limit the rise of polarization. We test some of our key theoretical predictions using survey data on inter-group polarization, sorting of racial groups and affective polarization in the United States over the past 50 years.