Although robust AI governance requires the engagement of diverse stakeholders across the artificial intelligence (AI) ecosystem, the US safety net has largely been excluded from this kind of collaboration. Without a reorientation of the AI governance agenda, marginalized patients will disproportionately bear the risks of AI in the US healthcare system. To prevent this replication of digital inequity and an organizational digital divide, we suggest specific next steps for diverse stakeholders to progress toward more equitable policy and practice.