Damage to the primary visual cortex (V1) results in visual field deficits on the contralateral side of the world corresponding to the damaged region. Patients with such loss nonetheless show varying residual vision within this apparently blind region, with the neural mechanisms underlying this ability obscured by small study populations. We identified lesions on structural scans from 39 patients (12 female) with hemianopia and occipital lobe damage. We estimated the proportion of damage throughout the visual cortex and correlated this with performance in up to three different tests of residual vision in the blind field. We then performed voxel-based lesion-symptom mapping (VLSM) to determine the precise neural regions associated with residual vision. V1 damage did not show a relationship with residual vision measured with any task, although the extent of damage to V4 and hMT+/V5 both correlated with the ability to detect contrast-modulated stimuli. Indeed, damage to hMT+/V5 severely impacted performance across all three tasks, particularly motion detection. Although hMT+/V5 appeared necessary for residual vision, some patients with intact hMT+/V5 had no residual vision, indicating this area alone was not sufficient. VLSM revealed that damage to the optic radiation laterally was most highly associated with poor residual vision. Thus, VLSM indicates that hMT+/V5 and the optic radiation specifically are critical for residual vision in hemianopia.