Decades of macaque research established the importance of prefrontal cortex for working memory. Surprisingly, recent human neuroimaging studies demonstrated that the contents of working memory can be decoded from primary visual cortex (V1). However the necessity of this mnemonic information remains unknown and contentious. Here we provide causal evidence that transcranial magnetic stimulation targeting human V1 disrupted the fidelity of visual working memory. Errors increased only for targets remembered in the portion of the visual field disrupted by stimulation. Moreover, concurrently measured electroencephalography confirmed that stimulation disrupted not only memory behavior, but neurophysiological signatures of working memory. These results change the question from whether visual cortex is necessary for working memory to what mechanisms it uses to support memory. Moreover, they point to models in which the mechanisms supporting working memory are distributed across brain regions, including sensory areas that here we show are critical for memory storage.