Our memory for the words we already know is best predicted by their associated meanings. However, the factors that influence whether we will remember a new word after we see it for the first time are unclear. We record memory performance for 2100 novel pseudowords across 1804 participants during a continuous recognition task. Participants show significant agreement across individuals for which novel words were memorable or forgettable, suggesting an intrinsic memorability for individual pseudowords. Pseudowords that are similar to low-frequency known words, with sparse orthographic neighbourhoods and rarely occurring letter pairs, are more memorable. Further, using intracranial recordings in 36 epilepsy patients we show a region in the anterior fusiform cortex that shows sensitivity to the memorability of these pseudowords. These results suggest that known words in our lexicon act as a scaffold for remembering novel word forms, with rare and unique known words providing the best support for novel word learning.