Our attention can sometimes be disrupted by salient but irrelevant objects in the environment. This distractor interference can be reduced when distractors appear frequently, allowing us to anticipate their presence. However, it remains unknown whether distractor frequency can be learned implicitly across distinct contexts. In other words, can we implicitly learn that in certain situations a distractor is more likely to appear, and use that knowledge to minimize the impact that the distractor has on our behavior? In two experiments, we explored this question by asking participants to find a unique shape target in displays that could contain a color singleton distractor. Forest or city backgrounds were presented on each trial, and unbeknownst to the participants, each image category was associated with a different distractor probability. We found that distractor interference was reduced when the image predicted a high rather than low probability of distractor presence on the upcoming trial, even though the location and (in Experiment 2) the color of the distractor was completely unpredictable. These effects appear to be driven by implicit rather explicit learning. We conclude that implicit learning of context-specific distractor probabilities can drive flexible strategies for the reduction of distractor interference.