Adaptation affects neuronal responsivity and selectivity throughout the visual hierarchy. However, because most prior studies have tailored stimuli to a single brain area of interest, we have a poor understanding of how exposure to a particular image alters responsivity and tuning at different stages of visual processing. Here we assess how adaptation with naturalistic textures alters neuronal responsivity and selectivity in primary visual cortex (V1) and area V2 of macaque monkeys. Neurons in both areas respond to textures, but V2 neurons are sensitive to higher-order image statistics which do not strongly modulate V1 responsivity. We tested the specificity of adaptation in each area with textures and spectrally-matched 'noise' stimuli. Adaptation reduced responsivity in both V1 and V2, but only in V2 was the reduction dependent on the presence of higher-order texture statistics. Despite this specificity, the texture information provided by single neurons and populations was reduced after adaptation, in both V1 and V2. Our results suggest that adaptation effects for a given feature are induced at the stage of processing that tuning for that feature first arises and that stimulus-specific adaptation effects need not result in improved sensory encoding.