Initial exposure to a rapidly evolving virus establishes B cell memory that biases later responses to antigenically drifted strains. This "immune imprinting" implies that subsequent exposure to a drifted strain can induce affinity maturation of memory B cells toward cross-reactivity with the drifted strain and hence toward greater overall breadth. Here, we used deep mutational scanning of H1 influenza hemagglutinins (HAs) to investigate how viruses evolve in response to these broad antibody response. We identified escape mutations from clonal antibody lineages that targeted the receptor binding site and lateral patch. By adjusting the antigen-antibody contacts, antibody affinity maturation restricted the potential escape routes for the eliciting strain. However, escape occurred readily in drifted strains. We attribute this escape-prone property of the drifted strains to epistatic networks within HA. Our data explain how the influenza virus continues to evolve in the human population by escaping even broad antibody responses.