Antibodies are key proteins in the immune system that can reversibly and non-covalently bind specifically to their corresponding antigens, forming antigen-antibody complexes. They play a crucial role in recognizing foreign or self-antigens during the adaptive immune response. Monoclonal antibodies have emerged as a promising class of biological macromolecule therapeutics with broad market prospects. In the process of antibody drug development, a key engineering challenge is to improve the affinity of candidate antibodies, without experimentally resolved structures of the antigen-antibody complexes as input for computer-aided predictive methods. In this work, we present an approach for predicting the effect of residue mutations on antibody affinity without the structures of the antigen-antibody complexes. The method involves the graph representation of proteins and utilizes a pre-trained encoder. The encoder captures the residue-level microenvironment of the target residue on the antibody along with the antigen context pre- and post-mutation. The encoder inherently possesses the potential to identify paratope residues. In addition, we curated a benchmark dataset specifically for mutations of the antibody. Compared to baseline methods based on complex structures and sequences, our approach achieves superior or comparable average accuracy on benchmark datasets. Additionally, we validate its advantage of not requiring antigen-antibody complex structures as input for predicting the effects of mutations in antibodies against SARS-CoV-2, influenza, and human cytomegalovirus. Our method shows its potential for identifying mutations that improve antibody affinity in practical antibody engineering applications.