Calcium blooming artifact produced by calcified plaque in coronary computed tomography angiography (CCTA) is a significant contributor to false-positive results for radiologists. Most previous research focused on general noise reduction of CT images, while performance was limited when facing the blooming artifact. To address this problem, we designed an automated and robust semantics-oriented adversarial network that fully exploits the calcified plaques as semantic regions in the CCTA. The semantic features were extracted using a feature extraction module and implemented through a global-local fusion module, a generator with a semantic similarity module, and a matrix discriminator. The effectiveness of our network was validated both on a virtual and a clinical dataset. The clinical dataset consists of 372 CCTA and corresponding coronary angiogram (CAG) results, with the assistance of two cardiac radiologists (with 10 and 21 years of experience) for clinical evaluation. The proposed method effectively reduces artifacts for three major coronary arteries and significantly improves the specificity and positive predictive value for the diagnosis of coronary stenosis.