The breast cancer is one of the most prevalent causes of cancer-related death globally. Preliminary diagnosis of breast cancer increases the patient's chances of survival. Breast cancer classification is a challenging problem due to dense tissue structures, subtle variations, cellular heterogeneity, artifacts, and variability. In this paper, we propose three hybrid deep-transfer learning models for breast cancer classification using histopathology images. These models use Xception model as a base model, and we add seven more layers to fine-tune the base model. We also performed an extensive comparative analysis of five prominent machine-learning classifiers, namely Random Forest Classifier (RFC), Logistic Regression (LR), Support Vector Classifier (SVC), K-Nearest Neighbors (KNN), and Ada-boost. We incorporate the best performing two classifiers, namely RFC and SVC, in the fine-tuned Xception model, and accordingly, they are named as Xception Random Forest (XRF) and Xception Support Vector (XSV), respectively. The fine-tuned Xception model with softmax classifier is termed as Multi-layer Xception Classifier (MXC). These three models are evaluated on the two publically available datasets: BreakHis and Breast Histopathology Images Database (BHID). Our all three models perform better than the state-of-the-art methods. The XRF provides the best performance at the 40 × magnification level on the BreakHis dataset, with an accuracy (ACC) of 94.44%, F1 score (F1) of 94.44%, area under the receiver operating characteristic curve (AUC) of 95.12%, Matthew's correlation coefficient (MCC) of 88.98%, kappa (K) of 88.88%, and classification success index (CSI) of 89.23%. The MXC provides the best performance on the BHID dataset, with an ACC of 88.50%, F1 of 88.50%, AUC of 95.12%, MCC of 77.03%, K of 77.00%, and CSI of 79.13%. Further, to validate our models, we performed fivefold cross-validation on both datasets and obtained similar results.