BACKGROUND: Pathological images of hepatocellular carcinoma (HCC) contain abundant tumor information that can be used to stratify patients. However, the links between histology images and the treatment response have not been fully unveiled. METHODS: We trained and evaluated a model by predicting the prognosis of 287 non-treated HCC patients postoperatively, and further explored the model's treatment response predictive ability in 79 sorafenib-treated patients. Based on prognostic relevant pathological signatures (PPS) extracted from CNN-SASM, which was trained by denoised recurrence label (DRL) under different thresholds, the PPS-based prognostic model was formulated. A total of 78 HCC patients from TCGA-LIHC were used for the external validation. RESULTS: We proposed the CNN-SASM based on tumor pathology and extracted PPS. Survival analysis revealed that the PPS-based prognostic model yielded the AUROC of 0.818 and 0.811 for predicting recurrence at 1 and 2 years after surgery, with an external validation reaching 0.713 and 0.707. Furthermore, the predictive ability of the PPS-based prognostic model was superior to clinical risk indicators, and it could stratify patients with significantly different prognoses. Importantly, our model can also stratify sorafenib-treated patients into two groups associated with significantly different survival situations, which could effectively predict survival benefits from sorafenib. CONCLUSIONS: Our prognostic model based on pathology deep learning provided a valuable means for predicting HCC patient recurrence condition, and it could also improve patient stratification to sorafenib treatment, which help clinical decision-making in HCC.