Colorectal cancer (CRC) is a prevalent gastrointestinal tumor worldwide with high morbidity and mortality. Predicting the survival of CRC patients not only enhances understanding of their life expectancies but also aids clinicians in making informed decisions regarding suitable adjuvant treatments. Although there are many clinical, genomic, and transcriptomic studies on this hot topic, only a few studies have explored the direction of integrating advanced deep learning algorithms and histopathological images. In addition, it is still unclear if combining histopathological images and molecular data can better predict patients' survival. To fill in this gap, we proposed in this study a novel multimodal deep learning computational framework using Multimodal Compact Bilinear Pooling (MCBP) to predict the 5-year survival of CRC patients from histopathological images, clinical information, and molecular data. We applied our framework to the cancer genome atlas (TCGA) CRC data, consisting of 84 samples with histopathological images, clinical information, mRNA sequencing data, and gene mutation data all available. Under the 5-fold cross-validation, the model using only histopathological images achieved an area under the curve (AUC) of 0.743. Whereas, the model combining image and clinical information and the model combining image and gene mutation information achieved AUCs of 0.771 and 0.773 respectively, better than that of the image solely. Our study demonstrates that histopathological images can reasonably predict the 5-year survival of CRC patients, and that the appropriate integration of these images with clinical or molecular data can further enhance predictive performance.