BACKGROUND: The current cervical cancer screening and diagnosis have limitations due to their subjectivity and lack of reproducibility. We describe the development of a deep learning (DL)-based diagnostic risk prediction model and evaluate its potential for clinical impact. METHOD: We developed and internally validated a DL model which accommodates both clinical data and colposcopy images in predicting the patients CIN2+ status using a retrospective cohort of 6356 cases of LEEP-conization/cone-biopsy (gold-standard diagnosis) following an abnormal screening result. The overall performance, discrimination, and calibration of the model were compared to expert clinician's colposcopic impression. The potential for clinical impact was assessed with rate of unnecessary conizations that could be avoided by using our model. RESULTS: The model combining clinical history and colposcopy images demonstrated superior performance prediction of CIN2+(AUC-ROC = 95.3 %, accuracy = 90.8 %, PPV = 94.1 %, NPV = 87.9 %) and better calibration compared to models that used image or clinical history data alone and outperformed clinician's colposcopic impressions. Moreover, if a decision threshold of 10 % is applied to the predicted probability from this model to recommend conization, up to 35 % of conizations could be avoided without missing any true CIN2+ cases. CONCLUSION: We present a novel DL model to predict cervical neoplasia with potential for reducing unnecessary conization. External validation studies are warranted for assessing generalizability.