In this paper, a two-stage task weakly supervised learning algorithm is proposed. It accurately achieved patient-level classification task of benign and malignant thyroid nodules based on ultrasound images from two scanning angles: long axis and short axis of the thyroid site. In the first stage, 68,208 ultrasound scanning images of 588 patients are used to train the underlying classification model. In the second stage, feature vectors of ultrasound images with dual scan angles are extracted using the classification model in the first stage. Then the feature vectors are assigned to position sequences in the order of visual reception by the physician. Finally, the location decision is made through a weakly supervised learning approach. Combined with the dual-angle difference information carried in the overall features, our method accurately achieved benign and malignant classification of thyroid nodules at the patient level. An accuracy of 93.81 % for benign and malignant classification of patients was obtained in our test set. The accuracy of benign and malignant classification of patients with thyroid nodules is improved by our weakly supervised learning method based on a two-stage classification task. It also reduced the pressure of imaging physicians in diagnosing a large number of images. In the clinical auxiliary diagnosis, it provides an effective reference for the timely determination of thyroid nodule patients.