OBJECTIVE: To compare the performance of four commonly used algorithms to differentiate benign from malignant adnexal masses when used by a novice operator. METHODS: Women with adnexal masses treated at Mayo Clinic, Rochester, Minnesota, in 2019 were identified retrospectively. Patients were included if they underwent surgery within 3 months of diagnosis or had at least 10 months of follow-up. A nonexpert operator (European Federation of Societies for Ultrasound in Medicine and Biology level I) classified each lesion using ADNEX (Assessment of Different Neoplasias in the Adnexa), two-step strategy (benign descriptors followed by ADNEX), O-RADS (Ovarian-Adnexal Reporting and Data System) 2019, and O-RADS 2022. The primary outcome measure was the area under the receiver operating characteristic curve (AUC) compared across the four algorithms. RESULTS: A total of 556 women were included in the analyses: 452 with benign and 104 with malignant masses. The AUCs of ADNEX, the two-step strategy, O-RADS 2019, and O-RADS 2022 were 0.90 (95% CI, 0.87-0.94), 0.91 (95% CI,0.88-0.94), 0.88 (95% CI,0.84-0.91), and 0.88 95% CI, (0.84-0.91), respectively. The two-step strategy performed significantly better than the O-RADS algorithms ( P =.005 and P =.002). With all the algorithms, the observed malignancy rate was 1.9-2.2% among lesions categorized as almost certainly benign, twofold higher than the expected less than 1.0%. Lesions wrongly classified as almost certainly benign were borderline tumors (n=4) and metastases (n=3). CONCLUSION: In the hands of a novice operator, all algorithms performed well and were able to distinguish benign from malignant lesions. Although the two-step strategy performed slightly better than the O-RADSs, the difference did not appear to be clinically meaningful. The malignancy rate among lesions classified as almost certainly benign was unexpectedly high at 1.9-2.3%, approximately double the expected rate of less than 1.0%.