Background This study aimed to examine the performance of the four risk of malignancy index (RMI) in discriminating borderline ovarian tumors (BOTs) and benign ovarian masses in daily clinical practice. Methods A total of 162 women with BOTs and 379 women with benign ovarian tumors diagnosed at the Second Affiliated Hospital of Harbin Medical University from January 2012 to December 2016 were enrolled in this retrospective study. Also, we classified these patients into serous borderline ovarian tumor (SBOT) and mucinous borderline ovarian tumor (MBOT) subgroup. Preoperative ultrasound findings, cancer antigen 125 (CA125) and menopausal status were reviewed. The area under the curve (AUC) of receiver operator characteristic curves (ROC) and performance indices of RMI I, RMI II, RMI III and RMI IV were calculated and compared for discrimination between benign ovarian tumors and BOTs. Results RMI I had the highest AUC (0.825, 95% CI: 0.790-0.856) among the four RMIs in BOTs group. Similar results were found in SBOT (0.839, 95% CI: 0.804-0.871) and MBOT (0.791, 95% CI: 0.749-0.829) subgroups. RMI I had the highest specificity among the BOTs group (87.6, 95% CI: 83.9-90.7%), SBOT (87.6, 95% CI: 83.9-90.7%) and MBOT group (87.6, 95% CI: 83.9-90.7%). RMI II scored the highest overall in terms of sensitivity among the BOTs group (69.75, 95% CI: 62.1-76.7%), SBOT (74.34, 95% CI: 65.3-82.1%) and MBOT (59.18, 95% CI: 44.2-73.0%) group. Conclusion Compared to other RMIs, RMI I was the best-performed method for differentiation of BOTs from benign ovarian tumors. At the same time, RMI I also performed best in the discrimination SBOT from benign ovarian tumors.