PURPOSE: This study aims to comprehensively evaluate the quality of radiomics research by examining unique papers from reviews using the radiomics quality score (RQS). METHODS: A literature search was conducted in PubMed (last search date: April 14, 2024). Systematic or non-systematic reviews using the RQS to evaluate radiomic studies were potentially included. Exclusion was applied at two levels: first, at the review level, and second, at the study level (i.e., for the individual articles previously evaluated within the reviews). Score-wise and item-wise analyses were performed, along with trend, multivariable, and subgroup analyses based on baseline study characteristics and validation methods. RESULTS: A total of 1574 unique papers (published online between 1999 and 2023) from 89 reviews were included in the final analysis. The median RQS percentage was 31% with an IQR of 25% (25th-75th percentiles, 14-39%). A positive correlation between median RQS percentage and publication year (2014-2023) was found, with Kendall's tau coefficient of 0.908 (p <
0.001), suggesting an improvement in quality over time. The quality of radiomics publications significantly varied according to different subfields of radiology (p <
0.001). Around one-third of the publications (32%) lacked a separate validation set. Papers with internal validation (54%) dominated those with external validation (14%). Higher-quality validation practices were significantly associated with better RQS percentage scores, independent of the validation's effect on the final score. Item-wise analysis revealed significant shortcomings in several areas. CONCLUSION: Radiomics research quality is low but improving according to RQS. Significant variation exists across radiology subfields. Critical areas were identified for targeted improvement. CLINICAL RELEVANCE STATEMENT: Our study shows that the quality of radiomics research is generally low but improving over time, with item-wise analysis highlighting critical areas needing improvement. It also reveals that the quality of radiomics research differs across subfields and validation methods. KEY POINTS: Overall quality of radiomics research remains low and highly variable, although a significant positive trend suggests an improvement in quality over time. Considerable variations exist in the quality of radiomics publications across different subfields of radiology and validation types. The item-wise analysis highlights several critical areas requiring attention, emphasizing the need for targeted improvements.