Based on high-resolution mass spectrometry (HRMS), nontarget analysis (NTA) can rapidly identify and characterize numerous hazardous substances in complex environmental samples. However, the intricate identification process often results in the underutilization of many mass spectrometry features. Even when chemical structures are identified, their toxicological effects and health outcomes may remain unknown. To address these challenges, this study introduces MSFragTox, a novel approach that leverages the rich fragmentation spectra inherent in high resolution tandem mass spectrometry (MS/MS) to directly predict toxicity. This method integrates MS/MS data with high-throughput screening (HTS) assays, focusing on seven endocrine disruption-related endpoints from Tox21, and uses MS-derived fingerprints: substructure fragmentation probability vectors to construct toxicity predictions using machine learning algorithms. The best model demonstrated robust performance with an average area under the receiver operating characteristic curve (AUROC) of 0.845 on the test set, outperforming models based on traditional molecular fingerprints and descriptors. Additionally, a web client (http://ms.envwind.site:8500) is provided for users to screen toxicity based on chemical MS/MS data. Furthermore, in-depth analyses of commonalities and differences in substructures reveal the mechanisms underlying across toxicity endpoints. Using MSFragTox, we validated the potential endocrine-disrupting effects of substances corresponding to MS/MS from real samples, highlighting the feasibility of directly studying toxicity through MS/MS and its potential applications in risk prediction and early warning for environmental samples.