Type 2 diabetes mellitus (T2DM) is the third most common chronic metabolic disorder worldwide and seriously dangerous. Novel therapeutics are sought due to the paucity of safe and effective metabolic disorder-related diabetes medicines. Intestinal flora impacts glucose and lipid balance, making it a unique T2DM therapeutic target. Due to gut fermentation, poly/oligo-saccharides are highly beneficial prebiotic carbohydrates for intestinal health. Moreover, supplementation with naturally occurring medicinal and edible homologous traditional Chinese medicines (MEHTCM) poly/oligo-saccharides has significant antidiabetic effects with few side effects. Now, a comprehensive review of research developments of MEHTCM poly/oligo-saccharides was presented to explore their prospects. We outlined the structural characteristics, structure classification, and structure-activity relationships. Notably, structure-activity relationships illustrated that molecular weight, monosaccharide composition, and glycosidic bond type could influence the hypoglycemic activity and prebiotic effect of MEHTCM poly/oligo-saccharides. Additionally, the review systematically summarized the effect and potential mechanism of MEHTCM poly/oligo-saccharide on T2DM, focusing on gut microbiota. The potential applications in formulations for special medical purposes, common food, health care product, agriculture and other fields have also been summarized. This review emphasizes MEHTCM poly/oligo-saccharides' potential as prebiotics for T2DM treatment. This information provides new insights and a theoretical foundation for MEHTCM poly/oligo-saccharide nutritional and medicinal research.