Glycosaminoglycans (GAGs) are essential bone extracellular matrix molecules that regulate osteoblast differentiation. Numerous studies have explored endogenous and exogenous GAG osteoanabolic activities using appropriate in vitro and in vivo models. However, GAGs' underlying the mechanism of action and structure-function relationships need to be elucidated in detail. Earlier, we showed that exogenous GAG can bring about osteogenesis in pre-osteoblast cells. In the present study, we have elucidated the mechanism of action of exogenous GAGs, especially of the chondroitin sulfate/dermatan sulfate (CS/DS) class on osteogenesis. GAGs were immobilized, and osteoblast differentiation was evaluated in MC3T3-E1 cells. Results indicated that GAGs supported osteoblast differentiation by promoting collagen production, extracellular matrix formation, and subsequent mineralization. We elucidated the mechanisms underlying these effects by assessing the key signaling molecules involved in osteogenesis in response to exogenous CS/DS with/without BMP2. CS/DS alone significantly increased pERK1/2 and ATF4 expression levels differentially in a time-dependent manner without significant effects on BMP2, RUNX2, and pSMAD5 protein expression. On the other hand, CS/DS, in the presence of BMP2, differentially increased BMP2, pSMAD5, pERK1/2, RUNX2, and ATF4 expression levels at various time points. Collectively, these results strongly suggest that CS/DS can promote osteogenesis, and in the presence of BMP2, it could promote SMAD-mediated ERK-dependent osteogenesis.