During early life stages, social isolation disrupts the proper brain growth and brain circuit formation, which is associated with the risk of mental disorders and cognitive deficits in adulthood. Nevertheless, the impact of juvenile social isolation on brain development, particularly regarding variations across age and sex, remains poorly understood. Here, we investigate the effects of social isolation stress (SIS) during early (3-5 weeks old) or late (5-7 weeks old) juvenile period on brain morphology in adult male and female mice using ultra high-field MRI (11.7 T). We found that both early and late SIS in female mice led to volumetric increases in multiple brain regions, such as the medial prefrontal cortex (mPFC) and hippocampus. Correlation tractography revealed that the fiber tracts in the right corpus callosum and right amygdala were positively correlated with SIS in female mice. In male mice, early SIS resulted in small volumetric increases in the isocortex, whereas late SIS led to reductions in the isocortex and hypothalamus. Furthermore, early SIS caused a negative correlation, while late SIS exhibited a positive correlation, with fiber tracts in the corpus callosum and amygdala in male mice. Using a Random Forest classifier, we achieved effective discrimination between socially isolated and control conditions in the brain volume of female mice, with the limbic areas playing a key role in the model's accuracy. Finally, we discovered that SIS led to context fear generalization in a sex-dependent manner. Our findings highlight the importance of considering both the time- and sex-dependent effects of juvenile SIS on brain development and emotional processing, providing new insights into its long-term consequences.