Social Anxiety Disorder (SAD) involves fear of negative evaluation and social avoidance, impacting quality of life. Early life adversities (ELA) are recognized as risk factors for SAD. Previous research indicated inconsistent alterations in resting state functional connectivity (RSFC) in SAD, particularly in the prefrontal cortex and precuneus. This study investigated the interaction between SAD and ELA at the RSFC level. Functional magnetic resonance imaging (fMRI) was conducted on 120 participants (aged 19-48). Four groups were formed: low/ high ELA controls (n = 49, n = 22) and low/ high ELA SAD participants (n = 30, n = 19). Seed-based correlation analyses (SCA) and multi-voxel pattern analysis (MVPA) were applied. A network in which ELA moderates the neural correlates of SAD during the resting state was identified, involving key nodes like the subgenual anterior cingulate cortex, left middle frontal gyrus, and an area in the calcarine fissure/precuneus. Five distinct interaction patterns of SAD and ELA were observed, showcasing opposite RSFC patterns in individuals with SAD based on ELA experience. Results remained significant when controlled for general anxiety and depression measures. Emotional aspects of ELA played a significant role in these interactions. These findings stress the necessity of considering primarily emotional ELA as covariate in neuroimaging studies investigating SAD and potentially also other psychiatric disorders, addressing inconsistencies in prior research. The left middle frontal gyrus emerges as a link in the SAD-ELA interaction during resting state and anxiety-relevant stimulation. Longitudinal studies, starting from childhood, are needed to understand ELA's impact on brain function and to identify potential neuromarkers for SAD predisposition post-ELA exposure.