Bipolar disorder (BD) and unipolar depression (UD) are defined as distinct diagnostic categories. However, due to some common clinical and pathophysiological features, it is a clinical challenge to distinguish them, especially in the early stages of BD. This study aimed to explore the common and disease-specific connectivity patterns in BD and UD. This study was constructed over 181 BD, 265 UD and 204 healthy controls. In addition, an independent group of 90 patients initially diagnosed with major depressive disorder at the baseline and then transferred to BD with the episodes of mania/hypomania during follow-up, was identified as initial depressive episode BD (IDE-BD). All participants completed resting-state functional magnetic resonance imaging (R-fMRI) at recruitment. Both network-based analysis and graph theory analysis were applied. Both BD and UD showed decreased functional connectivity (FC) in the whole brain network. The shared aberrant network across groups of patients with depressive episode (BD, IDE-BD and UD) mainly involves the visual network (VN), somatomotor networks (SMN) and default mode network (DMN). Analysis of the topological properties over the three networks showed that decreased clustering coefficient was found in BD, IDE-BD and UD, however, decreased shortest path length and increased global efficiency were only found in BD and IDE-BD but not in UD. The study indicate that VN, SMN, and DMN, which involve stimuli reception and abstraction, emotion processing, and guiding external movements, are common abnormalities in affective disorders. The network separation dysfunction in these networks is shared by BD and UD, however, the network integration dysfunction is specific to BD. The aberrant network integration functions in BD and IDE-BD might be valuable diagnostic biomarkers.