Construction and analysis of functional brain networks (FBNs) with resting-state functional magnetic resonance imaging (rs-fMRI) is a promising method to diagnose functional brain diseases. Nevertheless, the existing methods suffer from several limitations. First, the functional connectivities (FCs) of the FBN are usually measured by the temporal co-activation level between rs-fMRI time series from regions of interest (ROIs). While enjoying simplicity, the existing approach implicitly assumes simultaneous co-activation of all the ROIs, and models only their synchronous dependencies. However, the FCs are not necessarily always synchronous due to the time lag of information flow and cross-time interactions between ROIs. Therefore, it is desirable to model asynchronous FCs. Second, the traditional methods usually construct FBNs at individual level, leading to large variability and degraded diagnosis accuracy when modeling asynchronous FBN. Third, the FBN construction and analysis are conducted in two independent steps without joint alignment for the target diagnosis task. To address the first limitation, this paper proposes an effective sliding-window-based method to model spatiotemporal FCs in Transformer. Regarding the second limitation, we propose to learn common and individual FBNs adaptively with the common FBN as prior knowledge, thus alleviating the variability and enabling the network to focus on the individual disease-specific asynchronous FCs. To address the third limitation, the common and individual asynchronous FBNs are built and analyzed by an integrated network, enabling end-to-end training and improving the flexibility and discriminability. The effectiveness of the proposed method is consistently demonstrated on three data sets for mild cognitive impairment (MCI) diagnosis.