We reconcile the two worlds of dense and sparse modeling by exploiting the positive aspects of both. We employ a factor model and assume {the dynamic of the factors is non-pervasive while} the idiosyncratic term follows a sparse vector autoregressive model (VAR) {which allows} for cross-sectional and time dependence. The estimation is articulated in two steps: first, the factors and their loadings are estimated via principal component analysis and second, the sparse VAR is estimated by regularized regression on the estimated idiosyncratic components. We prove the consistency of the proposed estimation approach as the time and cross-sectional dimension diverge. In the second step, the estimation error of the first step needs to be accounted for. Here, we do not follow the naive approach of simply plugging in the standard rates derived for the factor estimation. Instead, we derive a more refined expression of the error. This enables us to derive tighter rates. We discuss the implications of our model for forecasting, factor augmented regression, bootstrap of factor models, and time series dependence networks via semi-parametric estimation of the inverse of the spectral density matrix.