In the stochastic volatility models for multivariate daily stock returns, it has been found that the estimates of parameters become unstable as the dimension of returns increases. To solve this problem, we focus on the factor structure of multiple returns and consider two additional sources of information: first, the realized stock index associated with the market factor, and second, the realized covariance matrix calculated from high frequency data. The proposed dynamic factor model with the leverage effect and realized measures is applied to ten of the top stocks composing the exchange traded fund linked with the investment return of the SP500 index and the model is shown to have a stable advantage in portfolio performance.