This paper introduces a new factor structure suitable for modeling large realized covariance matrices with full likelihood‐based estimation. Parametric and nonparametric versions are introduced. Because of the computational advantages of our approach, we can model the factor nonparametrically as a Dirichlet process mixture or as an infinite hidden Markov mixture, which leads to an infinite mixture of inverse‐Wishart distributions. Applications to 10 assets and 60 assets show that the models perform well. By exploiting parallel computing the models can be estimated in a matter of a few minutes.