This paper studies estimation of linear panel regression models with heterogeneous coefficients, when both the regressors and the residual contain a possibly common, latent, factor structure. Our theory is (nearly) efficient, because based on the GLS principle, and also robust to the specification of such factor structure because it does not require any information on the number of factors nor estimation of the factor structure itself. We first show how the unfeasible GLS estimator not only affords an efficiency improvement but, more importantly, provides a bias-adjusted estimator with the conventional limiting distribution, for situations where the OLS is affected by a first-order bias. The technical challenge resolved in the paper is to show how these properties are preserved for a class of feasible GLS estimators in a double-asymptotics setting. Our theory is illustrated by means of Monte Carlo exercises and, then, with an empirical application using individual asset returns and firms' characteristics data.