Panel data often contain stayers (units with no within-variations) and slow movers (units with little within-variations). In the presence of many slow movers, conventional econometric methods can fail to work. We propose a novel method of robust inference for the average partial effects in correlated random coefficient models robustly across various distributions of within-variations, including the cases with many stayers and/or many slow movers in a unified manner. In addition to this robustness property, our proposed method entails smaller biases and hence improves accuracy in inference compared to existing alternatives. Simulation studies demonstrate our theoretical claims about these properties: the conventional 95% confidence interval covers the true parameter value with 37-93% frequencies, whereas our proposed one achieves 93-96% coverage frequencies.