Various papers demonstrate the importance of inequality, poverty and the size of the middle class for economic growth. When explaining why these measures of the income distribution are added to the growth regression, it is often mentioned that poor people behave different which may translate to the economy as a whole. However, simply adding explanatory variables does not reflect this behavior. By a varying coefficient model we show that the returns to growth differ a lot depending on poverty and inequality. Furthermore, we investigate how these returns differ for the poorer and for the richer part of the societies. We argue that the differences in the coefficients impede, on the one hand, that the means coefficients are informative, and, on the other hand, challenge the credibility of the economic interpretation. In short, we show that, when estimating mean coefficients without accounting for poverty and inequality, the estimation is likely to suffer from a serious endogeneity bias.