Though out-of-sample forecast evaluation is systematically employed with modern machine learning methods and there exists a well-established classic inference theory for predictive ability, see, e.g., West (1996, Asymptotic Inference About Predictive Ability, Econometrica, 64, 1067-1084), such theory is not directly applicable to modern machine learners such as the Lasso in the high dimensional setting. We investigate under which conditions such extensions are possible. Two key properties for standard out-of-sample asymptotic inference to be valid with machine learning are (i) a zero-mean condition for the score of the prediction loss function
and (ii) a fast rate of convergence for the machine learner. Monte Carlo simulations confirm our theoretical findings. We recommend a small out-of-sample vs in-sample size ratio for accurate finite sample inferences with machine learning. We illustrate the wide applicability of our results with a new out-of-sample test for the Martingale Difference Hypothesis (MDH). We obtain the asymptotic null distribution of our test and use it to evaluate the MDH of some major exchange rates at daily and higher frequencies.