The interplay between abiotic (resource supply, temperature) and biotic (grazing) factors determines growth and loss processes in phytoplankton through resource competition and trophic interactions, which are mediated by morphological traits like size. Here, we study the relative importance of grazers, water physics, and chemistry on the daily net accumulation rates (ARs) of individual phytoplankton from natural communities, grouped into six size classes from circa 10 to 500 μm. Using a Random Forest modelling approach and 4 years of daily data from a lake, we find that water temperature is generally a pivotal control of all phytoplankton ARs. At the same time, nutrients and light are important for the smallest and the largest classes. Mesozooplankton abundance is a key predictor of the AR for small phytoplankton, with microzooplankton being important for the middle-size range. In our data, large and small phytoplankton have different (seasonal) blooming patterns: small forms are favoured by low temperature and grazing, and high phosphorus levels. Larger forms show positive ARs at high temperatures and low phosphorus (being relatively insensitive to zooplankton grazing). These results help us understand the opportunities and limitations of using size to explain and model phytoplankton responses to biotic and abiotic environmental change.