Dairy products made of raw milk are associated with hypervirulent L. monocytogenes clonal complexes (CCs) CC1, CC4, and CC6, and cause half of the reported listeriosis outbreaks in Europe. However, it is currently unknown whether the overrepresentation of L. monocytogenes hypervirulent clones in dairy products made of raw milk is conditioned by an alteration in the native raw milk microbiota growth and/or composition. In this study, the lag phase, maximal growth rate, and the final maximal concentration of mesophilic aerobic bacteria from native raw milk bacteria were measured at refrigerated temperature (4 °C) in the presence and absence of L. monocytogenes contamination. The raw milk microbiota composition and dynamics were evaluated in the presence and absence of L. monocytogenes hypervirulent (CC1, CC4, CC6), and hypovirulent (CC9 and CC121) clones at 4 °C by using 16S rRNA high-throughput sequencing. Our results showed that the growth and composition of the microbial communities naturally present in raw milk are independent of the contamination with hyper- or hypovirulent L. monocytogenes CCs at refrigeration temperature. Pseudomonas was the most abundant genus in raw milk on days 11 and 21, while Carnobacterium was the second most abundant genus regardless of the contaminant L. monocytogenes CCs. Altogether these results suggest that the overrepresentation of hypervirulent L. monocytogenes CC1, CC4, and CC6 in dairy products is not the consequence of a differential alteration in the native composition of the raw milk microbiota.