Prokaryotes have evolved a wide repertoire of defense systems to prevent invasion by mobile genetic elements (MGEs). However, because MGEs are vehicles for the exchange of beneficial accessory genes, defense systems could consequently impede rapid adaptation in microbial populations. Here, we study how defense systems impact horizontal gene transfer (HGT) in the short term and long term. By combining comparative genomics and phylogeny-aware statistical methods, we quantify the association between the presence of seven widespread defense systems and the abundance of MGEs in the genomes of 196 bacterial and one archaeal species. We also calculate the differences in the rates of gene gain and loss between lineages that possess and lack each defense system. Our results show that the impact of defense systems on HGT is highly taxon and system dependent and, in most cases, not statistically significant. Timescale analysis reveals that defense systems must persist in a lineage for a relatively long time to exert an appreciable negative impact on HGT. In contrast, for shorter evolutionary timescales, frequent coacquisition of MGEs and defense systems results in a net positive association of the latter with HGT. Given the high turnover rates experienced by defense systems, we propose that the inhibitory effect of most defense systems on HGT is masked by their strong linkage with MGEs. These findings help explain the contradictory conclusions of previous research by pointing at mobility and within-host retention times as key factors that determine the impact of defense systems on genome plasticity.