Pepper (Capsicum spp.) is a widely consumed vegetable with exceptionally large genomes in Solanaceae, yet its genomic evolutionary history remains largely unknown. Here we present 11 high-quality Capsicum genome assemblies, including two gap-free genomes, covering four wild and all five domesticated pepper species. We reconstructed the ancestral karyotype and inferred the evolutionary trajectory of peppers. The expanded and variable genome sizes were attributed to differential transposable element accumulations, which shaped 3D chromatin architecture and introduced mutations associated with traits such as fruit orientation and colour. Using a chromatin accessibility atlas of Capsicum, we highlight the influence of transposable elements on regulatory element evolution. Furthermore, by constructing a haploblock map of 124 pepper core germplasms, we uncover frequent introgressions that facilitate the formation of sweet blocky pepper and the acquisition of important traits such as resistance to pepper mild mottle virus. These findings on the genomic and functional evolution of Capsicum will benefit pepper breeding.