The tangential expansion of the human cerebral cortex, indexed by its surface area (SA), occurs mainly during prenatal and early postnatal periods, and is influenced by genetic factors. Here we investigate the role of rare copy number variants (CNVs) in shaping SA, and the underlying mechanisms, by aggregating CNVs across the genome in community-based cohorts (N = 39,015). We reveal that genome-wide CNV deletions and duplications are associated with smaller SA. Subsequent analyses with gene expression in fetal cortex suggest that CNVs influence SA by interrupting the proliferation of neural progenitor cells during fetal development. Notably, the deletion of genes with strong (but not weak) coexpression with neural progenitor genes is associated with smaller SA. Follow up analyses reveal similar mechanisms at play in three clinical CNVs, 1q21.1, 16p11.2 and 22q11.2. Together, this study of rare CNVs expands our knowledge about genetic architecture of human cerebral cortex.