Category-selective regions in ventral temporal cortex (VTC) have a consistent anatomical organization, which is hypothesized to be scaffolded by white matter connections. However, it is unknown how white matter connections are organized from birth. Here, we scanned newborn to 6-month-old infants and adults to determine the organization of the white matter connections of VTC. We find that white matter connections are organized by cytoarchitecture, eccentricity, and category from birth. Connectivity profiles of functional regions in the same cytoarchitectonic area are similar from birth and develop in parallel, with decreases in endpoint connectivity to lateral occipital, and parietal, and somatosensory cortex, and increases to lateral prefrontal cortex. Additionally, connections between VTC and early visual cortex are organized topographically by eccentricity bands and predict eccentricity biases in VTC. These data show that there are both innate organizing principles of white matter connections of VTC, and the capacity for white matter connections to change over development.