Single-cell chromosome conformations vary significantly among individual cells. We introduce a two-step dimensionality reduction method for density-based, unsupervised clustering of single-cell 3D chromosome structures from simulations or multiplexed 3D-FISH imaging. Our method clusters up to half of all structures into 5-12 prevalent conformational states per chromosome. These states are distinguished by subdivisions into chromosome territory domains, whose boundary locations influence subnuclear positions and speckle associations of certain genes and establish long-range structural variations of more than 10 Mb. Territory domain boundaries are found at few sequence locations, shared among cell types and often situated at syntenic breakpoints.