Understanding genetic differences between populations is essential for avoiding confounding in genome-wide association studies and improving polygenic score (PGS) portability. We developed a statistical pipeline to infer fine-scale Ancestry Components and applied it to UK Biobank data. Ancestry Components identify population structure not captured by widely used principal components, improving stratification correction for geographically correlated traits. To estimate the similarity of genetic effect sizes between groups, we developed ANCHOR, which estimates changes in the predictive power of an existing PGS in distinct local ancestry segments. ANCHOR infers highly similar (estimated correlation 0.98 ± 0.07) effect sizes between UK Biobank participants of African and European ancestry for 47 of 53 quantitative phenotypes, suggesting that gene-environment and gene-gene interactions do not play major roles in poor cross-ancestry PGS transferability for these traits in the United Kingdom, and providing optimism that shared causal mutations operate similarly in different populations.