Polygenic scores (PGSs) are being rapidly adopted for trait prediction in the clinic and beyond. PGSs are often thought of as capturing the direct genetic effect of one's genotype on their phenotype. However, because PGSs are constructed from population-level associations, they are influenced by factors other than direct genetic effects, including stratification, assortative mating, and dynastic effects ("SAD effects"). Our interpretation and application of PGSs may hinge on the relative impact of SAD effects, since they may often be environmentally or culturally mediated. We developed a method that estimates the proportion of variance in a PGS (in a given sample) that is driven by direct effects, SAD effects, and their covariance. We leverage a comparison of a PGS of interest based on a standard GWAS with a PGS based on a sibling GWAS-which is largely immune to SAD effects-to quantify the relative contribution of each type of effect to variance in the PGS of interest. Our method, Partitioning Genetic Scores Using Siblings (PGSUS, pron. "Pegasus"), breaks down variance components further by axes of genetic ancestry, allowing for a nuanced interpretation of SAD effects. In particular, PGSUS can detect stratification along major axes of ancestry as well as SAD variance that is "isotropic" with respect to axes of ancestry. Applying PGSUS, we found evidence of stratification in PGSs constructed using large meta-analyses of height and educational attainment as well as in a range of PGSs constructed using the UK Biobank. In some instances, a given PGS appears to be stratified along a major axis of ancestry in one prediction sample but not in another (for example, in comparisons of prediction in samples from different countries, or in ancient DNA vs. contemporary samples). Finally, we show that different approaches for adjustment for population structure in GWASs have distinct advantages with respect to mitigation of ancestry-axis-specific and isotropic SAD variance in PGS. Our study illustrates how family-based designs can be combined with standard population-based designs to guide the interpretation and application of genomic predictors.