Cognitive deficits in people with bipolar disorder (BD) may be the result of the illness or its treatment, but they could also reflect genetic risk factors shared between BD and cognition. We investigated this question using empirical genetic relationships within a sample of patients with BD and their unaffected relatives. Participants with bipolar I, II, or schizoaffective disorder ("narrow" BD, n = 69), related mood disorders ("broad" BD, n = 135), and their clinically unaffected relatives (n = 227) completed five cognitive tests. General cognitive function (g) was quantified via principal components analysis (PCA). Heritability and genetic correlations were estimated with SOLAR-Eclipse. Participants with "narrow" or "broad" diagnoses showed deficits in g, although affect recognition was unimpaired. Cognitive performance was significantly heritable (h