Breast cancer poses a significant health threat worldwide. Contrastive learning has emerged as an effective method to extract critical lesion features from mammograms, thereby offering a potent tool for breast cancer screening and analysis. A crucial aspect of contrastive learning is negative sampling, where the selection of hard negative samples is essential for driving representations to retain detailed lesion information. In large-scale contrastive learning applied to natural images, it is often assumed that extracted features can sufficiently capture semantic content, and that each mini-batch inherently includes ideal hard negative samples. However, the unique characteristics of breast lumps challenge these assumptions when dealing with mammographic data. In response, we introduce ManiNeg, a novel approach that leverages manifestations as proxies to select hard negative samples. As a condensed representation of a physician's domain knowledge, manifestations represent observable symptoms or signs of a disease and can provide a robust basis for choosing hard negative samples. This approach benefits from its invariance to model optimization, facilitating efficient sampling. We tested ManiNeg on the task of distinguishing between benign and malignant breast lumps. Our results demonstrate that ManiNeg not only improves representation in both unimodal and multimodal contexts but also offers benefits that extend to datasets beyond the initial pretraining phase. To support ManiNeg and future research endeavors, we have developed the MVKL mammographic dataset. This dataset includes multi-view mammograms, corresponding reports, meticulously annotated manifestations, and pathologically confirmed benign-malignant outcomes for each case. The MVKL dataset and our codes are publicly available at https://github.com/wxwxwwxxx/ManiNeg to foster further research within the community.