Boosting adversarial transferability in vision-language models via multimodal feature heterogeneity.

 0 Người đánh giá. Xếp hạng trung bình 0

Tác giả: Long Chen, Yuling Chen, Hui Dou, Zhi Ouyang, Haiwei Sang, Yangwen Zhang

Ngôn ngữ: eng

Ký hiệu phân loại:

Thông tin xuất bản: England : Scientific reports , 2025

Mô tả vật lý:

Bộ sưu tập: NCBI

ID: 741333

Vision-language pre-training models have achieved significant success in the field of medical imaging but have exhibited vulnerability to adversarial examples. Although adversarial attacks are harmful, they are valuable in revealing the weaknesses of VLP models and enhancing their robustness. However, due to the under-utilization of modal differences and consistent features in existing methods, the attack effectiveness and migration of adversarial samples are not satisfactory. To address this issue and enhance attack effectiveness and transferability, we propose the multimodal feature heterogeneous attack framework. To enhance the adversarial capability, we propose a feature heterogenization method based on triplet contrastive learning, utilizing data augmentation and cross-modal global contrastive learning, intra-modal contrastive learning, and cross-modal global-local mutual information contrastive learning. This further heterogenizes the consistent features between modalities into distinct features, thereby improving the adversarial capability. To improve transferability, we propose a cross-modal variance aggregation-based multi-domain feature perturbation method, using text-guided image attacks to perturb consistent spatial and frequency features while combining previous gradient momentum, achieving better transferability. Extensive experiments demonstrate MFHA's significant advantage in transferable attack capability, with an average improvement of 16.05%, and outstanding attack performance on multimodal large language models like MiniGPT4 and LLaVA. The work we did has been open-sourced on GitHub: https://github.com/doyoudooo/MFHA .
Tạo bộ sưu tập với mã QR

THƯ VIỆN - TRƯỜNG ĐẠI HỌC CÔNG NGHỆ TP.HCM

ĐT: (028) 36225755 | Email: tt.thuvien@hutech.edu.vn

Copyright @2024 THƯ VIỆN HUTECH