TOWARDS CROSS-ATTENTION PRE-TRAINING IN NEURAL MACHINE TRANSLATION

• Tác giả: Hong Buu Long Nguyen, Vinh Khang Pham
• Ngôn ngữ: eng
• Thông tin xuất bản: Tạp chí Khoa học - Trường Đại học Sư phạm TP Hồ Chí Minh, 2022
• Mô tả vật lý: tr.1749
• Bộ sưu tập: Metadata
• ID: 336500

Tóm tắt

The advent of pre-train techniques and large language models has significantly leveraged the performance of many natural language processing (NLP) tasks. However, pre-trained language models for neural machine translation remain a challenge as little information about the interaction of the language pair is learned. In this paper, we explore several studies trying to define a training scheme to pre-train the cross-attention module between the encoder and the decoder by using the large-scale monolingual corpora independently. The experiments show promising results, proving the effectiveness of using pre-trained language models in neural machine translation.