Generative language reconstruction from brain recordings.

 0 Người đánh giá. Xếp hạng trung bình 0

Tác giả: Qingyao Ai, Maarten de Rijke, Christina Lioma, Yiqun Liu, Tuukka Ruotsalo, Ziyi Ye, Min Zhang

Ngôn ngữ: eng

Ký hiệu phân loại:

Thông tin xuất bản: England : Communications biology , 2025

Mô tả vật lý:

Bộ sưu tập: NCBI

ID: 741294

Language reconstruction from non-invasive brain recordings has been a long-standing challenge. Existing research has addressed this challenge with a classification setup, where a set of language candidates are pre-constructed and then matched with the representation decoded from brain recordings. Here, we propose a method that addresses language reconstruction through auto-regressive generation, which directly uses the representation decoded from functional magnetic resonance imaging (fMRI) as the input for a large language model (LLM), mitigating the need for pre-constructed candidates. While an LLM can already generate high-quality content, our approach produces results more closely aligned with the visual or auditory language stimuli in response to which brain recordings are sampled, especially for content deemed "surprising" for the LLM. Furthermore, we show that the proposed approach can be used in an auto-regressive manner to reconstruct a 10 min-long language stimulus. Our method outperforms or is comparable to previous classification-based methods under different task settings, with the added benefit of estimating the likelihood of generating any semantic content. Our findings demonstrate the effectiveness of employing brain language interfaces in a generative setup and delineate a powerful and efficient means for mapping functional representations of language perception in the brain.
Tạo bộ sưu tập với mã QR

THƯ VIỆN - TRƯỜNG ĐẠI HỌC CÔNG NGHỆ TP.HCM

ĐT: (028) 36225755 | Email: tt.thuvien@hutech.edu.vn

Copyright @2024 THƯ VIỆN HUTECH