Mind Reader: Reconstructing Complex Images From Brain Activities

Background

Understanding how the brain encodes external stimuli and how these stimuli can be decoded from measurable brain activities are long-standing and challenging questions in neuroscience. To better understand visual encoding and decoding processes, researchers have curated multiple datasets recording functional magnetic resonance imaging (fMRI) signals while the subjects are viewing natural images. Progress has been made in reconstructing images from fMRI signals, but they are typically images of single objects or simple shapes. The advent of sophisticated deep learning models would introduce major advancements to this work, but the scarcity of fMRI datasets prevents researchers from applying state-of-the-art deep learning models to reconstructing images that are rich in semantic information and approach the likeness of an everyday scene.

Description

Researchers at the University of California, Santa Barbara have created a novel technology that reconstructs complex image stimuli from fMRI signals. This method constructs photo-realistic images observed by subjects from their brain signals. Unlike previous works that reconstructed images of single objects or simple shapes, this technology reconstructs image stimuli that are rich in semantics, closer to everyday scenes, and reveal more perspectives. With more objects and relationships presented in these images, an additional text modality is used to better capture the semantics. To achieve high performance with limited data, a pre-trained semantic space aligns the visual and text modalities. fMRI signals are encoded to this visual-language latent space before a generative model conditioned on the mapped embeddings reconstructs the image. Additional contrastive loss is introduced to add low-level visual features into this semantic-based pipeline. As a result, the reconstructed images are both photo-realistic and can faithfully reflect the original image content. This brain signal to image decoding pipeline opens new opportunities to study human brain functions through strategic input alterations and can even potentially be helpful for human-brain interfaces.

Advantages

Applications

Patent Status

Patent Pending

Related Materials

• Additional Information - 04/07/2023
• Publication - 09/30/2022