Mind Reader: Reconstructing Complex Images From Brain Activities

Tech ID: 33164 / UC Case 2023-866-0


Understanding how the brain encodes external stimuli and how these stimuli can be decoded from measurable brain activities are long-standing and challenging questions in neuroscience. Progress has been made in reconstructing images from fMRI signals, but they are typically images of single objects or simple shapes. The advent of sophisticated deep learning models would introduce major advancements to this work, but the scarcity of fMRI datasets prevents researchers from applying state-of-the-art deep learning models to reconstruct images that are rich in semantic information and approach the likeness of an everyday scene.


Researchers at the University of California, Santa Barbara have created a technology that reconstructs complex, photo-realistic images observed by subjects from their brain signals. With more objects and relationships presented in these images, an additional text modality is used to better capture the semantics. To achieve high performance with limited data, a pre-trained semantic space aligns the visual and text modalities. fMRI signals are encoded to this visual-language latent space before a generative model conditioned on the mapped embeddings reconstructs the image. Additional contrastive loss is introduced to add low-level visual features into this semantic-based pipeline. As a result, the reconstructed images are both photo-realistic and can faithfully reflect the original image content. This brain signal-to-image decoding pipeline opens new opportunities to study human brain functions through strategic input alterations and can even potentially be helpful for human-brain interfaces.


  • Reconstructs a complex, photo-realistic image of what a human subject is seeing or visualizing based on fMRI signals
  • Overcomes data scarcity with pre-trained semantic space that aligns visual and text modalities


  • Neuroscience
    • Brain-computer interfaces (BCIs)

Related Materials


Learn About UC TechAlerts - Save Searches and receive new technology matches


  • Lin, Sikun
  • Singh, Ambuj K.

Other Information


Mind reader technology, Image reconstruction, Brain activities, fMRI signals, Deep learning models, Semantic information, Complex images, Photo-realistic images, Text modality

Categorized As