Systems and Methods for Accurate and General-purpose Semantic Decoding Using Brain Signals

Abstract

Researchers at the University of California, Davis have developed an advanced system that decodes semantic content directly from neural signals to enable natural language communication for users with speech impairments.

Full Description

This technology provides systems and methods for decoding semantic meaning from brain neural signals by mapping recorded neural activity to semantic embeddings in the latent space of large language models. Leveraging transformer-based semantic neural encoders trained with paired intracortical neural and language data, the system decodes attempted communication into natural language. This process includes iterative refinement using lookup tables and large language model guess generation to enhance accuracy. The method supports various neural recording modalities and is designed to assist users with language impairments, such as aphasia, enabling improved communication through synthesized speech or display devices.

Applications

• Assistive communication devices for individuals with aphasia, paralysis, or other severe speech disabilities (resulting, for example, from stroke, traumatic brain injury, or neurodegenerative diseases such as primary progressive aphasia, amytropic lateral sclerosis), as well as potentially individuals who never learned to speak due to e.g. cerebral palsy, autism, or cerebellar disorders. 
• Neuroprosthetics enabling natural language interaction for patients with brain injuries or neurodegenerative diseases. 
• Advanced brain-computer interfaces for speech-impaired users in clinical and rehabilitation settings. 
• Augmentation tools for speech therapy and cognitive rehabilitation programs. 
• Integration with vocal synthesis and display technologies for seamless communication systems. 
• Research platforms for neural decoding and cognitive neuroscience investigations.

Features/Benefits

• Decodes semantic content directly from neural activity, bypassing speech-motor generation. 
• Captures richer meaning by using large-language-model embeddings rather than surface text features. 
• Applicable to invasive and noninvasive neural recording modalities to broaden deployment options. 
• Improves decoding robustness by applying transformer-based neural encoders and semantic models. 
• Boosts sentence accuracy by iteratively refining candidate outputs during decoding. 
• Delivers communication through synthesized speech or visual text output. 
• Generalizes across users/sessions by leveraging self-supervised training and data augmentation. 
• Restores communication for people with aphasia or other speech impairments. 
• Enables expression of intended meaning when speech production is limited or impossible. 
• Translates complex neural signals into natural language beyond simple command/classification outputs. 
• Improves semantic decoding fidelity without requiring intact speech motor function.

Patent Status

Patent Pending