Semantic decoding of natural speech production from sEEG via deep learning

One of the most devastating neurological diseases that result in the loss of communication is amyotrophic lateral sclerosis (ALS) which is characterized by loss of voluntary muscle control and affects around 5 persons per 100,000 population. For people suffering from this disease, brain-computer-interface (BCI) technology aims to provide a means of communication to improve their quality of life and, to improve the communication rate of a BCI system, semantic decoding can be exploited. In this context this master thesis, developed in collaboration with Nick Ramsey’s BCI lab at UMC Utrecht, focuses on the feasibility of decoding semantic representations of speech, in the form of word embeddings, measuring neural activity through stereotactic electroencephalography (sEEG). The study involves two participants who read aloud a chapter from the Harry Potter book while sEEG electrodes recorded their brain activity. Word embeddings are generated using Bidirectional Encoder Representations from Transformers (BERT) model and Recurrent neural networks (RNN) are employed as decoding models to decode word embeddings directly from high frequency band [60-150 Hz] neural activity. This project is the first study that tried to decode word embeddings directly from brain activity and, due to the complexity of the task and the sparse coverage of sEEG electrodes, the decoding model achieves an accuracy only of 52%. Despite achieving poor accuracy, the findings of this exploratory study may support future projects in further developing speech BCI applications based on semantic decoding