Combined EEG and EMG decoding of reach-to-grasping: experimental protocol and processing pipeline

This thesis is focused on the design and evaluation of a multimodal convolutional neural network (CNN) combining EEG and EMG signals to decode reach-to-grasp movements. A set-up was designed to perform reach-to-grasp movements toward four different objects displayed randomly in front of the participant; also a no-movement condition was included, corresponding to no object presentation. The set-up consisted in a motorized rotatory wheel where the objects were placed around its circumference. Twelve participants, starting from a rest position called base, had to reach and grasp the object presented, and return to the base, or not moving if no object presented. Triggers generated by sensors were used to reconstruct phases distinguishing between a preparation phase (before a ‘go’ signal) during which the object was presented and a movement phase (after the ‘go’ signal) during which the subject performed the movement. 64 EEG and 16 EMG channels were acquired during the task. After preprocessing steps, EEG signals were analyzed in the time domain, exploring Movement Related Cortical Potential around the ‘go’ signal. This preliminary analysis proved significant differences encoded in the EEG signals among no-movement and movement classes, and, although to a less extent, among some object classes. Then a CNN with two branches (for EEG and EMG respectively) was designed based on EEGNet. Signals were split in chunks (representing the inputs of the network) using a moving window to simulate a real-time classification in an off-line decoding. The output features of the branches were concatenated before the final classification. A 2- class classifier (no-movement vs. movement) was first used to evaluate the improvements of the multimodal network compared to the unimodal ones (based only on EEG or EMG). Then, the multimodal network was used for 5-class classification provided good accuracy during the movement execution but limited during movement preparation.