For the past couple months I’ve been head first into the world of brain computer interfaces (BCIs), a realm where we can all become magicians controlling the external world using our mind🧠🤯.
A paper I recently came across was particularly interesting, where it aimed to find discriminative EEG-based features and classification methods to categorize brainwave patterns based on their level of activity or frequency for mental state recognition. To get my hands further dirty in the space, I decided to replicate and build out the project for myself.
In this article, we’ll cover the fundamentals of the study on “Mental State Classification using EEG-based Brain-Machine Interface”, understanding the aim, data acquisition, statistical extraction, feature selection, model training, and end results. Following this, I’ll walk you through how I used an EEG dataset on mental states of subjects who were watching movies, to build a model and predict the emotional state of a subject during a given movie scene.
Brain computer interfaces are an emerging technology where brain signals are used to control external machines/environments. It involves three primary steps
- collecting brain signals
- interpreting the brain signals
- executing a command based on the interpretation
This research aims to find discriminative EEG-based features and classification methods to categorize brainwave patterns based on their level of activity or frequency for mental state recognition.
For step 1, where brain signals are collected, this can be done using one of the following three methods
Sensors are placed on the scalp to measure the electrical potentials produced by the brain (EEG) or the magnetic field (MEG).
Electrodes are placed on the exposed surface of the brain collecting electrocorticography (ECoG) data.
Micro-electrodes are placed directly into the cortex, measuring the activity of a single neuron.
In this study, electroencephalogram (EEG) data was non-invasively gathered from a range of subjects during these three emotional states
Using the features extracted from the EEG headband, five signals (alpha, beta, theta, delta, gamma) were tested a combination of different features selection algorithms and classifier models to compare their performance in terms of recognition accuracy and number of features needed.
With the many statistics extracted from the temporal data to form a dataset, classification was performed to predict the mental state based on the EEG statistics.
The application considered here when distinguishing between different mental states is to be able to potentially interface with a robot depending on the emotional state classified. This could also be useful in the domains of healthcare, education, and home automation.
Experimental Set-up & Data Acquisition
This study collected EEG signals using the non-invasive approach. The subjects (1 male, 1 female) used the Muse headband, where the data was collected for 3 minutes per state — positive, neutral, and negative. The Muse is a commercial EEG sensing device with five dry application sensors, one used as a reference point (NZ) and four (TP9, AF7, AF8, TP10) to record brain wave activity.
They placed the electrodes on and around the cranium to detect small electrical signals (voltage) produced by the brain, which is a side effect of the brain’s electrical internal activities.
Since all conscious human activities are triggered in the brain, we can use EEG to discern what a person is thinking, doing, and feeling.
Six minutes of resting neutral data was also recorded. The stimuli used to evoke the emotions are below.
After a short amount of time into the stimulus starting, as to not gather data with an inaccurate class, the EEG data from the Muse Headband was automatically recorded for sixty seconds.
The data was observed to be streaming at a variable frequency within the range of 150–270 Hz.
BlueMuse was used for interfacing the device to a computer, and Muselsl was used to convert the Muse signals to MicroVolts and record the data into a preliminary dataset ready for feature extraction.
The figure below shows a live stream of EEG data, blinking can be seen in the troughs of TP9 and TP10 (forehead sensors).
To prevent the interference of electromyographic signals, nonverbal tasks that required little to no movement were set. Blinking, though providing interference to the AF7 and AF8 sensors, was neither encouraged nor discouraged to retain a natural state.
In addition, subjects were asked not to close their eyes during any of the tasks.
The statistics extracted from each temporal window was used to produce the full dataset.
Each of these statistics for the time windows were then labelled to the nominal emotion, forming a dataset with very high dimensionality.
A major challenge with BCI applications is inferring how momentary mental states are mapped into a particular pattern of brain activity. One of the main issues with classifying EEG signals is the amount of data needed to properly describe the different states, since the signals are complex, non-linear, non-stationary, and random in nature.
The signals are considered stationary only within short intervals, which is why the best practice is to to apply short-time windowing technique in order to detect local discriminative features to meet this requirement.
Since the waves collected are complex, the data was down-sampled and then a sliding window of length 1 was run over it. This occurred every 0.5 seconds. The sampling rate was decimated to 200 Hz based on fast Fourier transformations along a given axis. Because the Fourier method is used, the signal is assumed to be periodic.
The set of features considered in this work to adequately discriminate different classes of mental states rely on statistical techniques, time-frequency based on fast Fourier transform (FFT), Shannon entropy, max-min features in temporal sequences, log-covariance, and others.
All features proposed to classify the mental states are computed in terms of the temporal distribution of the signal in a given time window.
This sliding window is defined as a period of 1 second and is computed every 0.5 seconds. All of the following statistical extractions are applied to each of the 5 brainwaves.
Since the dataset produced had over 2500 dimensions, being extremely intensive to classify, and possibly wasting computational resources, feature selection was performed to reduce the dataset. Four subsets were produced through feature selection.
Many different machine learning models were used to test which classification model was the best to categorize brain waves for mental state recognition.
The single models used included Random Trees, SVM, Naïve Bayes, Bayes Network, Logistic Regression, and a Deep Neural Network. Ensemble models like Random Forest and Adaptive Boosting were also tested.
The best single model was the deep neural network which achieved 94.89% in terms of accuracy, but this was outperformed by the random forest classifier which has an overall best score of 97.89% accuracy.
How I Built My Own Mental Emotional Sentiment Classifier
I used another dataset of EEG brainwaves from positive and negative emotional experiences captured from the brain. This was recorded from when subjects were watching a series of five different movie scenes, and I used the dataset to try to predict the emotional state of a given subject during a given movie.
I replicated this from a tutorial by Gabriel Atkin (see sources at the end for it). If you’d like to check out my Google Collab notebook for the code, here you go!
Ps. The data collected here is different that of which was referred to in this paper.
I used a tensorflow recurrent neural network to make the predictions.
I began by importing
- numpy and pandas for working with the data
- matplotlib.pyplot and seaborn for visualizations
- train_test.split function from sklearn for making the split
- confusion_matrix and classification_report from sklearn to analyze the results.
- csv EEG dataset file
Then read the data using pd.read_csv()
This was treated like timeseries data, and I used sequential processing methods in the model to see if that would improve its performance.
Here we looked at part of the data as a graph, took a sample and started with the first row to plot it.
Then I took a look at the class distribution by getting the value count for the label column at the end. It displayed a fairly balanced class distribution, just slightly more of a neutral split. Then I created a label mapping to match each of the emotion states to an integer value.
Here I converted the label column from categorical to numerical by applying the label mapping from the previous piece of code. Then the dependent and independent variables from the data frame were separated, and then split into training data and testing data. Here we are using 70% of the data for training and 30% of the data for testing.
A model was created with one Gated Recurrent Unit (GRU) layer, a type of recurrent neural network.
For the model, adam is used as the optimizer and accuracy as the metric. Then the model was fit with the training data.
Here the accuracy of the model is calculated by comparing the predicted y values to the actual y value. The test accuracy was then printed (3 decimal places, in percentage form), which showed the model to be 96.562% accurate.
Then, I got a confusion_matrix (cm) to show how many of the predictions made were correct and how many were incorrect for each of the three emotional state classifications using a visual plot. The classification_report that was also printed shows the model’s performance metrics such as recall, f1-score, and precision, in addition to the accuracy. Here we can see that the metrics are all above 95% which indicates that the model is performing well.
- Kaggle Dataset: ‘EEG Brainwave Dataset: Feeling Emotions’
- Paper: Mental Emotional Sentiment Classification with an EEG-based Brain-machine Interface
- Paper: A Deep Evolutionary Approach to Bioinspired Classifier Optimization for Brain-Machine Interaction
- Paper: A Study on Mental State Classification using EEG-based Brain-Machine Interface
- Jordan Bird’s Code Documentation
- Gabriel Atkin’s Tutorial Part 1, Part 2 [from where I learnt about, conceptually understood, and then replicated this project]