Open Access
Add to BibSonomyAbstract
Previous studies reported mental stress as one of the major contributing factors leading to various diseases such as heart attack, depression and stroke. An accurate stress assessment method may thus be of importance to clinical intervention and disease prevention. We propose a joint independent component analysis (jICA) based approach to fuse simultaneous measurement of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) on the prefrontal cortex (PFC) as a means of stress assessment. For the purpose of this study, stress was induced by using an established mental arithmetic task under time pressure with negative feedback. The induction of mental stress was confirmed by salivary alpha amylase test. Experiment results showed that the proposed fusion of EEG and fNIRS measurements improves the classification accuracy of mental stress by +3.4% compared to EEG alone and +11% compared to fNIRS alone. Similar improvements were also observed in sensitivity and specificity of proposed approach over unimodal EEG/fNIRS. Our study suggests that combination of EEG (frontal alpha rhythm) and fNIRS (concentration change of oxygenated hemoglobin) could be a potential means to assess mental stress objectively.
© 2016 Optical Society of America
1. Introduction
Stress is defined as the non-specific response of the body/mind to any demand of change [1]. Stress can affect the responsiveness of our central-peripheral regulatory system leading to a poorer health [2]. Stress causes the activation of hypothalamus-pituitary-adrenocortical axis (HPA axis) and sympathetic nervous system (SNS) leading to an increase in the stress hormone (cortisol) in adrenal cortex. The continuous release of cortisol has a direct impact on our body, function and structure of the brain. It increases blood pressure, weakens immune system, and stops generating new neurons in the hippocampus [3, 4]. Besides, it increases the size and the activity of amygdala which involved in storing memories associated with emotional events [5]. Stress has been recognized as one of the major factors contributing to chronic disorders and productivity loss. Long-term exposure to stress has been linked to a variety of health problems such as heart disease, obesity, diabetes, stroke and depression [6–10]. Therefore, stress assessment at early stage is important before our health deteriorates.
Stress can be measured and evaluated based on perceptual, behavioural and physiological responses. Physicians traditionally evaluate stress using questionnaires [11]. However, evaluating stress using questionnaires is a subjective method [12, 13]. A more objective measure is cortisol [14, 15] and alpha amylase [16, 17]. Alternative measures with faster response time are bio-signals, namely: heart rate, blood pressure (BP) and skin conductivity (SC) [18–22]. Direct measurements on cortical response to stress can be obtained non-invasively through modern neuroimaging modalities. Electroencephalography (EEG) is one of the most commonly used neuroimaging modalities to study brain functions and conditions. EEG measures the fluctuations of electrical fields due to en-masse neuronal activity at millisecond resolution [23, 24]. EEG signals are often assessed in several distinct frequency bands, such as: Delta (1-4 Hz), Theta (4-8 Hz), Alpha (8-12.5 Hz) and Beta (12.5-30 Hz) to examine their relationship with the emotional states. Alpha and beta frequency power are linked to negative mood, stress and depression [25–28]. Separately, Marshall and Lopez-Duran reported about negative correlation between EEG alpha power rhythm and stressful events in the prefrontal cortex where alpha rhythm reduced with stress [29, 30]. Choi et al found a positive correlation between EEG beta power rhythms with stress in the temporal lobe [31]. On the other hand, frontal theta power has been linked to task difficulty (theta rhythm decreases with increasing task difficulty) [32]. Several studies have showed that EEG signals could be used to classify mental stress from resting state [33–37]. However, EEG is susceptive to noise and has a poor spatial resolution [38]. To overcome this, combining EEG with another neuroimaging modality that has a complementary nature may be an option.
Functional Near-Infrared Spectroscopy (fNIRS) is a new neuroimaging technique which measure the cerebral hemodynamics associated with neural activity. The technique sends near-infrared light directly into the head [39]. Based on the absorptivity, the change in concentrations of oxygenated (O2Hb) and deoxygenated (HHb) hemoglobin can be estimated using modified Beer-Lambert law [40]. Compared to EEG, fNIRS offers a better spatial resolution of cortical activation. The fNIRS has found its applications in cognitive and behavioural studies. Commonly used tasks to activate the prefrontal cortex (PFC) include mental arithmetic, word generation, colour-word matching, Stroop task, mental rotation, working memory task and inhibition [41–43]. The PFC is related to working memory which enables us to hold in mind and mentally manipulate information over a short period of time. These aforementioned tasks are used in brain-computer interface (BCI) and attention studies [44–48]. More recently, fNIRS has been accepted as an assistive tool to differentiate depression, bipolar and schizophrenia [49, 50].
In this work, we hypothesize that fusion of EEG and fNIRS can assess mental stress more accurately than using either technique alone. The EEG and fNIRS have several advantages over other neuroimaging modalities (e.g. Magnetic Resonance Imaging MRI, Positron Emission Tomography PET) as they are non-invasive, portable, less expensive, safe for long-term monitoring, and reported to be a good complementary [51, 52]. We propose joint Independent Component Analysis (jICA) to fuse EEG and fNIRS measurements [53–55]. The jICA technique has been previously developed for integrating EEG and fMRI signals, to improve spatio-temporal resolution [56, 57]. Whilst the fusion may be achieved by several approaches, we used the fNIRS response to identify key EEG measurement nodes, and then introduced jICA fusion technique to integrate features from both modalities at feature level.
2. Materials and methods
2.1 Participants
Twenty two healthy male, right-handed adults (aged 26 ± 4 with head size of 56 ± 2 cm) participated in the simultaneous EEG and fNIRS measurement study. To avoid influences of circadian rhythm on alpha amylase measurement, the experiment was conducted between 3.00 and 4.30 p.m [58]. All participants were asked not to eat or chew gum at least two hours before the experiment. They were informed about the experiment and gave written consent prior to the experiment. The experiment procedures were in accordance with the declaration of Helsinki and approval granted by ethics review committee of Universiti Teknologi PETRONAS. None of these participants has a history of psychiatric, neurological disorders or psychotropic drug use. The participants were seated in a comfortable room with good air condition to avoid any environmental stress and they were also asked to minimize head movement throughout the entire experiment.
2.2 Experiment setup and task sequence
The experiment was designed based on Montreal Imaging Stress Task (MIST) [59]. The experiment protocol was performed in four steps. Step 1: a brief introduction was given to the participants to familiarize them with the proposed tasks. Step 2: the participants were trained for five minutes in a mental arithmetic (MA) task in order to estimate time taken to answer each question by individual. The task involved 3 one-digit integers (ranging from 0 to 9) and the operators were limited to + or – (for example 7-3 + 1). The answer for each question was displayed on a computer monitor in the sequence of ‘0’ to ‘9’ (as shown in Fig. 1) and participant has to select the right answer by single left-click on the mouse. Step 3 (i.e. the control phase): simultaneous measurements of EEG and fNIRS were performed for a total of 5 minutes without time limit per question. Participants were instructed to answer the questions as quickly and as accurately as possible, and they received no feedback if their answer was correct. Step 4 (i.e. stress phase): the average time recorded during the training phase was reduced by 10% and was set as a time limit. Answering wrongly or failing to answer each question within the time limit, the participants would receive a negative feedback, i.e. a message of “Incorrect” or “Time’s up” being displayed on the monitor. Furthermore, the average peer performance set to 90% was displayed on the screen to further increase the stress on the participants. In actual fact, the participants were expected to score 40-50% when the time given to answer each question was reduced by 10%. The entire recording (control phase and stress phase) took a total duration of nearly 25 minutes, each phase consisted of five blocks of EEG + fNIRS recording. Prior to each recording phase (control and stress), baseline was measured for a total duration of 20 s. During the baseline recording participants were instructed to look at a fixation cross on the computer monitor and to get ready for the next task. Figure 1 gives an overview of the block design of the task. In each block, mental arithmetic task was introduced for 30 s followed by 20 s rest.
Fig. 1 Experiment block design. A total of five active blocks existed for each of the (a) control and (b) stress condition. In each block, arithmetic tasks produced for thirty second followed by twenty second rest. During the 30 s task, several arithmetic questions would be posted depend on how fast the respond of the participant in answering each question. If the respond rate is 2 s per question, the total number of questions per block would be 15 questions and the total number of questions would be 75 questions, for example. During the 20 s rest, the computer screen would display with a fixation cross with black background and participants were instructed to look at the fixation cross as a visual cue for trial onset. The red dashed-line marks the start of the task and the green dashed-line marks the end of the task (the marker was presented in every block). The stressors were based on time pressure and negative feedback of individual performance as demonstrated in (b). Five samples (S1-S5) of alpha amylase were collected; 5-minutes before the control condition as baseline for control, immediately after control condition, 5-minutes before stress condition, immediately after stress condition and 5-minutes after stress condition as marked in the figure with yellow rectangular.
During the experiment, all participants were instructed to answer each question correctly and not to guess the answer. To evaluate if the participants paid attention to the task, their accuracies in answering the questions were calculated. The average score was 90% accuracy in the control phase, and 40% in the stress phase, as expected in the original MIST article [59].
2.3 Salivary alpha amylase sample collection
We used a hand-held monitor COCORO meter (Nipro, Osaka, Japan) to measure salivary alpha amylase activity. Five samples of salivary alpha amylase were collected for each participant during the experiment. The first sample was collected five minutes before the beginning of the experiment as a baseline sample (S1). Second sample was collected immediately at the end of control task (S2). Third sample was collected five minutes after the control phase as recovery (S3). Fourth sample was collected immediately at the end of the stress phase (S4). Fifth sample was collected five minutes after the stress phase (S5). Note that, in the analysis section, the third sample (S3) was also used as the baseline for the stress phase. The entire sequence for sample collection is as demonstrated in Fig. 1.
2.4 Data acquisition
Simultaneous measurements of EEG and fNIRS were performed during both phases (control and stress). Brain activities were recorded at the PFC area using a custom designed probe holder for seven EEG electrodes [FP1, F7, F3, Fz, FP2, F8, and F4] plus one reference electrode [A1] attached to the earlobe and 16 fNIRS optodes (equivalent of 23 fNIRS channels). The layout of the probe holder is shown in Fig. 2. The sampling frequency for EEG (BrainMaster 24E system) was set to 256 Hz and the impedance was minimized by using small amount of gel directly to the scalp (maximum impedance was kept to be less than 5kΩ in this experiment). The fNIRS system OT-R40 (Hitachi Medical Corp, Japan) was equipped with light sources at wavelength of 695 nm and 830 nm, and a sampling rate of 10 Hz. The distance between each of the source and detector of fNIRS probes layout was set to 3 cm. The control of simultaneous measurement was implemented in MATLAB and triggers were sent to BrainMaster 24E and OT-R40 systems through parallel and serial ports respectively to mark the start and the end of each task.
Fig. 2 EEG + fNIRS probe holder, (a) fNIRS channels and electrodes marking in the probe holder, (b) inner-view of the probe holder, (c) outer-view of the probe holder. The holder consisted of eight sources/emission probes and eight detection probes. Total of twenty three channels and seven active EEG electrodes involved in the probe holder.
2.5 Pre-processing
EEG data were pre-processed with EEGLAB 2013a toolbox [60]. Firstly, the EEG data were bandpass filtered in the range of 0.5 Hz to 30 Hz using 3rd order Butterworth filter. Secondly, eye blink artifacts were removed using Independent Component Analysis (ICA) technique. Then, EEG data were further processed using wavelet transform (WT) to divide the EEG data into different frequency bands with a resolution matched to its frequency scale. In this work, we used WT to decompose the EEG data into four frequency bands; Delta (1-4 Hz), Theta (4-8 Hz), Alpha (8-12.5 Hz) and Beta (12.5-30 Hz). From the wavelet coefficients of Alpha and Beta frequency bands signals we extracted our features. These features were then used for statistical analysis and classification.
The fNIRS signals were transformed to the change in products of mean pathlengths and concentration changes of O2Hb, HHb and total Hb using modified Beer-Lambert law. In order to reduce noise and artifacts, fNIRS signals were passed through several pre-processing steps using plug-in analysis software Platform for Optical Topography Analysis Tool [61]. The process included filtering the signal in the range of 0.012 to 0.8 Hz using 5th order Butterworth filter to reduce the physiological noise of low and high frequency such as respiration and cardiac related fluctuations in oxygen supply. Additionally, baseline correction, epoch extraction and moving average were applied to the fNIRS signals. In baseline correction, we defined a period starting from the onset of the task condition to the end of each task condition as one analysis block. Then, we applied linear regression by least mean squares to determine the linear trend of its baseline [62, 63]. Finally, data were averaged in all the analysis blocks [64]. As responses were more pronounced in O2Hb, we limited our analysis to O2Hb signals [65].
2.6 Feature extraction and classification
For each analysis block, features were extracted by calculating the mean powers of EEG in Alpha and Beta frequency bands signals and the mean concentration of O2Hb over the analysis block using a moving time-window of 500 ms, according to the following equations:
where Pj is the EEG power, xj(n) represents the segmented EEG signal in Alpha band at j = 1 and Beta band at j = 2 and N is the length of the signal.where ΔO2Hb represents the segmented O2Hb signal and N is the length of O2Hb signal.In this study, we used support vector machine (SVM) as a common classifier for the performance assessment of individual modality and the fusion of EEG and fNIRS modality [66]. SVM was selected for its ability to model linear boundaries as well as more complex decision planes. The kernel function used in this paper was based on Radial Basis Function (RBF) defined as:
where x and y are the two data points and is the width of RBF.The EEG and fNIRS signals were computed at 0.5 Hz in feature extraction stage. This gave 60 samples of EEG/fNIRS per analysis block. These samples were then fed as input features for the SVM. In this study, we adopted leave-one-out approach for cross validation among the 22 subjects.
2.7 EEG and fNIRS fusion
In this study, EEG and fNIRS data were fused in two consecutive steps. Firstly, we used the advantages of spatial resolution of fNIRS to identify spatial locations of interest (regions of interest). The selection of EEG electrodes was based on the significance of their neighbouring fNIRS channel response to the mental stress task. Secondly, we explored the spatial and temporal advantages of each modality. EEG sources were transformed via temporal ICA into tICA components and fNIRS sources on the other hand were transformed via spatial ICA into sICA components. We then applied joint independent component analysis (jICA) to compute the mixing matrix A, between tICA components of EEG and sICA components of fNIRS using the following generative model:
Assuming we have two sources per modality (EEG and fNIRS) and two subjects, then the mixed data for EEG is and the mixed tICA components/sources of EEG is:Similarly, the mixed data for fNIRS modality of the two subjects is presented by and the fNIRS sICA components is:And the shared linear mixing matrix A is given by:To form a data and source vector for each subject, the two data and sources set then concatenated and the given equations can be written in a single matrix equation,To estimate the mixing matrix A, we employed the infomax algorithm [67]. The infomax algorithm uses a natural gradient ascent technique to maximize the output entropy of a neural network. In this context, entropy refers to the independence between the ICA components. The weight matrix of the neural network, W refers to the inverse matrix of the shared mixing matrix, A. The optimization of the weight matrix is achieved by weight updating rule: where I is the identity matrix, , are the estimated independent sources of EEG and fNIRS, and are the regenerated EEG and fNIRS data respectively. From Eq. (14), is the nonlinear transfer function in the neural network [68]. The initial value for W, W(0) is a matrix composed of random vectors [69]. In our fusion approach, we assumed that the sources associated with EEG and fNIRS data modulated the same way across all subjects. This assumption of common linear covariation for both modalities presents a parsimonious way to link multiple data types and has led to improved results in fMRI studies [70, 71]. Furthermore, unlike Calhoun model [72], our fusion is performed at feature level.3. Result and analysis
3.1 Salivary alpha amylase
The increase in salivary alpha amylase level in response to stress task was noticeable in all the participants (refer to Fig. 3). We further analyzed the salivary alpha amylase responses using two-sample t-test. Assuming the two distributions have the same variance, the t-test was calculated using the following equation:
where,. Here is the grand standard deviation, the order of represents the group in control and stress condition respectively, are the unbiased estimators of the variance. Note that, the denominator of t is the standard error of the differences between mean of control and mean of stress group. The degree of freedom was set to 2n-2, where n is the number of participants in each group. We found the increase in alpha amylase level during stress condition to be significant, as compared to control condition, with mean p-value of <0.001. This confirmed that the proposed task induced stress successfully.Fig. 3 Salivary alpha amylase responses under control and stress condition. Blue color shows the salivary alpha amylase response under control condition at three measurement instances (5 min before (baseline), at the end of control condition (Task), 5 min after the task (recovery). Red colour shows the salivary alpha amylase response under stress condition with three measurement times (5 min before (baseline), at the end of stress condition (Task), 5 min after the stress task (recovery).The marks “***” indicate that, the task is significant with p<0.001.
3.2 EEG analysis
The cortical activation of the brain during the stress task revealed an increase in the beta rhythm power and a significant decrease in the alpha rhythm power on the PFC, respectively. Figure 4 shows the boxplot representing the normalized mean power values of alpha and beta under control and stress condition in all the 22 subjects. The result also demonstrated that, alpha rhythm responded more significantly to mental stress, compared to beta rhythm. This suggests alpha rhythm may be a better indicator of mental stress.
Fig. 4 Normalized Alpha and Beta rhythm power values in two mental state: control and stress for average of 22-subjects. The Alpha and Beta rhythm power values were calculated from all the measured electrodes on the PFC.
3.3 fNIRS analysis
The fNIRS result demonstrated significant increase in O2Hb concentration change during control condition in most PFC regions. In contrast, less cortical activation and, in some cases, significant decline in O2Hb concentration change from baseline was observed during stress phase. The mean time-course of O2Hb concentration change on selected channels is shown in Fig. 5. Figure 5(a) shows the time-course at Ch14 in which the O2Hb concentration was significantly increased from baseline during the control condition (red line), but did not increase when given similar stimuli during stress condition (blue line). Figure 5(b) shows the time-course at Ch17 in which the O2Hb concentration significantly increased during the control condition (red line) and increased at much reduced amplitude during stress condition (blue line). Under both control and stress conditions, the concentration change in O2Hb returns to their baseline at the end of rest condition (last 20 s). The overall behavioural of PFC regions under control and stress conditions is presented by their topographical maps in Fig. 6.
Fig. 5 Mean time-courses of oxygenated hemoglobin concentration changes, (a) control and stress conditions at Ch14 and (b) control and stress conditions at Ch17. The vertical red dash-line marks the start of the task and the vertical green dash-line marks the end of the task condition.
Fig. 6 Topographical map of oxygenated hemoglobin activation for average of 22 subjects, (a) under control condition and (b) under stress condition. Red colour indicates higher activation and blue colour indicates less activation. Under stress condition, reduced hemodynamic response around the right PFC region.
The topographical maps of O2Hb response measured and averaged from all the subjects are shown in Fig. 6(a) (control case) and Fig. 6(b) (stress case), with channel number labelling. Examining closer, we found, on average, the decreased O2Hb response was highly localized to the right PFC.
4.1 Statistical analysis
Statistical analysis on all EEG electrodes and all fNIRS channels are summarized in Table 1. We identified the EEG electrodes and fNIRS channels of interest to be used for performance evaluation/classification. For EEG modality, six electrodes with the highest t values were selected; F4, FP2, F8, Fz, F3 and F7 (see Table 1). Since alpha rhythm responds more significantly to stress than beta rhythm, we only considered alpha rhythm features for further analysis. For fNIRS modality, since most of the channels respond significantly to the task, we set a threshold value of t ≥ 5 for channel selection. Six channels respond above the threshold, with four channels located on the right PFC and two channels located on the left PFC, namely Ch4, Ch9, Ch14, Ch15, Ch18 and Ch23 (see Table 1). These channels were then selected for fNIRS performance evaluation/classification.
Table 1. Statistical analysis of EEG alpha and beta and O2Hb of fNIRS measurements based on two-sample t-test. In terms of electrode naming, F4A represents EEG electrode F4 in Alpha band, F4B represents EEG electrode F4 in Beta band, and so on.
For EEG and fNIRS fusion, we maintained the same number of features for performance evaluation. Three fNIRS channels with the highest t values were selected, and the EEG electrodes located in the closest proximity to these three fNIRS channels were also selected. The three fNIRS + EEG pairs are: Ch-4 with F4, Ch-15 with FP2, and Ch-18 with F7 (refer to Fig. 2(a) and Table 1), and their signals were used as inputs to the proposed jICA model. The fusion model was then evaluated using SVM, in the same manner as unimodal EEG and fNIRS.
The classification accuracy obtained from SVM classifier is shown in Fig. 7. It was calculated as the summation of all true data points (correctly labelled as stress or control) over the total data points of both conditions. The classification results are illustrated as boxplot for all the three modalities, namely EEG, fNIRS and EEG + fNIRS. The classification accuracy was 91.7 ± 5.3% for EEG, 84.1 ± 6.8% for fNIRS, and 95.1 ± 3.9% for fusion of EEG + fNIRS for averaged of 22-subjects. The fusion increased the accuracy by an average of + 3.4% compared to sole EEG and + 11.0% to fNIRS alone. The improvements was found to be significant, p<0.001 using t-test.
Fig. 7 Boxplots representing the classification accuracy measured by SVM for 22 subjects. The results demonstrate significant improvements in the mean classification accuracy when combining both modalities, p<0.001. High improvement in the classification due to combining both modalities with + 3.4% compared to EEG alone and + 11% compared to fNIRS alone.
The classification results on sensitivity and specificity are shown in Fig. 8 and Fig. 9. The sensitivity is defined as the probability of obtaining a positive test result (i.e. classifier prediction) given that stress is present. Similarly, the specificity is the probability of obtaining a negative test result given that stress is absent. The classification sensitivities of EEG, fNIRS and EEG + fNIRS were 90.4 ± 5.7%, 82.4 ± 6.3% and 94.2 ± 4.3%, respectively. There were significant improvements (p<0.001) in the sensitivity of fusion approach (EEG + fNIRS) compared to unimodals with an average improvement of + 3.8% for EEG and + 11.8% for fNIRS alone, respectively. The classification specificities of EEG, fNIRS and EEG + fNIRS, on the other hand, were 93.4 ± 4.4%, 86 ± 7.2% and 96.6 ± 2.8%, respectively. Significant improvements due to the fusion over individual modality were observed, as expected, with p<0.001.
Fig. 8 Boxplots representing the classification sensitivity calculated for 22 subjects. High improvement in the sensitivity occurred when combining both modalities with + 3.8% compared to EEG alone and + 11.8% compared to fNIRS alone, p<0.001.
Fig. 9 Boxplots representing the classification specificity calculated for 22 subjects. High improvements in the specificity obtained when combining both modalities with + 3.2% compared to EEG alone and + 10.6% compared to fNIRS alone, p<0.001.
4. Discussion
This study investigated if the fusion of EEG and fNIRS signals could help improve the detection of mental stress. The mental states of twenty two healthy subjects performing mental arithmetic task under neutral-control and stressful condition were studied. Brain hemodynamic responses and electrical brainwaves were acquired simultaneously using fNIRS and EEG. The key features were identified using statistical analysis and were fused based on jICA technique. The detection rate of mental stress by the proposed fusion approach was then compared and analyzed statistically with the detection rate by individual modality (EEG/fNIRS).
Using alpha amylase as a reference, we confirmed that the applied time- and peer-pressure did increase significantly (p<0.001) the stress level of all participants, as compared to baseline and control condition. The EEG results showed significant decrease in alpha rhythm power (p<0.01) and increase in beta rhythm power (p<0.02) on the PFC under stress condition. The decrease in alpha rhythm power in this study is consistent with previous emotional and anxiety studies [28, 73–75]. Additionally, previous EEG studies, reported an increase in alpha and decrease in beta rhythm in the frontal cortex under fatigue (fatigue task with 90-150 minutes time) which is not the case in this study [76, 77]. In this study, we intentionally kept each recording phase limited to five minutes. This is to ensure the increment in alpha amylase level in this study is due to stress but not to other factors [78].
The fNIRS results showed significant reduction in hemodynamic response (O2Hb concentration change) on the PFC during stress condition compared to control condition, with p<0.01. This observation of reduction in cortical activities on the PFC is consistent with the results from previous fMRI human and animal stress studies [79–84]. Our statistical analysis results further demonstrate that stress response was highly localized to the right PFC, (p<0.00001). As we know, gender difference affects brain functions such as memories, emotions, solving certain problems, decision making [85–87]. Thus, we aim to extend this to female subjects in our future study.
Using support vector machine (SVM) as classifier, we were able to classify brain activities under stress from that of neutral-control state significantly. The performance of SVM classifier was evaluated based on accuracy, sensitivity and specificity. Using features from EEG only gives a mean classification of 91.7 ± 5.3%, 90.4 ± 5.7% and 93.4 ± 4.4% accuracy, sensitivity and specificity respectively. In contrast, features from fNIRS were classified with 84.15 ± 6.8, 82.4% ± 6.3 and 86% ± 7.2 accuracy, sensitivity and specificity respectively. These fNIRS results outperformed previous studies [47, 88–98]. It however needs to be cautious as the data sets were different. Direct comparison is thus not possible. Fusion of EEG and fNIRS signals in the feature level demonstrated the highest classification performance. Using the same classifier as in individual modality (SVM), the proposed fusion technique demonstrated mean classification of 95.1% ± 3.9, 94.2% ± 4.3 and 96.6% ± 2.8 accuracy, sensitivity and specificity respectively.
The experiment results revealed that significant improvements were achieved by the proposed fusion approach in all three performance metrics, with p<0.001. This confirmed our hypothesis that fusion of EEG and fNIRS could improve the mental stress detection. The proposed feature-fusion model enabled us to take the advantages of the strengths of both modalities in unified analytic. The improvements in classification accuracy achieved in this study is also consistent with previous EEG and fNIRS fusion studies but not in all cases [99–101]. The inconsistency in fusion results may due to the level of fusion being adopted. Both Fazli and Putze applied fusion at decision level, i.e. using a meta classifier to integrate the outputs from one EEG classifier and one fNIRS classifier. It is likely that the outputs from the EEG classifier and the fNIRS classifier were highly correlated with little complementary information. In contrast, Yin was able to improve consistently the decoding of motor imagery tasks when considered feature level fusion of bimodal EEG and fNIRS [102]. Our experiment results further support the use of feature level fusion.
In this work, we used simple preprocessing techniques to extract fNIRS features. Admittedly, the processed fNIRS data involved the systemic signal that might affect the results. Future work will explore advanced signal processing methods such as depth-resolved techniques and general linear model (GLM) with systemic regression techniques as suggested by Tachtsidis [103] and Katura [104] to separate the fNIRS signals. With a better definition of the source signals, we expect the detection rate of mental stress to be improved.
5. Conclusion
Simultaneous recording of EEG and fNIRS signals was exploited for mental stress assessment. We found that the right PFC region was sensitive to mental stress (induced by time- and peer-pressure) during arithmetic tasks. The results of joint EEG and fNIRS features using the proposed fusion approach demonstrated + 3.4% and 11.0% improvement in the classification accuracy of mental stress, as compared to unimodal EEG and fNIRS, respectively. This suggests that fusion of EEG and fNIRS signals using proposed jICA approach can help improve the diagnosis of mental stress.
Funding
This research is funded by the Ministry of Education, Malaysia under Higher Institution Centre of Excellence (HiCOE) scheme.
References and links
1. J. Decker, “The Stress Syndrome,” Am. J. Nurs. 65(3), 97–99 (1965). [PubMed]
2. L. R. Murphy, “Stress management in work settings: a critical review of the health effects,” Am. J. Health Promot. 11(2), 112–135 (1996). [CrossRef] [PubMed]
3. B. Czéh, T. Michaelis, T. Watanabe, J. Frahm, G. de Biurrun, M. van Kampen, A. Bartolomucci, and E. Fuchs, “Stress-induced changes in cerebral metabolites, hippocampal volume, and cell proliferation are prevented by antidepressant treatment with tianeptine,” Proc. Natl. Acad. Sci. U.S.A. 98(22), 12796–12801 (2001). [CrossRef] [PubMed]
4. C. M. Vander Weele, C. Saenz, J. Yao, S. S. Correia, and K. A. Goosens, “Restoration of hippocampal growth hormone reverses stress-induced hippocampal impairment,” Front. Behav. Neurosci. 7, 66 (2013). [CrossRef] [PubMed]
5. A. Vyas, R. Mitra, B. S. Shankaranarayana Rao, and S. Chattarji, “Chronic stress induces contrasting patterns of dendritic remodeling in hippocampal and amygdaloid neurons,” J. Neurosci. 22(15), 6810–6818 (2002). [PubMed]
6. B. S. McEwen, “Central effects of stress hormones in health and disease: Understanding the protective and damaging effects of stress and stress mediators,” Eur. J. Pharmacol. 583(2-3), 174–185 (2008). [CrossRef] [PubMed]
7. P. C. Strike and A. Steptoe, “Systematic review of mental stress-induced myocardial ischaemia,” Eur. Heart J. 24(8), 690–703 (2003). [CrossRef] [PubMed]
8. A. Tsutsumi, K. Kayaba, and S. Ishikawa, “Impact of occupational stress on stroke across occupational classes and genders,” Soc. Sci. Med. 72(10), 1652–1658 (2011). [CrossRef] [PubMed]
9. R. A. Ajjan and P. J. Grant, “Cardiovascular disease prevention in patients with type 2 diabetes: The role of oral anti-diabetic agents,” Diab. Vasc. Dis. Res. 3(3), 147–158 (2006). [CrossRef] [PubMed]
10. C. Hammen, “Stress and depression,” Annu. Rev. Clin. Psychol. 1(1), 293–319 (2005). [CrossRef] [PubMed]
11. S. Cohen, R. C. Kessler, and L. U. Gordon, Measuring Stress: A Guide for Health and Social Scientists (Oxford University Press on Demand, 1997).
12. T. H. Holmes and R. H. Rahe, “The social readjustment rating scale,” J. Psychosom. Res. 11(2), 213–218 (1967). [CrossRef] [PubMed]
13. T.-K. Liu, Y.-P. Chen, Z.-Y. Hou, C.-C. Wang, and J.-H. Chou, “Noninvasive evaluation of mental stress using by a refined rough set technique based on biomedical signals,” Artif. Intell. Med. 61(2), 97–103 (2014). [CrossRef] [PubMed]
14. D. H. Hellhammer, S. Wüst, and B. M. Kudielka, “Salivary cortisol as a biomarker in stress research,” Psychoneuroendocrinology 34(2), 163–171 (2009). [CrossRef] [PubMed]
15. M. Gröschl, M. Rauh, and H.-G. Dörr, “Circadian rhythm of salivary cortisol, 17α-hydroxyprogesterone, and progesterone in healthy children,” Clin. Chem. 49(10), 1688–1691 (2003). [CrossRef] [PubMed]
16. D. A. Granger, K. T. Kivlighan, M. el-Sheikh, E. B. Gordis, and L. R. Stroud, “Salivary α-amylase in biobehavioral research: recent developments and applications,” Ann. N. Y. Acad. Sci. 1098(1), 122–144 (2007). [CrossRef] [PubMed]
17. V. Engert, S. Vogel, S. I. Efanov, A. Duchesne, V. Corbo, N. Ali, and J. C. Pruessner, “Investigation into the cross-correlation of salivary cortisol and alpha-amylase responses to psychological stress,” Psychoneuroendocrinology 36(9), 1294–1302 (2011). [CrossRef] [PubMed]
18. R. R. Singh, S. Conjeti, and R. Banerjee, “A comparative evaluation of neural network classifiers for stress level analysis of automotive drivers using physiological signals,” Biomed Sig Proc. Cont 8(6), 740–754 (2013). [CrossRef]
19. J. A. Healey and R. W. Picard, “Detecting stress during real-world driving tasks using physiological sensors,” IEEE Trans. Intell. Transp. Syst. 6(2), 156–166 (2005). [CrossRef]
20. G. K. Verma and U. S. Tiwary, “Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals,” Neuroimage 102(Pt 1), 162–172 (2014). [CrossRef] [PubMed]
21. H. Ashton, R. D. Savage, J. W. Thompson, and D. W. Watson, “A method for measuring human behavioural and physiological responses at different stress levels in a driving simulator,” Br. J. Pharmacol. 45(3), 532–545 (1972). [CrossRef] [PubMed]
22. T. G. Vrijkotte, L. J. van Doornen, and E. J. de Geus, “Effects of work stress on ambulatory blood pressure, heart rate, and heart rate variability,” Hypertension 35(4), 880–886 (2000). [CrossRef] [PubMed]
23. L. Vézard, P. Legrand, M. Chavent, F. Faïta-Aïnseba, and L. Trujillo, “EEG classification for the detection of mental states,” Appl. Soft Comput. 32, 113–131 (2015). [CrossRef]
24. C. M. Michel and M. M. Murray, “Towards the utilization of EEG as a brain imaging tool,” Neuroimage 61(2), 371–385 (2012). [CrossRef] [PubMed]
25. M. Huiku, K. Uutela, M. van Gils, I. Korhonen, M. Kymäläinen, P. Meriläinen, M. Paloheimo, M. Rantanen, P. Takala, H. Viertiö-Oja, and A. Yli-Hankala, “Assessment of surgical stress during general anaesthesia,” Br. J. Anaesth. 98(4), 447–455 (2007). [CrossRef] [PubMed]
26. G. Chanel, J. Kronegg, D. Grandjean, and T. Pun, “Emotion assessment: Arousal evaluation using EEG’s and peripheral physiological signals,” Multimedia Content Representation, Classification and Security 4105, 530–537 (2006). [CrossRef]
27. T. Takahashi, T. Murata, T. Hamada, M. Omori, H. Kosaka, M. Kikuchi, H. Yoshida, and Y. Wada, “Changes in EEG and autonomic nervous activity during meditation and their association with personality traits,” Int. J. Psychophysiol. 55(2), 199–207 (2005). [CrossRef] [PubMed]
28. R. E. Wheeler, R. J. Davidson, and A. J. Tomarken, “Frontal brain asymmetry and emotional reactivity: A biological substrate of affective style,” Psychophysiology 30(1), 82–89 (1993). [CrossRef] [PubMed]
29. A. C. Marshall, N. R. Cooper, R. Segrave, and N. Geeraert, “The effects of long-term stress exposure on aging cognition: a behavioral and EEG investigation,” Neurobiol. Aging 36(6), 2136–2144 (2015). [CrossRef] [PubMed]
30. N. L. Lopez-Duran, R. Nusslock, C. George, and M. Kovacs, “Frontal EEG asymmetry moderates the effects of stressful life events on internalizing symptoms in children at familial risk for depression,” Psychophysiology 49(4), 510–521 (2012). [CrossRef] [PubMed]
31. Y. Choi, M. Kim, and C. Chun, “Measurement of occupants’ stress based on electroencephalograms (EEG) in twelve combined environments,” Build. Environ. 88, 65–72 (2015). [CrossRef]
32. P. Missonnier, F. R. Herrmann, C. Rodriguez, M.-P. Deiber, P. Millet, L. Fazio-costa, G. Gold, and P. Giannakopoulos, “Age-related differences on event-related potentials and brain rhythm oscillations during working memory activation,” J Neural Transm (Vienna) 118(6), 945–955 (2011). [CrossRef] [PubMed]
33. N. Sharma and T. Gedeon, “Objective measures, sensors and computational techniques for stress recognition and classification: A survey,” Comput. Methods Programs Biomed. 108(3), 1287–1301 (2012). [CrossRef] [PubMed]
34. A. Alberdi, A. Aztiria, and A. Basarab, “Towards an automatic early stress recognition system for office environments based on multimodal measurements: A review,” J. Biomed. Inform. 59, 49–75 (2016). [CrossRef] [PubMed]
35. L. Xin, C. Zetao, Z. Yunpeng, X. Jiali, W. Shuicai, and Z. Yanjun, “Stress State Evaluation by Improved Support Vector Machine,” J. Med Imag. Health Inform 5(4), 742–747 (2015). [CrossRef]
36. N. Sharma and T. Gedeon, “Modeling observer stress for typical real environments,” Expert Syst. Appl. 41(5), 2231–2238 (2014). [CrossRef]
37. G. Chanel, J. J. Kierkels, M. Soleymani, and T. Pun, “Short-term emotion assessment in a recall paradigm,” Int. J. Hum. Comput. Stud. 67(8), 607–627 (2009). [CrossRef]
38. C. Babiloni, V. Pizzella, C. D. Gratta, A. Ferretti, and G. L. Romani, “Fundamentals of electroencefalography, magnetoencefalography, and functional magnetic resonance imaging,” Int. Rev. Neurobiol. 86, 67–80 (2009). [CrossRef] [PubMed]
39. A. Villringer and B. Chance, “Non-invasive optical spectroscopy and imaging of human brain function,” Trends Neurosci. 20(10), 435–442 (1997). [CrossRef] [PubMed]
40. A. Sassaroli and S. Fantini, “Comment on the modified Beer-Lambert law for scattering media,” Phys. Med. Biol. 49(14), N255–N257 (2004). [CrossRef] [PubMed]
41. M. J. Herrmann, M. M. Plichta, A.-C. Ehlis, and A. J. Fallgatter, “Optical topography during a Go-NoGo task assessed with multi-channel near-infrared spectroscopy,” Behav. Brain Res. 160(1), 135–140 (2005). [CrossRef] [PubMed]
42. M. Boecker, M. M. Buecheler, M. L. Schroeter, and S. Gauggel, “Prefrontal brain activation during stop-signal response inhibition: an event-related functional near-infrared spectroscopy study,” Behav. Brain Res. 176(2), 259–266 (2007). [CrossRef] [PubMed]
43. W. C. Vogt, E. Romero, S. M. LaConte, and C. G. Rylander, “Mechanical indentation improves cerebral blood oxygenation signal quality of functional near-infrared spectroscopy (fNIRS) during breath holding,” in SPIE BiOS, 85782K–85787 (2013).
44. S. D. Power, A. Kushki, and T. Chau, “Automatic single-trial discrimination of mental arithmetic, mental singing and the no-control state from prefrontal activity: toward a three-state NIRS-BCI,” BMC Res. Notes 5(1), 141 (2012). [CrossRef] [PubMed]
45. S. Cutini, S. B. Moro, and S. Bisconti, “Functional near infrared optical imaging in cognitive neuroscience: an introductory,” J. Near Infrared Spectrosc. 20(1), 75–92 (2012). [CrossRef]
46. R. Sitaram, H. Zhang, C. Guan, M. Thulasidas, Y. Hoshi, A. Ishikawa, K. Shimizu, and N. Birbaumer, “Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain-computer interface,” Neuroimage 34(4), 1416–1427 (2007). [CrossRef] [PubMed]
47. H.-J. Hwang, J.-H. Lim, D.-W. Kim, and C.-H. Im, “Evaluation of various mental task combinations for near-infrared spectroscopy-based brain-computer interfaces,” J. Biomed. Opt. 19(7), 077005 (2014). [CrossRef] [PubMed]
48. G. Derosière, S. Dalhoumi, S. Perrey, G. Dray, and T. Ward, “Towards a near infrared spectroscopy-based estimation of operator attentional state,” PLoS One 9(3), e92045 (2014). [CrossRef] [PubMed]
49. H. Sato, R. Aoki, T. Katura, R. Matsuda, and H. Koizumi, “Correlation of within-individual fluctuation of depressed mood with prefrontal cortex activity during verbal working memory task: optical topography study,” J. Biomed. Opt. 16(12), 126007 (2011). [CrossRef] [PubMed]
50. R. Takizawa, M. Fukuda, S. Kawasaki, K. Kasai, M. Mimura, S. Pu, T. Noda, S. Niwa, Y. Okazaki, and Joint Project for Psychiatric Application of Near-Infrared Spectroscopy (JPSY-NIRS) Group, “Neuroimaging-aided differential diagnosis of the depressive state,” Neuroimage 85(Pt 1), 498–507 (2014). [CrossRef] [PubMed]
51. D. N. Lenkov, A. B. Volnova, A. R. Pope, and V. Tsytsarev, “Advantages and limitations of brain imaging methods in the research of absence epilepsy in humans and animal models,” J. Neurosci. Methods 212(2), 195–202 (2013). [CrossRef] [PubMed]
52. A. Machado, J.-M. Lina, J. Tremblay, M. Lassonde, D. K. Nguyen, F. Lesage, and C. Grova, “Detection of hemodynamic responses to epileptic activity using simultaneous Electro-EncephaloGraphy (EEG)/Near Infra Red Spectroscopy (NIRS) acquisitions,” Neuroimage 56(1), 114–125 (2011). [CrossRef] [PubMed]
53. V. D. Calhoun and T. Adal, “Feature-based fusion of medical imaging data,” IEEE Trans Info. Technol. Biomed. 13(5), 711–720 (2009).
54. V. D. Calhoun, T. Adali, N. R. Giuliani, J. J. Pekar, K. A. Kiehl, and G. D. Pearlson, “Method for multimodal analysis of independent source differences in schizophrenia: combining gray matter structural and auditory oddball functional data,” Hum. Brain Mapp. 27(1), 47–62 (2006). [CrossRef] [PubMed]
55. A. R. Groves, C. F. Beckmann, S. M. Smith, and M. W. Woolrich, “Linked independent component analysis for multimodal data fusion,” Neuroimage 54(3), 2198–2217 (2011). [CrossRef] [PubMed]
56. J. Sui, T. Adali, G. Pearlson, H. Yang, S. R. Sponheim, T. White, and V. D. Calhoun, “A CCA+ICA based model for multi-task brain imaging data fusion and its application to schizophrenia,” Neuroimage 51(1), 123–134 (2010). [CrossRef] [PubMed]
57. V. T. Nguyen, M. Breakspear, and R. Cunnington, “Fusing concurrent EEG-fMRI with dynamic causal modeling: application to effective connectivity during face perception,” Neuroimage 102(Pt 1), 60–70 (2014). [CrossRef] [PubMed]
58. Å. M. Hansen, A. H. Garde, and R. Persson, “Sources of biological and methodological variation in salivary cortisol and their impact on measurement among healthy adults: a review,” Scand. J. Clin. Lab. Invest. 68(6), 448–458 (2008). [CrossRef] [PubMed]
59. K. Dedovic, R. Renwick, N. K. Mahani, V. Engert, S. J. Lupien, and J. C. Pruessner, “The Montreal Imaging Stress Task: using functional imaging to investigate the effects of perceiving and processing psychosocial stress in the human brain,” J. Psychiatry Neurosci. 30(5), 319–325 (2005). [PubMed]
60. A. Delorme and S. Makeig, “EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis,” J. Neurosci. Methods 134(1), 9–21 (2004). [CrossRef] [PubMed]
61. S. Sutoko, H. Sato, A. Maki, M. Kiguchi, Y. Hirabayashi, H. Atsumori, A. Obata, T. Funane, and T. Katura, “Tutorial on platform for optical topography analysis tools,” Neurophotonics 3(1), 010801 (2016). [CrossRef] [PubMed]
62. T. J. Huppert, “Commentary on the statistical properties of noise and its implication on general linear models in functional near-infrared spectroscopy,” Neurophotonics 3(1), 010401 (2016). [CrossRef] [PubMed]
63. H. Atsumori, M. Kiguchi, T. Katura, T. Funane, A. Obata, H. Sato, T. Manaka, M. Iwamoto, A. Maki, H. Koizumi, and K. Kubota, “Noninvasive imaging of prefrontal activation during attention-demanding tasks performed while walking using a wearable optical topography system,” J. Biomed. Opt. 15(4), 046002 (2010). [CrossRef] [PubMed]
64. H. Sato, M. Kiguchi, F. Kawaguchi, and A. Maki, “Practicality of wavelength selection to improve signal-to-noise ratio in near-infrared spectroscopy,” Neuroimage 21(4), 1554–1562 (2004). [CrossRef] [PubMed]
65. X. Cui, S. Bray, and A. L. Reiss, “Functional near infrared spectroscopy (NIRS) signal improvement based on negative correlation between oxygenated and deoxygenated hemoglobin dynamics,” Neuroimage 49(4), 3039–3046 (2010). [CrossRef] [PubMed]
66. C. C. Chang and C. J. Lin, “LIBSVM: A Library for support vector machines,” ACM Trans. Intell. Syst. Technol. 2(3), 27 (2011). [CrossRef]
67. T. W. Lee, M. Girolami, and T. J. Sejnowski, “Independent component analysis using an extended infomax algorithm for mixed subgaussian and supergaussian sources,” Neural Comput. 11(2), 417–441 (1999). [CrossRef] [PubMed]
68. A. J. Bell and T. J. Sejnowski, “An information-maximization approach to blind separation and blind deconvolution,” Neural Comput. 7(6), 1129–1159 (1995). [CrossRef] [PubMed]
69. Z. Yuan and J. Ye, “Fusion of fNIRS and fMRI data: Identifying when and where hemodynamic signals are changing in human brains,” Front. Hum. Neurosci. 7, 676 (2013). [PubMed]
70. V. D. Calhoun, J. Liu, and T. Adali, “A review of group ICA for fMRI data and ICA for joint inference of imaging, genetic, and ERP data,” Neuroimage 45(1Suppl), S163–S172 (2009). [CrossRef] [PubMed]
71. M. Moosmann, T. Eichele, H. Nordby, K. Hugdahl, and V. D. Calhoun, “Joint independent component analysis for simultaneous EEG-fMRI: principle and simulation,” Int. J. Psychophysiol. 67(3), 212–221 (2008). [CrossRef] [PubMed]
72. V. D. Calhoun, T. Adali, G. D. Pearlson, and K. A. Kiehl, “Neuronal chronometry of target detection: fusion of hemodynamic and event-related potential data,” Neuroimage 30(2), 544–553 (2006). [CrossRef] [PubMed]
73. R. Thibodeau, R. S. Jorgensen, and S. Kim, “Depression, anxiety, and resting frontal EEG asymmetry: a meta-analytic review,” J. Abnorm. Psychol. 115(4), 715–729 (2006). [CrossRef] [PubMed]
74. R. S. Lewis, N. Y. Weekes, and T. H. Wang, “The effect of a naturalistic stressor on frontal EEG asymmetry, stress, and health,” Biol. Psychol. 75(3), 239–247 (2007). [CrossRef] [PubMed]
75. M. Gärtner, S. Grimm, and M. Bajbouj, “Frontal midline theta oscillations during mental arithmetic: effects of stress,” Front. Behav. Neurosci. 9, 96 (2015). [PubMed]
76. C. Zhao, M. Zhao, J. Liu, and C. Zheng, “Electroencephalogram and electrocardiograph assessment of mental fatigue in a driving simulator,” Accid. Anal. Prev. 45, 83–90 (2012). [CrossRef] [PubMed]
77. C. Zhao, C. Zheng, M. Zhao, Y. Tu, and J. Liu, “Multivariate autoregressive models and kernel learning algorithms for classifying driving mental fatigue based on electroencephalographic,” Expert Syst. Appl. 38(3), 1859–1865 (2011). [CrossRef]
78. K. Mizuno, M. Tanaka, K. Yamaguti, O. Kajimoto, H. Kuratsune, and Y. Watanabe, “Mental fatigue caused by prolonged cognitive load associated with sympathetic hyperactivity,” Behav. Brain Funct. 7(1), 17 (2011). [CrossRef] [PubMed]
79. R. Sinha, C. Lacadie, P. Skudlarski, R. K. Fulbright, B. J. Rounsaville, T. R. Kosten, and B. E. Wexler, “Neural activity associated with stress-induced cocaine craving: a functional magnetic resonance imaging study,” Psychopharmacology (Berl.) 183(2), 171–180 (2005). [CrossRef] [PubMed]
80. S. Qin, E. J. Hermans, H. J. van Marle, J. Luo, and G. Fernández, “Acute psychological stress reduces working memory-related activity in the dorsolateral prefrontal cortex,” Biol. Psychiatry 66(1), 25–32 (2009). [CrossRef] [PubMed]
81. L. Ossewaarde, S. Qin, H. J. Van Marle, G. A. van Wingen, G. Fernández, and E. J. Hermans, “Stress-induced reduction in reward-related prefrontal cortex function,” Neuroimage 55(1), 345–352 (2011). [CrossRef] [PubMed]
82. K. Dedovic, M. Rexroth, E. Wolff, A. Duchesne, C. Scherling, T. Beaudry, S. D. Lue, C. Lord, V. Engert, and J. C. Pruessner, “Neural correlates of processing stressful information: an event-related fMRI study,” Brain Res. 1293, 49–60 (2009). [CrossRef] [PubMed]
83. C. Liston, B. S. McEwen, and B. J. Casey, “Psychosocial stress reversibly disrupts prefrontal processing and attentional control,” Proc. Natl. Acad. Sci. U.S.A. 106(3), 912–917 (2009). [CrossRef] [PubMed]
84. M. Hinwood, J. Morandini, T. A. Day, and F. R. Walker, “Evidence that microglia mediate the neurobiological effects of chronic psychological stress on the medial prefrontal cortex,” Cereb. Cortex 22(6), 1442–1454 (2012). [CrossRef] [PubMed]
85. T. Li, Q. Luo, and H. Gong, “Gender-specific hemodynamics in prefrontal cortex during a verbal working memory task by near-infrared spectroscopy,” Behav. Brain Res. 209(1), 148–153 (2010). [CrossRef] [PubMed]
86. Z. F. Zaidi, “Gender differences in human brain: a review,” The Open Anatomy Journal 2(1), 1 (2010).
87. N. R. Lighthall, M. Sakaki, S. Vasunilashorn, L. Nga, S. Somayajula, E. Y. Chen, N. Samii, and M. Mather, “Gender differences in reward-related decision processing under stress,” Soc. Cogn. Affect. Neurosci. 7(4), 476–484 (2012). [CrossRef] [PubMed]
88. S. D. Power, T. H. Falk, and T. Chau, “Classification of prefrontal activity due to mental arithmetic and music imagery using hidden Markov models and frequency domain near-infrared spectroscopy,” J. Neural Eng. 7(2), 026002 (2010). [CrossRef] [PubMed]
89. L. C. Schudlo and T. Chau, “Dynamic topographical pattern classification of multichannel prefrontal NIRS signals: II. Online differentiation of mental arithmetic and rest,” J. Neural Eng. 11(1), 016003 (2014). [CrossRef] [PubMed]
90. M. Stangl, G. Bauernfeind, J. Kurzmann, R. Scherer, and C. Neuper, “A haemodynamic brain-computer interface based on real-time classification of near infrared spectroscopy signals during motor imagery and mental arithmetic,” J. Near Infra Spec 21(3), 157–171 (2013). [CrossRef]
91. G. Bauernfeind, R. Scherer, G. Pfurtscheller, and C. Neuper, “Single-trial classification of antagonistic oxyhemoglobin responses during mental arithmetic,” Med. Biol. Eng. Comput. 49(9), 979–984 (2011). [CrossRef] [PubMed]
92. S. D. Power, A. Kushki, and T. Chau, “Intersession consistency of single-trial classification of the prefrontal response to mental arithmetic and the no-control state by NIRS,” PLoS One 7(7), e37791 (2012). [CrossRef] [PubMed]
93. S. D. Power and T. Chau, “Automatic single-trial classification of prefrontal hemodynamic activity in an individual with Duchenne muscular dystrophy,” Dev. Neurorehabil. 16(1), 67–72 (2013). [CrossRef] [PubMed]
94. N. Naseer, M. J. Hong, and K.-S. Hong, “Online binary decision decoding using functional near-infrared spectroscopy for the development of brain-computer interface,” Exp. Brain Res. 232(2), 555–564 (2014). [CrossRef] [PubMed]
95. S. D. Power, A. Kushki, and T. Chau, “Towards a system-paced near-infrared spectroscopy brain-computer interface: differentiating prefrontal activity due to mental arithmetic and mental singing from the no-control state,” J. Neural Eng. 8(6), 066004 (2011). [CrossRef] [PubMed]
96. M. J. Khan, M. J. Hong, and K.-S. Hong, “Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface,” Front. Hum. Neurosci. 8, 244 (2014). [CrossRef] [PubMed]
97. K.-S. Hong, N. Naseer, and Y.-H. Kim, “Classification of prefrontal and motor cortex signals for three-class fNIRS-BCI,” Neurosci. Lett. 587(1), 87–92 (2015). [CrossRef] [PubMed]
98. B. Abibullaev and J. An, “Classification of frontal cortex haemodynamic responses during cognitive tasks using wavelet transforms and machine learning algorithms,” Med. Eng. Phys. 34(10), 1394–1410 (2012). [CrossRef] [PubMed]
99. S. Fazli, J. Mehnert, J. Steinbrink, G. Curio, A. Villringer, K.-R. Müller, and B. Blankertz, “Enhanced performance by a hybrid NIRS-EEG brain computer interface,” Neuroimage 59(1), 519–529 (2012). [CrossRef] [PubMed]
100. F. Putze, S. Hesslinger, C. Y. Tse, Y. Huang, C. Herff, C. Guan, and T. Schultz, “Hybrid fNIRS-EEG based classification of auditory and visual perception processes,” Front. Neurosci. 8, 373 (2014). [CrossRef] [PubMed]
101. Y. Blokland, L. Spyrou, D. Thijssen, T. Eijsvogels, W. Colier, M. Floor-Westerdijk, R. Vlek, J. Bruhn, and J. Farquhar, “Combined EEG-fNIRS decoding of motor attempt and imagery for brain switch control: an offline study in patients with tetraplegia,” IEEE Trans. Neural Syst. Rehabil. Eng. 22(2), 222–229 (2014). [CrossRef] [PubMed]
102. X. Yin, B. Xu, C. Jiang, Y. Fu, Z. Wang, H. Li, and G. Shi, “A hybrid BCI based on EEG and fNIRS signals improves the performance of decoding motor imagery of both force and speed of hand clenching,” J. Neural Eng. 12(3), 036004 (2015). [CrossRef] [PubMed]
103. I. Tachtsidis and F. Scholkmann, “False positives and false negatives in functional near-infrared spectroscopy: issues, challenges, and the way forward,” Neurophotonics 3(3), 031405 (2016). [CrossRef] [PubMed]
104. T. Katura, H. Sato, Y. Fuchino, T. Yoshida, H. Atsumori, M. Kiguchi, A. Maki, M. Abe, and N. Tanaka, “Extracting task-related activation components from optical topography measurement using independent components analysis,” J. Biomed. Opt. 13(5), 054008 (2008). [CrossRef] [PubMed]
