A comprehensive study of EEG-based control of artificial arms

Комплексное исследование об управлении искусственной рукой с помощью электроэнцефалографии (ЭЭГ)

Опсежна студија о управљању вештачком руком помоћу електроенцефалографије (ЕЕГ)

Ihab Abdulrahman Satama ihab.satam@uni-obuda.hu

Óbuda University, Hungary

This work is licensed under Creative Commons Attribution 4.0 International.

The absence of the upper limb results in severe impairment in everyday life, which can further influence both the social and mental state (Abdulrahman Satam, 2021). For these reasons, developments in cosmetic and body-driven prostheses date from some centuries ago, and they have been evolving ever since. Research showed that the estimated percentage of impaired people is rising up due to wars, conflicts, diseases, accidents, and forgotten minefields from previous battles and wars.

A prosthesis is much more than a device; it also completes a wearer`s sense of wholeness. It gives emotional comfort. The history of prosthetics isnot just about the advancement of medical science, it is a history of human beings who miss an essential part of themselves. The earliest known prosthetic wasnot an eye, leg, or arm. It was a toe, first made by Egyptians around 3000 years ago. Then development continued with the Roman Empire to the end of the Middle Ages and finally to the civil war in the United States of America.

A decade ago, prosthetic limbs were developed as a practical complementary system for impaired people. Prosthetics, or artificial limbs, are used to replace limbs that were lost or absent limbs from birth. They enable those with congenital limb differences and amputees alike to improve function and mobility. Due to advances in medical science, prosthetics have improved and are capable of remarkable things (Osama & Allauddin, 2022).

In addition to the development of the prosthetic arm (Figure 1) design, scientists are focusing on improving the control of the techniques for the purpose of accuracy, performance enhancement, and the comfort of the prosthesis.

Figure 1
Prosthetic arm

Рис. 1 – Протез руки

Слика 1 – Простетичка рука

Ensuring the smoothness and effective control techniques of the prosthetic limb is an important factor in the interface between the wearable prosthesis and the human since the prosthetic limb is donned by a human.

Therefore, those control strategies can be classified according to the human-robot interaction method. The control of the prosthetic arm is influenced by electrophysiological signals. These signals have been well-known tools to examine the capacity and conduct of the human movement in ongoing research.

Electroencephalography (EEG) has been one of frequently used physiological signals in the control techniques of prosthetic limbs, especially in the upper limbs. EEG is considered a non-invasive and convenient method that may be appropriate for realistic application. Recently, it was found that fewer endeavors have been made to efficiently audit these reviews, as a way of offering analysts and specialists a synopsis of the current, best-in-class EEG-based control systems utilized for assistive innovation. Hence, this research has three primary objectives.

The primary aim is to deliberately assemble, abridge, assess, and organize data with respect to accuracy and estimations of the past research distributed in the publications between 2011 and 2018.

The second objective is to broadly report on all the trial results of this domain’s present research. It is methodically performed to give a clear picture and grounded proof of the momentum conditions of research covering EEG-based control uses and benefits for controlling assistive robotics to every specialist and researcher. The third objective is to perceive the whole of information that requests in-depth examination and to suggest ways for future research in this domain (Mandekar et al, 2022). To achieve these objectives, the following research questions (RQs) have been put forward:

(Q1) What are the types of EEG signals that are used to control the prosthetic arm?

(Q2) How do these signals translate to control commands?

The solutions to these questions will guide the reader and enhance their knowledge of the recent development of prosthetic arms based on EEG signals. A more extensive image of various emergent topics/themes, experiments, and concepts will be offered. This paper is structured into six sections.

The following section provides a background of EEG signals and prosthetic limbs. The third section describes the methodology through which the review processes were conducted. The fourth section presents the SLR results, followed by the fifth section which reports on the results of the research questions as organized according to their sequences. Finally, the sixth section presents a discussion of the review and its conclusion.

Table 1
Symbols

Таблица 1 – Обозначения

Табела 1 – Симболи

In the BCI system, EEG signals are most commonly used not only for prosthetics but also for any controllable devices such as robotics arms, Exoskeletons, Wheelchairs, drones, etc. Bridges et al. (Bridges et al, 2011) and his team provide an overviewof human-machine interface architecture. The article contains good information about the control system. Yanagisawa et al. (Yanagisawa et al, 2011) shows a new method of controlling a prosthetic arm using ECoG signals. The system proved its effectiveness in decoding the hand movement of a patient who suffered from a stroke and used that signal to control a prosthetic hand. Another research implemented by Taha and his team (Beyrouthy et al, 2017) is about a system that extracts the EEG signals from the brain and uses them to control a smart 3D prosthetic arm. The system showed great results and presented a reliable alternative for an invasive system. Researchers in (Bright et al, 2016) succeeded in developing an EEG-based brain control system for the prosthetic arm using a BCI Neurosky mind wave set. The system reached an accuracy of 80 %. The team of researchers in (Elstob& Secco, 2016) controlled a 5 DOF robotic and prosthetic hand. They used two software frameworks. The method showed good results both technically and economically. A study of experimenting how transradial amputees could control grasp preshaping in a prosthetic arm using an EEG-based closed Loop BMI system is done by Agashe et al. (Agashe et al, 2016). The results showed that the EEG-based BMI system is a feasible solution. Healthy participants involved in a study implemented by Vidaurre et al. (Vidaurre et al, 2016) were able to use non-invasive Motor Imagery BCI to achieve linear control of an Upper Limb FES controlled Neuro Prosthesis. An embedded system was designed by (Rashid et al, 2018) in order to control the finger movement of the prosthetic arm using EEG signals. The signal classification accuracy of this study reached an acceptable percent of 79 %. Faiman et al. (Faiman et al, 2018) investigated whether spontaneous resting-state functional connectivity could predict the degree of motor adaptation of the right (dominant) upper limb reaching in response to a robot-mediated force field. Spontaneous neural activity was measured using resting-state electroencephalography (EEG) in healthy adults before a single session of motor adaptation. Noel & Snider (Noel & Snider, 2019) used the Deep Neural Network to control the prosthetic arm. The Neural Network was used to classify the signal to detect person's intention of extending the right index finger. The model achieved an accuracy of 63.3%. Gannouni et al. (Gannouni et al, 2020) presented a study that uses machine learning in order to anticipate the movement of all five fingers. The proposed system achieved a signal classification accuracy of 81%. A 62% accuracy was achieved for an inexpensive mind-controlled prosthetic arm based on EEG signals. The system was implemented by (Chinta et al, 2020). Fuentes-Gonzalez et al. (Fuentes-Gonzalez et al, 2021) designed a prosthetic arm using blender software. The control of the prosthetic arm was done using EEG signals. The prosthetic arm was fitted to a 64-year-old man who had suffered from an electric shock. Ali et al. (Ali et al, 2021) build an inexpensive smart functional prosthesis arm in accordance with functional and non-functional requirements to meet users’ goals and requirements. Setiawan et al. (Setiawan et al, 2021) designed a system to control a prosthetic hand using EEG signals to execute flexion and extension of fingers. Chaudhry et all (Chaudhry et al, 2022) discussed EEG control algorithms for prosthetic arms. They developed a cheap three-dimensional prosthetic arm;however, it was only a prototype and couldnot be applied for amputees. An EEG-based control system is not restricted to prosthetic arms only sinceexoskeleton and robotic arms can also be included in that area. Xu, et al. (Xu et al, 2011) developed a rehabilitation system for an upper limb stroke patient where the assistive device was based on motor imaginary EEG. The system proved feasibile and is fully capable of exploring patientʹs motor initiatives and guiding stroke patients to perform rehabilitation training effectively. The teams in (Ramos-Murguialday et al, 2012) developed a robotic hand exoskeleton based BCI to move fingers in flexion and extension movements. The results suggest that feedback contingency (proprioceptive stimulation paired with EEG SMR desynchronization) influences the motor network enhancing significantly SMR down-regulation. Formaggio et al. (Formaggio et al, 2013) present a study to perform a robot assisted task using a Bi-Manu track robot assisted arm trainer. Eight subjects participated in the study. The results suggest new perspectives for the assessment of patients with neurological disease. Tung et al. (Tung et al, 2013) performed a study of EEG to track the effect of a BCI based therapy on brain plasticity. The results suggest that motor recovery improvement comes from increasing activation in the lesion hemisphere during the BCI therapy. Krichner et al. (Krichner et al, 2014)carried out an experiment to prove that EEG and EMG can improve the adaptability of assistive devices in accordance with demands of users. The results show that both EEG and EMG predict a movement before it is physically executed. Witkowski et al. (Witkowski et al, 2014) introduced and tested a novel hybrid brain-neural computer interaction (BNCI) system fusing electroencephalography (EEG) and electrooculography (EOG) to enhance reliability and safety of continuous hand exoskeleton-driven grasping motions. Looned et al. (Looned et al, 2014) introduceda wearable and portable system consisting of a novel lightweight Robotic Arm Orthosis (RAO), a Functional Electrical Stimulation (FES) system, and a simple wireless Brain-Computer Interface (BCI). This system is able to process electroencephalographic (EEG) signals and translate them into motions of the impaired arm. The researchers in (Hortal et al, 2015) created a system based on a hybrid upper limb exoskeleton for neurological rehabilitation. The movement was controlled by an EEG-based BMI. The system showed the combined use of a hybrid upper limb exoskeleton. Brauchle and his team in (Brauchle et al, 2015) tested the feasibility of a 3D robotic assistant to produce movements with a multi-joint exoskeleton during MI synchronization of sensorimotor oscillations in the B-band. The team of researchers in (Elnady et al, 2015) tested the feasibility of using FES. Robotic training devices facilitate motor task completion in post-stroke individuals. A robotic training device was operated to assist a pre-defined goal-directed motor task. The results showed that the participants' ability to use proprioception to control a motor output did not affect their ability to use the BCI-driven exoskeleton with FES. A novel system for the neuro-motor rehabilitation of upper limbs was presented in (Comani et al, 2015). The system was validated in three sub-acute post-stroke patients. The system permits synchronized cortical and kinematic measures by integrating high-resolution EEG, a passive robotic device and Virtual Reality. The brain functional re-organization was monitored in association with motor patterns replicating activities of daily living (ADL). The patients underwent 13 rehabilitation sessions. Soekadar et al. (Soekadar et al, 2015) introduced a novel brain/neural-computer interaction (BNCI) system that integrates electroencephalography (EEG) and electrooculography (EOG) to improve control of assistive robotics in daily life environments. In (Bhagat et al, 2016), researchers demonstrated the feasibility of detecting a motor intent from the brain activity ofchronic stroke patients using an asynchronous electroencephalography (EEG)-based brain machine interface (BMI). Another investigation was implemented in (Tang et al, 2016). They investigated whether self-induced variations of the electroencephalogram (EEG) can be useful as control signals for a man-made upper-limb exoskeleton. A BMI based on event-related desynchronization/synchronization (ERD/ERS) is proposed. The study showed that the system is effective to control the upper limb exoskeleton. A rehabilitation approach based on BCI providing contingent sensory feedback of brain activity was presented by Frolov et al. (Frolov et al, 2017). The results proved that adding BCI control to exoskeleton assistive devices can improve the rehabilitation process for post stroke patients. The researchers in (Buerkle et al, 2021) presented a novel approach of how upper-limb movement intentions can be measured with a mobile electroencephalogram (EEG). The results suggested high detection accuracies and potential time gains of up to 513 ms to be achieved in a semi-online system. Thus, the time advantages included in a simulation demonstrated the potential to increase a system’s reaction time and therefore improve the safety and the fluency of Human-Robot Collaboration. The EEG based control systems have application in the field of robotic arms. Steinisch et al. (Steinisch et al, 2013) proposed a system for neuro-motor rehabilitation of the upper limbs in stroke survivors. The system is composed of a passive robotic device (Trackhold) for kinematic tracking and gravity compensation, five dedicated virtual reality (VR) applications for training of distinct movement patterns, and high-resolution EEG for synchronous monitoring of cortical activity. Another study was conducted by Shedeed et al. (Shedeed et al, 2013). They presented a BMI system based on EEG signals to control three movements (open arm, close arm, and closehand). The signal classification accuracy reached up 91%. The researchers in (Bhattacharyya et al, 2014) proposed a novel approach toward EEG-driven position control of a robot arm by utilizing motor imagery. The results showed that the system is effective in the rehabilitation process. The team in (Xu et al, 2015) designed a BCI-based online robot control system. The study included 30 participants. The system proved its effectiveness and reliability. The total accuracy of the system reached up to 91 %. Meng et al. (Meng et al, 2016) designed a system to control a robotic arm to perform reach and grasp based on non-invasive BCI technology. Thirteen participants were included in this research. The system showed that the subjects can control the arm through modulation of their brain with the training. Karakoe et al. (Karakoc et al, 2017) designed a robotic arm using solidwork software. The arm can be controlled using brainwaves. The study was successful –however, although the arm was successfully controlled, it was not applicable (only a prototype). Bousseta et al. (Bousseta et al, 2018) proposed a novel BCI system that consists of controlling a robot arm based on the user’s thoughts. Four subjects (1 female and 3 males) aged between 20 and 29 participated in the experiment. They were instructed to imagine the execution of movements of the right hand, the left hand, both right and left hands or the movement of the feet depending on the protocol established. A dynamical system conceptual and preliminary design together with system modeling are introduced in (Szabolcsi, 2019). Both dynamical system design and analysis tasks based on classical and modern control engineering approaches are handled in (Szabolcsi, 2020) using MATLAB.

An extensive literature search was carried out. The search covered studies between 2011 and 2018. Only full-text papers published in English were considered. In this research, the combination of keywords (BCI or Brain-Computer Interface or EEG or Electroencephalography) and (Prosthetic Limb, Prosthetic Arms or Robot) and ( Control Method ) is used. Figure 2 shows the process of how the chosen papers were selected in this research.

Figure 2
Research method

Рис. 2 – Метод исследования

Слика 2 – Метод истраживања

From the figure above, the number of selected papers was decreased due to the application of several filters depending on the type of the papers,e.g. full text or not.Only English language papers were chosen, also depending on the type of input signals, i.e. EEG signals. Only the articles that dealt with an upper limb (Arm, Hand) were included.

Figure 3 shows the distributions of the articles regarding the EEG-Based Control method for artificial upper limbs (as prosthetic arms or assistive devices or robotic arms). Years from 2013 till mid of 2016 witnessed a rise in research interest in this area.

Figure 3
Publication distribution

Рис. 3 – Распространение публикаций

Слика 3 – Расподела публикација

Background

Prosthetic limbs

In this section, the focus of the prosthetic limb will be on the prosthetic arms type. The prosthetic arm consists of several components that work together to make the arm useful.

· Limb. The limbs of a prosthetic arm are formed out of lightweight, yet durable materials.

· Socket. The socket connects the prosthesis to the residual limb to ensure that it fits securely. A poor fit can cause considerable discomfort and reduce the function of the prosthetic arm. To circumvent this problem, prosthetics are made using a personalized mold to fit the exact shape of the residual limb.

· Suspension system. The suspension system is the component that secures the prosthetic to the residual limb. There are different suspension systems, including a harness, an elastic sleeve, a suction socket, or a self-suspending socket.

· Control system. While the brain controls a natural limb and nerve impulses, a prosthetic arm cannot be controlled the same way. Control systems are myoelectric, body-powered, or motor-controlled.

Electroencephalography (EEG)

Electroencephalography (EEG) is the most common brain signal that has been utilized in brain-machine interface applications. This popularity is due to several facts: EEG signals are non-invasive, low cost, compatible, portable and have a high temporal resolution in comparison with other brainwave measurements such as electrocorticograms (ECoGs), magnetoencephalograms (MEGs), functional magnetic resonance imaging (fMRI) and near-infrared spectroscopy (fNIRS).

Electroencephalography can be defined as the measurement of the electric brain activity caused by currents induced by neurons within the brain (Murphy et al, 2017). The EEG signal can be detected in a non-invasive way by placing the electrode on the scalp. This justifies why the EEG measurement is the most widespread brain activity measurement technique. In addition, it is comparatively affordable and provides a high temporal resolution (about 1 ms). However, it has a weak signal and is prone to several artifacts and relatively poor spatial resolution.

In EEG measurement, detected waveforms reveal cortical electrical activity. The signal intensity of EEG activity is often quite small and measured in the microvolt (μV) range (Übeyli, 2009; Acharya et al, 2019). The main EEG rhythms are classified based on the frequency range as alpha (α), beta (β), delta (δ), theta (θ) and gamma as shown in Table 1a.

Table 1a
EEG frequencies

Таблица 1a – Частоты ЭЭГ

Tабела 1a – ЕЕГ фреквенције

During the systematic review, 39 articles were chosen for principal studies and all of them were using EEG as an input signal.

However, the output was either a prosthetic limb, an exoskeleton device, or a robotic arm.

Figure 4 shows the number of studies that dealt with each one considered in this paper.

Figure 4
Articles dealing with EEG application

Рис. 4 – Статьи о применении ЭЭГ

Слика 4 – Чланци о примени ЕЕГ апликације

The systematic review results in 39 papers, chosen as principal studies and published in the field of EEG-based control of prosthetic arms, exoskeleton, and robotic arms, are shown in Tables 2a, 2b, 3a, 3b and 4, respectively.

Table 2a
Prosthetic arm

Таблица 2a – Протез руки

Табела 2a –Простетичка рука

Table 2b
Prosthetic arm

Таблица 2б – Протез руки

Табела 2б –Простетичка рука

Table 3а
Exoskeleton

Таблица 3а –Экзоскелет

Табела 3а –Егзоскелет

Table 3b
Exoskeleton

Таблица 3б – Экзоскелет

Табела 3б –Егзоскелет

Table 4
Robotic arm

Таблица 4 – Роботизированная (бионическая) рука

Табела 4 – Роботичка рука

Figure 5
EEG signal control in the prosthetic arm research studiesfrom Table 2ab

Рис. 5 – Управление сигналом ЭЭГ в исследованииях о протезировании руки из Таблицы 2аб

Слика 5 – Управљање ЕЕГ сигналом у истраживачким студијама које се баве простетичком руком (табела 2аб)

Figure 6
EEG signal control in the exoskeleton research studiesfrom Table 3ab

Рис. 6 – Управление сигналом ЭЭГ в исследованиях экзоскелета из Таблицы 3аб

Слика 6 – Управљање ЕЕГ сигналом у истраживачким студијама које се баве егзоскелетом (табела 3аб)

Figure 7
EEG signal control in the robotic arm research studiesfrom Table 4

Рис. 7 – Управление сигналом ЭЭГ в исследованиях роботизированной руки из Таблицы 4

Слика 7 –Управљање ЕЕГ сигналом у истраживачким студијама које се баве роботичком руком (табела 4)

EEG signal types

EEG is nowadays considered a successful non-invasive realistic and practical Brain-Machine Interface BMI Technique. This is due to the fact that other techniques are considered high cost, e.g. magnetoencephalography (MEG) and positron emission tomography (PET).

Three key elements characterise the EEG-based prosthetic arm: the type of EEG signals, which part of the prosthetic arm is under control, and how to translate the EEG signal to a control command to manage the prosthesis. Figure 8 shows these key elements.

Figure 8
Research elements

Рис. 8 – Элементы исследования

Слика 8 – Елементи истраживања

Endogenous and exogenous EEG signals

Depending on the movementtype, the prosthetic arm can be managed by utilizing exogenous or endogenous EEG signals.

Table 5 shows the differences between these two types.

Table 5
EEG signal types

Таблица 5 – Типы сигналов ЭЭГ

Табела 5 – Типови ЕЕГ сигнала

Translating the EEG signal to the control command

EEG signal decoding has several stages in order to reach the desired output. These common stages are preprocessing, feature extraction, and classification. At each stage, an algorithm is applied.

А - Preprocessing

EEG records electrical potentials generated by nerve cells. Electrodes are placed on the scalp and recorded by amplification. By this procedure, the obtained data shows a continuous graphic with the spatial distribution of the voltage changes over time. In order to translate brain activity into commands, there are three steps to be applied. First, the brain activity is recorded with an acquisition device. Then, artifacts are unwanted faulty parts of signals that are removed from signals (Gupta & Singh,1996).

B - Feature extraction

Relevant features are extracted by methods such as Fast Fourier Transform (FFT), Wavelet Transform (WT), and Eigenvectors (Zhang et al, 2008). There are different methods for feature extraction of the signal such as FFT, WT, Eigenvectors, Time-frequency Distributions, and Autoregressive Method. Table 6 shows every method mentioned earlier with their advantages and disadvantages.

Table 6
Feature extraction techniques

Таблица 6 – Методы выявления признаков

Табела 6 – Технике издвајања карактеристика

C - Classification

A classifier utilizes values for independent variables (features) as inputs to predict the corresponding class to which an independent variable belongs. A classifier has a number of parameters that require training from a training dataset (Wen et al, 2021). A trained classifier will model the association between classes and corresponding features and is capable of identifying new instances in an unseen testing dataset. Several techniques of classification are explained in Table 7.

Table 7
Classification techniques

Таблица 7 – Методы классификации

Табела 7 – Технике класификације

The study of EEG-based prosthetic arms includes a wide range of fields to be familiar with, such as anatomy, signal processing, control methods, and design. All these fields represent sciences in themselves. Although researchers have done good work regarding prosthetic arm control, more needs to be done, such as increasing the accuracy in EEG signal extracting and faster control.

The use of automatic prosthetic arms is already an accepted method. It is considered one of the methods that aim to be applicable for lower limbs, wheelchairs, cars, and even drones.

In future researchbased ona Ph.D. thesis,the five extraction methods mentioned in Table 6 and the six classification methods mentioned in Table 7 will be used to control the full motion of the arm moving an object from one place to another using LabView or MATLAB simulation.Subsequently, a method with high accuracy will be applied for an actual prosthetic arm using Raspberry pi 4 controller.

FIELD: mechanics, electronics

ARTICLE TYPE: original scientific paper

https://scindeks.ceon.rs/article.aspx?artid=0042-84692301009S (html)

https://aseestant.ceon.rs/index.php/vtg/article/view/41366 (pdf)

https://doaj.org/article/644b5cae3a3f4b4e897b886ef94e0b74 (pdf)

https://www.elibrary.ru/item.asp?id=50193869 (pdf)

http://www.vtg.mod.gov.rs/archive/2023/military-technical-courier-1-2023.pdf (pdf)