PRE2017 3 Groep7

Introduction

For the course Robots Everywhere, the goal is to do a project about robots. The groups can choose any topic, as long as it contains robots and can be finished in 7 weeks. The course started by brainstorming on this topic, after which Group 7 decided to look at robot prosthetics. Since this is quite a broad topic, it was necessary to narrow it down. The group wanted to do something challenging and new. Therefore, it was decided to look at wrists, because there are no robotic wrist prosthetics on the market yet. Since the fine motor skills come from wirsts and fingers, this could be a valuable addition to current prosthetics. The idea was to control these prosthetics directly with the brain, so paralysed people can use it as well. To turn the EEG-data of the brain into a movement, a machine learning algorithm was used.

On this wiki, the whole process of the project can be found. First, the problem statement and all objectives will be discussed, then the USE-aspects, deliverables and approach of the project, including the planning. After that, there is a literature study, to know what is already on the market. Finally, it will be discussed how the group worked towards a result and what this result looked like. This final part will contain three chapters: the EEG-data, the machine learning algorithm and the functional model.

Problem Statement and Objectives

In the U.S., it is estimated that one in every 190 people have suffered from limb loss. This shows the importance and opportunity of prosthetics. For all these people, part of their old functions can be regained with the use of a prosthetic limb. While technology evolves, the prosthetics are able to perform more functions and are easier to handle. Where old prosthetics are mostly body-powered, they are now myoelectric or even robotic. These robotic prosthetics should eventually be able to mimic all old functions of the lost limb.

One of the challenges in the field is the hand. Especially the wrist is difficult. This joint creates a lot of movement, working with the forearm. To make a joint with as much movement as the wrist and the same power, has proven to be a difficult task. How can one design a full flexible wrist, while also giving it enough strength to lift objects? That will be one of the main questions during this project.

Another big challenge is how to control the prosthetic. Some of the newer, robotic limbs can actually work with the nerve system or the brain. This still proves difficult, since the user will need to learn to use the prosthetic in a natural way and some limbs, like the arm, have many different degrees of freedom to take into account. However, machine learning might prove a way for the prosthetic and the user to meet halfway: the user has to adjust to the prosthetic, and the prosthetic will learn how the user behaves. How do these algorithms work and what could it mean for the industry? By making our own algorithm, we will try to find out whether this might be a breakthrough for prosthetic use.

USE

It is important to look at the user aspects of wrist prostheses. For this there will be looked at the end users, society and enterprise.

Users

The users of a hand prothesis will be people who have somehow lost their hand or arm entirely. Their current way of living would be improved by giving them the possibility to use their prosthesis to pick up objects, which would also help them with basic uses of the hand. Further development could lead to even more possibilities such as catching objects or typing. This would greatly improve the quality of life for these people.

The preferences of the users would of course be a full-fuctioning arm and hand with perhaps even additional functions. The question is whether or not this is desirable, since if a prosthetic hand is more useful than a human one it could incentivise people to have a prostetic hand even if their current hand is still functioning.

Society

A hand prothesis will allow people who lose their arm or hand to more easily rehabilitate to their normal lives and jobs. This can decrease the amount of unemployed people, so they will not have to depend on a government allowance anymore. This can be beneficial for the economy.

However, easier and shorter rehabilitation can cause nurses to lose their jobs. On the other hand, there is a lot of pressure on people in the health care sector right now, so it can also be seen as a reduction on work pressure.

Furthermore, if the prosthetics become too advanced, people might be inclined to cut off their own arm. This should be prevented, unless humanity will turn into cyborgs in the future.

Moreover, the prosthetics can cause discrimination if they become so advanced that companies rather have a person with a prosthetic than one without it. They might be able to work faster and more precise than people who still have their arm.

Enterprise

The enterprise will be able to sell prostheses. To create this new technology, there will be more employment opportunities.

Deliverables and Approach

The results of this project will be presented in the form of a 3D Functional Model (FuMo) together with an algorithm. The model will consist of a design for a prosthetic hand and wrist in a 3D environment (Probably NX10). Since the design will be virtual as of now, the two deliverables cannot be combined to give a single result. However the algorithm's functionality will still be proven in another way. If in the next couple weeks it is concluded that a physical model is possible to make within the given time frame, the option will be considered.

Planning

Planning and milestones

Name	Week 1	Week 2	Week 3	Week 4
Eva	Search for useful sources and make a planning; Do research on robot hands	Relate research on human hands to the research on robotic hands; Do research on EEGs; Make an appointment with Spierings	Do research on EEGs and wrist prosthetics	Contact companies for EEG results/equipment; Test dynamixel
Jurre	Finding sources; do research on human hands	Relate research on human hands to the research on robotic hands; Contact companies for EEG results/equipment	Do research on EEGs; Contact companies for EEG results/equipment	Contact companies for EEG results/equipment; Test dynamixel
Karsten	Write about deliverables; Do research on control mechanism for prosthetic hand	Start on the design (FuMo); Contact Rommers	Get dynamixels and do research on them; Do research on RS485; Order USB2Dynamixel	Contact companies for EEG results/equipment; Test dynamixel
Steven	Write about users; Do research on machine learning for prostheses	Start on machine learning algorithm	Work on machine learning algorithm	Work on machine learning algorithm
Thijs	Do research on machine learning for prostheses	Start on machine learning algorithm	Work on machine learning algorithm	Work on machine learning algorithm
Name	Week 5	Week 6	Week 7	Week 8
Eva	Test dynamixel	Finish wrist movement dynamixel/CAD; Work on presentation	Presentation and finish Wiki	-
Jurre	Test dynamixel	Finish wrist movement dynamixel; Work on presentation	Presentation and finish Wiki	-
Karsten	Work on EEG-results from internet; Test dynamixel	Finish wrist movement dynamixel/CAD; Work on presentation	Presentation and finish Wiki	-
Steven	Improvements in efficiency/running time of algorithm	Finish algorithm; Work on presentation	Presentation and finish Wiki	-
Thijs	Improvements in efficiency/running time of algorithm	Finish algorithm; Work on presentation	Presentation and finish Wiki	-

The yellow fields are the milestones.

Literature study

The human basics

The human hand

The human hand can be separated into three main parts. The forearm, the wrist and the fingers. For clarity we will look at each part separately. The fingers consist of two hinge joints and condyloid joint at the base of the digit. While they are separate joints, they cannot work independently. A tendon connects each digit with the associated muscles in the forearm. For each digit there is a pair of muscles of which one extends and one curls the digit. There is a second group of muscles also situated in the forearm that spreads the fingers apart. .

The thumb is the only digit that slightly differs from this. By allowing one more degree of freedom to the first joint and adding a muscle in the hand that allows movement in this direction.

The wrist consists of several bones that together function as a single condyloid joint. This allows for it to flex, extend and deviate to both sides. The degrees of motion for the joint are 60° for flexing and extending and 20-30° deviation to both sides. .

The forearm acts like a pivot joint using the two bones there to rotate the wrist. This gives the wrist about 180° of rotation. The forearm is also the place where almost all of the muscles controlling the hand are situated. This allows for the muscles to become larger and therefore stronger than if they were confined within the hand.

Movement in the brain

Generating movement is very difficult. It starts with the brain, in the primary motor cortex. This has to decide not only which muscles to contract for the movement, but also how much power will be needed. For example, to pick up a glass the brain has to detect where the hand is relative to the glass, so it knows how the hand should move towards it, and it should estimate the weight of the glass in order the know the force needed to pick it up. If a limb needs more control, there is more space reserved for it in the brain. Hands and wrists are quite complex and therefore need a lot of control, so they are largely represented in the motor cortex. When a signal is produced in the brain, it goes through the spinal cord to the muscles. Signals in the left half of the brain go to the right side of the body.

State of the Art

Robot hands

In prosthetic and robot arms, the hand and wrist are difficult to develop. Many prosthetics do not even have a wrist motion and fingers cannot move separately, since these are difficult to mimic. Below, three state-of-the-art robots are discussed: first the SoftHand Pro-H, a dexterous hand and the wrist mechanism of the humanoid robot SARA.

Prosthetics can have different kinds of hands. The older ones have fingers, but they cannot move separately from each other. Some can, like the SoftHand Pro-H. This is one of the newer prosthetic devices, which had nineteen degrees of freedom. The fingers can grasp and make a fist, and are soft yet robust, which allows them to also hold pencils or irregular shapes. It is designed in a way that it can actually grip with a force of 40N.

Also, a dexterous prosthetic hand has been developed. This hand works with micro servos and is connected to the brain. Electroencephalography controls the hand. All fingers can move separately, and it has many degrees of freedom. The finger bones are moved by servo motors, which are installed at the base of the fingers. The fingers cannot only bend, but also move sideways. The thumb is even controlled by two motors, giving it more degrees of freedom. It can even touch all the other fingers. This already comes very close to an actual human hand, perfect for amputees.

Wrists in humanoid robots are not the same as in humans. In robots, there are typically two types. The first contains of one or more joints with one degree of freedom, while the second consists of one joint with either two degrees of freedom (a condylar joint) or three degrees of freedom (ball-and-socket joint). Wrists of the first type are often rigid and have high carrying capacity and reliability. They have high positioning accuracy, since they have low backlash. This is important for motion control. Wrists with only one joint and multiple degrees of freedom, are actually more human-like and are combined with artificial muscles.

Control of prosthetics

Several ways to control prosthetics have been used in the past. Some were mechanic, but lately also electromyography and electroencephalography are used. Prosthetic arms working with the spinal cord have been developed, so the wearer only has to think about the movement to do it. Many existing prosthetics work on the twitching of remaining muscles. Other research is looking at gripping intuitively, so the user can grip something without consciously thinking about it. Working with the neural network could also allow sensory feedback, so the user can have ‘a sense of touch’. This way, the imposed pressure can be better regulated. It will give the user their feeling back.

One of the methods used for prosthetics, is surface electromyography (sEMG). It can be used to control myoelectric prosthetics. In order to know which sEMG signal corresponds to which wrist movement, the sEMG of people making certain wrist movements was recorded. This movement was then classified by the system. It resulted in an accuracy of 84.93% in real-time classification, so the prosthetic would make the right movement in about 85% of the cases.

Furthermore, EEGs can be used to control a prosthetic hand as well. However, EEGs of movement are not yet understood completely. Most of the times, only a linear model is used for them, which can only explain 10% of variance in cortical response, and over 80% of the response is nonlinear. Now, a new, dynamic model was designed to try to explain this. This model could explain 46% of the variance, which is a significant increase. This research has been done with wrist movement, so in order to use a prosthetic wrist, this could mean more precision in control of the prosthesis.

A research about EEG-signals versus cortical current source (CCS) signals, found that EEG-CCS signals were more accurate in movement than only EEG-signals. EEG-signals could cause oscillatory movements, while EEG-CCS signals displayed the muscle movement better. Another research tried to fuse sEMG and EEG signals. This turned out to be significally better than either sEMG or EEG signals. The classification accuracy increased more than 14%. Furthermore, EEGLAB exists. This is an open source toolbox, used to analyse EEG dynamics. It runs under the MATLAB environment and processes single-trial EEG data of a number of channels. The data can be visualized , preprocessed and an independent component analysis can be performed. New scripts can be made with info from the EEGLAB script.

Since EEGs are measured by putting electrodes on the scalp, there will be no difference for many paralysed people. Mostly, these people are paralysed because of a fault in the spinal cord. Movement already starts in the brain, so when controlling a prosthetic with EEG-data, there will be no blockade in the spine. Hence, paralysed people will be able to use them as well.

Feedback control for prosthetics research

Sensors for the measurement of applied force can be used in prosthetic hands to control the strength of the hand when grabbing objects. This technology is already implemented in 'soft' grippers in the form of a pneumatic soft sensor (PSS), consisting of a silicon body and a flexible pressure sensor and has been tested succesfully on rubber balls. This could be advantageous for picking up fragile objects, raw eggs or fruit for example, in order to not break or bruise the object.

Since interacting with humans and fragile (biological) objects is a requirement for hand prosthetics, soft and flexible sensors and actuators are vital. Soft actuators work through the integration of microscopic changes at molecular level into a macroscopic deformation. This allows them to be 3D-printed at a reasonable price. This holds that these actuators can be embedded within the soft tissue of the skin-like silicon if necessary.

A prosthesic hand behaves similar to an exoskeleton when controlling forces and movement. There has been alot of research on the topic of controlling exoskeletons. The mechanism can be controlled by: a model, hierarchy, parameter or usage based control system. Most exoskeletons nowadays use a combination of mentioned control systems to optimise performance. The sensory feedback can also be interpreted by the human carrier of the exoskeleton / prosthesis. Through the use of discrete (event-driven) over continuous feedback, the traditional limitation of neural delays can be circumvented.

Force feedback turns out to be difficult for haptic devices, since they have limited force accuracy. New research is conducted to use a variable motion mapping method. Now the motion mapping coefficient could be regulated according to the object stiffness. This method could lead to identifying objects without having to see them. However, the stiffness of the object has to be estimated beforehand. It could still be more efficient than current methods.

Machine Learning

A machine learning algorithm is an algorithm that trains itself on patterns in the data it is given. There are different kinds of ways to train such an algorithm. There is supervised and unsupervised learning. In supervised learning the algorithm is trained using a data set for which the result is already known. This means that the network can get to a result and check its own performance and optimize for this training set data. This way it can become very good at predicting the results which were given. However, it will not recognise any results that were not provided in the training data. There is also unsupervised learning. What this does is it tries to group the data or find patterns based on de difference between specific data points. Then there is also deep learning. This is using a neural network that has many layers in order to group data. This can be done supervised or unsupervised. As we are only interested in the signals that stimulate wrist movement (and not in grouping all brain activity), it is logical to use supervised learning. For EEG's it was found that the random forest algorithm works very well, and better then the other tested algorithms (including neural networks) . Thus was chosen for usual supervised machine learning using the random forest algorithm.

The random forest algorithm

The random forest algorithm is an ensamble approach. This means that the data is split up into many different smaller data sets. The algoritm tries to optimalize each of these n sets. This way there are many small optimalized splits. A vector addition of these results then form the final result. Note that since n is very large, most of the data will be equally well represented. This is better than jsut running the algorithm on the full data set. The reason for this is that there is always some noise and variance in the given data set. If the full data set is taken for training then the algorithm will be training on all of the data, so also the specific outliers of that data set, which leads to overfitting on these precise points.

More in detail, the data is first split into many different parts. Then a single data split is created by adding some of these parts together. On this set a single tree is trained. A tree will split the data in such a way that data with the same outcome fall into the same categories. This is done by using the gini-index of the tree, which tells you how much data with the same outcome is in the same category. This way it will continue to make split points until the data is grouped optimally with the constrainst of a mininal group size and a maximal tree depth. This way a decision tree is made. Given a data point it can use this decision tree to predict the outcome. Now this is done for n trees (with n a large integer) with each given a different split of the data. This way n decision trees are created for which you can predict the outcome based on a single data point. The algorithm is now trained. If one now inserts a data point on this trained forest the random forest algorithm will ask the outcome of each of these trees and take the most represented answer.

Machine learning for prostheses

Machine learning has been studied as a way to make prostheses react to electric pulses in the brain. Research has been done on this subject with rhesus monkeys. In this research, rhesus monkeys were first trained to perform a 3D outreaching task, and later implanted with chronic electrode arrays. After that, Principal Component Analysis (PCA) was used to reduce the amount of neurons being used in the analysis. Then a machine learning algorithm was employed to map the neurons being used to the direction of movement. The model made in this way was tested, and could correctly predict when the hand was in motion 81% of the time, and the motion ended less than 3 cm from the correct endpoint in less than 50% of the cases, so there's room for improvement in this algorithm.

There has been some extensive research to how cells in the motor cortex of the brain are related to the direction of movement of a hand. This research shows that cells have a ‘preferred direction’ and are only dependent in the movement of the hand, so not the position. These directions can be determined by measuring the discharge rate of these cells and the real activity of the hand. The total movement of the hand can then be predicted using a ‘population vector’, calculated by a sum of all of the cells times a weighing factor based on the discharge rate of the cell. This vector is within the 95% directional cone for more than 80% of the time using 475 cells (and more cells lead to higher accuracy) Unfortunately this will not be very useful for our further research, since we will not have the option to implant devices on human brains in order to detect their discharge rate for obvious reasons.

EEG-data

At first, the idea was to get proper EEG-equipment to achieve data of the right wrist movements. One or more of the group members would be the test person, and if possible Arie Rommers as well. The plan was to compare the results, because for all the group members it is known that they have their hand and can perform the movements, but Rommers does not have his arm anymore and has not performed any wrist movement for a long time. This could cause a difference in the way the brain would perform the movement. If the prosthesis would ever actually be used on somebody, this could be important information.

However, it quickly turned out getting the equipment was difficult. The university itself works with the equipment for research in epilepsy, but when contacting them they did not answer. When trying the hospital, they said only academic hospitals lend their equipment to externals. Therefore, the Radboud University in Nijmegen was contacted, but using the state-of-the-art EEG-equipment of the Donders Institute at the university was extremely expensive and not possible in the time available for the project. During one of the tutor meetings, this problem was discussed and the advice was to look online for available data. Quickly, a site was found with many external links to free EEG-data. Sadly, there was barely any data of wrist movements, and if there was, it was not what was needed for this project. To solve this, other data was chosen from the Brain Computer Interface competition III. This data did not necessarily contain specific movements, but movement of specific limbs. Since the wrist has four directions to move in, a data set was chosen with 4 different limbs to move: the left hand, right hand, foot and tongue. This test was done with three people, but only the data of test person 1 was used to train the algorithm. It might not be ideal not having data of the wrist movement only, but it is enough to prove the idea of a robotic prosthetic controlled by the brain might work.

When looking for the EEG-data online, a Matlab toolbox to process EEG-data was found as well, called EEG-lab (see Literature Study). It was expected that this might help for the algorithm, but turned out not to be necessary. However, an extension of this toolbox was also found to predict which limb was used when getting certain data. Since the idea of this project was to predict the wrist movement with a machine learning algorithm, this extension could help with recognising patterns in the data. It was therefore looked into a bit, but not too extensively, since it is not desirable to copy it.

Were there more time for this project, it might be possible to achieve the desired data and train the algorithm on that. However, specific movement of the wrist could be more difficult than recognising certain limbs. The control of each limb is placed on a certain place within the motor cortex. Within the area of a limb, the movements are generated. Since all wrist movement will be located in the wrist section of the brain, it will all be close together and therefore difficult to be distinguished by the algorithm. The electrodes will have to be put closely together and can therefore have noise from each other. It is however shown that the algorithm can actually distinguish the signals. More about that in the next chapter.

Coaching questions

Coaching Questions Group 7