PRE2017 3 Groep7

Introduction

For the course Robots Everywhere, the goal is to do a project about robots. The groups can choose any topic, as long as it contains robots and can be finished in 7 weeks. The course started by brainstorming on this topic, after which Group 7 decided to look at robot prosthetics. Since this is quite a broad topic, it was necessary to narrow it down. The group wanted to do something challenging and new. Therefore, it was decided to look at wrists, because there are no robotic wrist prosthetics on the market yet. Since the fine motor skills come from wirsts and fingers, this could be a valuable addition to current prosthetics. The idea was to control these prosthetics directly with the brain, so paralysed people can use it as well. To turn the EEG-data of the brain into a movement, a machine learning algorithm was used.

On this wiki, the whole process of the project can be found. First, the problem statement and all objectives will be discussed, then the USE-aspects, deliverables and approach of the project and the planning the group. After that, there is a literature study, to know what is already on the market. Finally, it will be discussed how the group worked towards a result and what this result looked like. This final part will contain three chapters: the EEG-data, the machine learning algorithm and the functional model.

Problem Statement and Objectives

In the U.S., it is estimated that one in every 190 people have suffered from limb loss. This shows the importance and opportunity of prosthetics. For all these people, part of their old functions can be regained with the use of a prosthetic limb. While technology evolves, the prosthetics are able to perform more functions and are easier to handle. Where old prosthetics are mostly body-powered, they are now myoelectric or even robotic. These robotic prosthetics should eventually be able to mimic all old functions of the lost limb.

One of the challenges in the field is the hand. Especially the wrist is difficult. This joint creates a lot of movement, working with the forearm. To make a joint with as much movement as the wrist and the same power, has proven to be a difficult task. How can one design a full flexible wrist, while also giving it enough strength to lift objects? That will be one of the main questions during this project.

Another big challenge is how to control the prosthetic. Some of the newer, robotic limbs can actually work with the nerve system or the brain. This still proves difficult, since the user will need to learn to use the prosthetic in a natural way and some limbs, like the arm, have many different degrees of freedom to take into account. However, machine learning might prove a way for the prosthetic and the user to meet halfway: the user has to adjust to the prosthetic, and the prosthetic will learn how the user behaves. How do these algorithms work and what could it mean for the industry? By making our own algorithm, we will try to find out whether this might be a breakthrough for prosthetic use.

USE

USE:

It is important to look at the user aspects of wrist prostheses. For this there will be looked at the end users, society and enterprise.

Users: The users of a hand prothesis will be people who have somehow lost their hand or arm entirely. Their current way of living would be improved by giving them the possibility to use their prosthesis to pick up objects, which would also help them with basic uses of the hand. Further development could lead to even more possibilities such as catching objects or typing. This would greatly improve the quality of life for these people.

The preferences of the users would of course be a full-fuctioning arm and hand with perhaps even additional functions. The question is whether or not this is desirable, since if a prosthetic hand is more useful than a human one it could incentivise people to have a prostetic hand even if their current hand is still functioning.

Society: A hand prothesis will allow people who lose their arm or hand to more easily rehabilitate to their normal lives and jobs.

Enterprise: The enterprise will be able to sell prostheses.

Deliverables and Approach

The results of this project will be presented in the form of a 3D Functional Model (FuMo) together with an algorithm. The model will consist of a design for a prosthetic hand and wrist in a 3D environment (Probably NX10). Since the design will be virtual as of now, the two deliverables cannot be combined to give a single result. However the algorithm's functionality will still be proven in another way. If in the next couple weeks it is concluded that a physical model is possible to make within the given time frame, the option will be considered.

Planning

Planning and milestones

Name	Week 1	Week 2	Week 3	Week 4
Eva	Search for useful sources and make a planning; do research on robot hands	Relate research on human hands to the research on robotic hands; do research on EEGs; make an appointment with Spierings	Do research on EEGs and wrist prosthetics	Try to get EEG results; build wrist; test dynamixel
Jurre	Finding sources; do research on human hands	Relate research on human hands to the research on robotic hands; get hands on EEG equipment	Do research on EEGs	Try to get EEG results; test dynamixel
Karsten	Write about deliverables; do research on control mechanism for prosthetic hand	Start on the design (FuMo); make appointment with Rommers	Build wrist	Build wrist; test dynamixel
Steven	Write about users; do research on machine learning for prostheses	Start on machine learning algorithm	Work on machine learning algorithm	Finish first version of algorithm
Thijs	Do research on machine learning for prostheses	Start on machine learning algorithm	Work on machine learning algorithm	Finish first version of algorithm
Name	Week 5	Week 6	Week 7	Week 8
Eva	Work on EEG results	Finish wrist movement; work on presentation	Presentation	-
Jurre	Work on EEG results	Finish wrist movement; work on presentation	Presentation	-
Karsten	Help with the EEGs	Finish wrist movement; work on presentation	Presentation	-
Steven	Improvements in efficiency/running time of algorithm	Finish algorithm; work on presentation	Presentation	-
Thijs	Improvements in efficiency/running time of algorithm	Finish algorithm; work on presentation	Presentation	-

The yellow fields are the milestones.

Literature study

State of the art

The human hand

The human hand can be separated into three main parts. The forearm, the wrist and the fingers. For clarity we will look at each part separately. The fingers consist of two hinge joints and condyloid joint at the base of the digit. While they are separate joints, they cannot work independently. A tendon connects each digit with the associated muscles in the forearm. For each digit there is a pair of muscles of which one extends and one curls the digit. There is a second group of muscles also situated in the forearm that spreads the fingers apart. .

The thumb is the only digit that slightly differs from this. By allowing one more degree of freedom to the first joint and adding a muscle in the hand that allows movement in this direction.

The wrist consists of several bones that together function as a single condyloid joint. This allows for it to flex, extend and deviate to both sides. The degrees of motion for the joint are 60° for flexing and extending and 20-30° deviation to both sides. .

The forearm acts like a pivot joint using the two bones there to rotate the wrist. This gives the wrist about 180° of rotation. The forearm is also the place where almost all of the muscles controlling the hand are situated. This allows for the muscles to become larger and therefore stronger than if they were confined within the hand.

Robot hands

In prosthetic and robot arms, the hand and wrist are difficult to develop. Many prosthetics do not even have a wrist motion and fingers cannot move separately, since these are difficult to mimic. Below, three state-of-the-art robots are discussed: first the SoftHand Pro-H, a dexterous hand and the wrist mechanism of the humanoid robot SARA.

Prosthetics can have different kinds of hands. The older ones have fingers, but they cannot move separately from each other. Some can, like the SoftHand Pro-H. This is one of the newer prosthetic devices, which had nineteen degrees of freedom. The fingers can grasp and make a fist, and are soft yet robust, which allows them to also hold pencils or irregular shapes. It is designed in a way that it can actually grip with a force of 40N.

Also, a dexterous prosthetic hand has been developed. This hand works with micro servos and is connected to the brain. Electroencephalography controls the hand. All fingers can move separately, and it has many degrees of freedom. The finger bones are moved by servo motors, which are installed at the base of the fingers. The fingers cannot only bend, but also move sideways. The thumb is even controlled by two motors, giving it more degrees of freedom. It can even touch all the other fingers. This already comes very close to an actual human hand, perfect for amputees.

Wrists in humanoid robots are not the same as in humans. In robots, there are typically two types. The first contains of one or more joints with one degree of freedom, while the second consists of one joint with either two degrees of freedom (a condylar joint) or three degrees of freedom (ball-and-socket joint). Wrists of the first type are often rigid and have high carrying capacity and reliability. They have high positioning accuracy, since they have low backlash. This is important for motion control. Wrists with only one joint and multiple degrees of freedom, are actually more human-like and are combined with artificial muscles.

Control of prosthetics

Several ways to control prosthetics have been used in the past. Some were mechanic, but lately also electromyography and electroencephalography are used. Prosthetic arms working with the spinal cord have been developed, so the wearer only has to think about the movement to do it. Many existing prosthetics work on the twitching of remaining muscles. Other research is looking at gripping intuitively, so the user can grip something without consciously thinking about it. Working with the neural network could also allow sensory feedback, so the user can have ‘a sense of touch’. This way, the imposed pressure can be better regulated. It will give the user their feeling back.

One of the methods used for prosthetics, is surface electromyography (sEMG). It can be used to control myoelectric prosthetics. In order to know which sEMG signal corresponds to which wrist movement, the sEMG of people making certain wrist movements was recorded. This movement was then classified by the system. It resulted in an accuracy of 84.93% in real-time classification, so the prosthetic would make the right movement in about 85% of the cases.

Furthermore, EEGs can be used to control a prosthetic hand as well. However, EEGs of movement are not yet understood completely. Most of the times, only a linear model is used for them, which can only explain 10% of variance in cortical response, and over 80% of the response is nonlinear. Now, a new, dynamic model was designed to try to explain this. This model could explain 46% of the variance, which is a significant increase. This research has been done with wrist movement, so in order to use a prosthetic wrist, this could mean more precision in control of the prosthesis.

A research about EEG-signals versus cortical current source (CCS) signals, found that EEG-CCS signals were more accurate in movement than only EEG-signals. EEG-signals could cause oscillatory movements, while EEG-CCS signals displayed the muscle movement better. Another research tried to fuse sEMG and EEG signals. This turned out to be significally better than either sEMG or EEG signals. The classification accuracy increased more than 14%. Furthermore, EEGLAB exists. This is an open source toolbox, used to analyse EEG dynamics. It runs under the MATLAB environment and processes single-trial EEG data of a number of channels. The data can be visualized , preprocessed and an independent component analysis can be performed. New scripts can be made with info from the EEGLAB script.

Feedback control for prosthetics research

Sensors for the measurement of applied force can be used in prosthetic hands to control the strength of the hand when grabbing objects. This technology is already implemented in 'soft' grippers in the form of a pneumatic soft sensor (PSS), consisting of a silicon body and a flexible pressure sensor and has been tested succesfully on rubber balls. This could be advantageous for picking up fragile objects, raw eggs or fruit for example, in order to not break or bruise the object.

Since interacting with humans and fragile (biological) objects is a requirement for hand prosthetics, soft and flexible sensors and actuators are vital. Soft actuators work through the integration of microscopic changes at molecular level into a macroscopic deformation. This allows them to be 3D-printed at a reasonable price. This holds that these actuators can be embedded within the soft tissue of the skin-like silicon if necessary.

A prosthesic hand behaves similar to an exoskeleton when controlling forces and movement. There has been alot of research on the topic of controlling exoskeletons. The mechanism can be controlled by: a model, hierarchy, parameter or usage based control system. Most exoskeletons nowadays use a combination of mentioned control systems to optimise performance. The sensory feedback can also be interpreted by the human carrier of the exoskeleton / prosthesis. Through the use of discrete (event-driven) over continuous feedback, the traditional limitation of neural delays can be circumvented.

Force feedback turns out to be difficult for haptic devices, since they have limited force accuracy. New research is conducted to use a variable motion mapping method. Now the motion mapping coefficient could be regulated according to the object stiffness. This method could lead to identifying objects without having to see them. However, the stiffness of the object has to be estimated beforehand. It could still be more efficient than current methods.

Machine learning for prostheses

Machine learning has been studied as a way to make prostheses react to electric pulses in the brain. Research has been done on this subject with rhesus monkeys. In this research, rhesus monkeys were first trained to perform a 3D outreaching task, and later implanted with chronic electrode arrays. After that, Principal Component Analysis (PCA) was used to reduce the amount of neurons being used in the analysis. Then a machine learning algorithm was employed to map the neurons being used to the direction of movement. The model made in this way was tested, and could correctly predict when the hand was in motion 81% of the time, and the motion ended less than 3 cm from the correct endpoint in less than 50% of the cases, so there's room for improvement in this algorithm.

There has been some extensive research to how cells in the motor cortex of the brain are related to the direction of movement of a hand. This research shows that cells have a ‘preferred direction’ and are only dependent in the movement of the hand, so not the position. These directions can be determined by measuring the discharge rate of these cells and the real activity of the hand. The total movement of the hand can then be predicted using a ‘population vector’, calculated by a sum of all of the cells times a weighing factor based on the discharge rate of the cell. This vector is within the 95% directional cone for more than 80% of the time using 475 cells (and more cells lead to higher accuracy) Unfortunately this will not be very useful for our further research, since we will not have the option to implant devices on human brains in order to detect their discharge rate for obvious reasons.

Machine Learning

A machine learning algorithm is an algorithm that trains itself on patterns in the data it is given. There are different kinds of ways to train such an algorithm. There is supervised and unsupervised learning. In supervised learning the algorithm is trained using a data set for which the result is already known. This means that the network can get to a result and check its own performance and optimize for this training set data. This way it can become very good at predicting the results which were given. However, it will not recognise any results that were not provided in the training data. There is also unsupervised learning. What this does is it tries to group the data or find patterns based on de difference between specific data points. Then there is also deep learning. This is using a neural network that has many layers in order to group data. This can be done supervised or unsupervised. As we are only interested in the signals that stimulate wrist movement (and not in grouping all brain activity), it is logical to use supervised learning. For EEG's it was found that the random forest algorithm works very well, and better then the other tested algorithms (including neural networks) . Thus was chosen for usual supervised machine learning using the random forest algorithm.

The random forest algorithm

The random forest algorithm is an ensamble approach. This means that the data is split up into many different smaller data sets. The algoritm tries to optimalize each of these n sets. This way there are many small optimalized splits. A vector addition of these results then form the final result. Note that since n is very large, most of the data will be equally well represented. This is better than jsut running the algorithm on the full data set. The reason for this is that there is always some noise and variance in the given data set. If the full data set is taken for training then the algorithm will be training on all of the data, so also the specific outliers of that data set, which leads to overfitting on these precise points.

More in detail, the data is first split into many different parts. Then a single data split is created by adding some of these parts together. On this set a single tree is trained. A tree will split the data in such a way that data with the same outcome fall into the same categories. This is done by using the gini-index of the tree, which tells you how much data with the same outcome is in the same category. This way it will continue to make split points until the data is grouped optimally with the constrainst of a mininal group size and a maximal tree depth. This way a decision tree is made. Given a data point it can use this decision tree to predict the outcome. Now this is done for n trees (with n a large integer) with each given a different split of the data. This way n decision trees are created for which you can predict the outcome based on a single data point. The algorithm is now trained. If one now inserts a data point on this trained forest the random forest algorithm will ask the outcome of each of these trees and take the most represented answer.

Coaching questions

Coaching Questions Group 7