View source for First project: Braille tablet

= 0LAUK0: Robots Everywhere Group 2 =

= Group members =
*Yngwie Baron (0936539)
*Axel Deenen (0947031)
*Moos Müller (0936214)
*Dimitar Nikolov (1000095)
*Wybe van Vlokhoven (0914565)

= Project definition = 
'''After contacting someone who specializes in the technology that had our initial interest, we decided to switch to another subject. Below is what we had done up till then.'''

A huge amount of information is communicated to people through text. Many of that is not obtainable for fully blind or near-blind people. It is possible to make a large portion of this information accessible by designing a portable device to assist the blind. This device would be able to scan text and present it in braille code. The goal is to make it easy for the blind to read books, hand written texts and text contained within images. A hypothetical concept for this device will be designed for fully blind people.

==Approach==
At first literature will be studied to learn about the current technological aids for the blind and for the current state of OCR (optical character recognition) and refresh-able braille displays. Then contact with a institute, involved in research for helping the blind, will be sought to gain feedback on the device. This feedback will then be carefully incorporated into the device to ensure its practical use.

==Results from literature research==
After conducting a vast literature research, a brief summary of all articles and patents deemed relevant has been made. These summaries are listed below.

'''A BRAILLE O.C.R. FOR BLIND PEOPLE'''<ref>Hermida, X. F., Rodríguez, A. C., & Rodríguez, F. M. (2008). A BRAILLE O.C.R. FOR
BLIND PEOPLE</ref><br>
In this paper the principles for an optical character recognizer, OCR, for written braille code are developed. Several image processing techniques are incorporated such as adaptive thresholding and skew angle detection. The paper was sponsored by the National Organization of Spanish Blind People.

'''OCR-OPTICAL CHARACTER RECOGNITION'''<ref>Verma, A., Arora, S., & Verma, P. (2016). Ocr-Optical Character Recognition. 7th
International Conference on Recent Innovations in Science, Engineering and
Management, 230–240.</ref><br>
In this paper a simple, efficient and less costly OCR for regular text is presented. The OCR uses a database to recognize the characters to lower the computational cost of the program. The paper discusses various stages of the program including preprocessing and post processing.

'''Portable Braille Computer Device'''<ref>Kahn, S. (2003). PORTABLE BRAILLE COMPUTER DEVICE.
doi: 10.1126/science.Liquids</ref><br>
This patent describes a portable device that can scan text using a scanner, interpret the scanned text using an OCR program and present it using a braille display. The patent also claims to provide a complete information management system within the device. Such as a personal agenda and notes.

'''Wearable assistive devices for the blind'''<ref>Velazquez, R. (2010). Wearable assistive devices for the blind. Lecture Notes in Electrical
Engineering, 75 LNEE, 331–349. doi: 10.1007/978-3-642-15687-8-17</ref><br>
This paper reviews important steps made in the field of wearable assistive devices for the blind. From these important steps universal design concepts for these wearable devices, and also other systems for the blind, are extracted. The future potential of various prototypes are also discussed.

'''A Blind Person’s Interactions with Technology'''<ref>Tenenberg, J. (n.d.). A Blind Person’s Interactions with Technology.
COMMUNICATIONS OF THE ACM, 52 (8). doi: 10.1145/1536616.1536636</ref><br>
This paper considers a blind person's interaction with various technologies. The technologies are discussed both practically and socially, since both are important when using it in one’s daily life. Findings about the interaction with technology of a person blind from birth are presented.

'''An interactive and multi-functional refreshable Braille devicefor the visually impaired'''<ref>Fatih Basciftci, A. E. (2016). An interactive and multi-functional refreshable Braille device for the visually impaired. Displays, 41 , 9.</ref><br>
The authors of this paper have developed a book reader with 96 braille cells, and also includes to option to convert text to audio. The articles describes in depth the design of the device, including all the buttons and their functions. It uses a scanner which sends information to the device via USB or Bluetooth, which then converts the information to braille. Tests done by the authors with visually impaired people that can or cannot read braille indicate a high usability of this device. Overall, this device is very similar to our initial conception.

'''Methods for presenting braille characters on a mobile device with a touchscreen and tactile feedback'''<ref>Rantala, J., Raisamo, R., Lylykangas, J., Surakka, V., Raisamo, J., Salminen, K., Hippula, A. (2009). Methods for presenting braille characters on a mobile device with a touchscreen and tactile feedback. IEEE Transactions on Haptics, 2 (1), 28–39. doi: 10.1109/TOH.2009.3</ref><br>
In order to circumvent the high cost of a mechanical braille reader (usually in the range of 5000 to 15.000 USD, according to the authors), different methods for implementing braille reading in a mobile device were researched. The authors identified three different ways of presenting braille characters, and showed the feasibility of all methods through experiments in which participants were asked to identify a single character. Both the accuracy and reading time  varied, and the authors stated that further research would be required to find out how to present words or sentences.

'''Determining the optimum font size for braille on capsule paper'''<ref>Watanabe, T. (2014). Determining the optimum font size for braille on capsule paper.
IEICE Transactions on Information and Systems, E97-D(8), 2191–2194. doi:10.1587/transinf.E97.D.2191</ref><br>
This paper presented research on what is the optimal size of a braille front. The results showed that there was an optimum size, which gave the highest reading speed and the lowest error. The optimal size showed little variation for different age groups. The most important conclusion to be drawn is that enlarging the font size doesn’t necessarily improve either reading accuracy or reading time.

'''Camera Reading for Blind People'''<ref>Neto, R., & Fonseca, N. (2014). Camera Reading for Blind People. Procedia Technology,
16 , 1200–1209. Retrieved from http://linkinghub.elsevier.com/retrieve/pii/S2212017314003624 doi: 10.1016/j.protcy.2014.10.135</ref><br>
This article describes the foundation for a technique that should enable blind people to photograph a piece of text using a mobile device, and subsequently have that device read out the text. The paper present an overview and state of art (from 2014) of Optical Character Recognition (OCR) and Text-To-Speech (TTS) software.   

'''A Review on Optical Character Recognition Techniques'''<ref>Modi, H., Scholar, P. G., & Parikh, M. C. (2017). A Review on Optical Character
Recognition Techniques. International Journal of Computer Applications, 160 (6), 975–8887. Retrieved from http://www.ijcaonline.org/archives/volume160/number6/modi-2017-ijca-913061.pdf</ref><br>
This paper describes more in depth the process of OCR. The steps through which an OCR protocol goes are subdivided into digitization, pre-processing, segmentation and feature extraction.

'''Computerized microfluidic cell culture using elastomeric channels and Braille displays'''<ref>Gu, W., Zhu, X., Futai, N., Cho, B. S., & Takayama, S. (2004). Computerized microfluidic cell culture using elastomeric channels and Braille displays. Proceedings of the National Academy of Sciences, 101 (45), 15861–15866. Retrieved from http://www.pnas.org/cgi/doi/10.1073/pnas.0404353101 doi: 10.1073/pnas.0404353101</ref><br>
This article describes a new computer-controlled microfluidics system used on a refreshable Braille display. It is a grid of 320 vertical pins to depict the braille. Due to a new method used to power the integrated pumps and valves, controlling the pins, in the silicone rubber the system is a lot more efficient. More pumps and valves can be put in a smaller space improving the display immensely.

'''Optical Character Recognition for Handwritten Cursive English characters'''<ref>Aparna, A., & Muthumani, P. I. (2014). Optical Character Recognition for Handwritten
Cursive English characters. International Journal of Computer Science and Information Technologies, 5 (1), 847–848.</ref><br>
This paper talks about OCR for handwritten text. To improve OCR for handwritten text a new method is developed with noise cancelling. A median filter is applied to filter out unwanted scribbles to improve the results. To get the results the following steps are used: image acquisition, image preprocessing, segmentation, feature extraction, recognition.

'''Real-time scene text localization and recognition'''<ref>Neumann, L., & Matas, J. (2012). Real-time scene text localization and recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 3538–3545. doi: 10.1109/CVPR.2012.6248097</ref><br>
An article about text recognition. It discusses a method for real-time text recognition on images for example. It creates an Extremal Region (ER), a box, around the text region to include the entire text without missing anything, while not including areas where no text is present. The system is tested by two methods. The ICDAR 2011 and the Street View Text dataset.

'''A system for converting print into Braille'''<ref>Blenkhorn, P. (1997). A system for converting print into Braille. IEEE Transactions on
Rehabilitation Engineering, 5 (2), 121–129. doi: 10.1109/86.593266</ref><br>
The paper goes over different methods of braille that are in use. It adds certain language rules to make converting text to braille easier. It does however also state that the rules should be developed further.

'''REFRESHABLE BRAILLE DISPLAY SYSTEM'''<ref>Schmidt, R. (2002). REFRESHABLE BRAILLE DISPLAY SYSTEM. Patent no: '''US 6,354,839 B1'''</ref><br>
This is a patent of a refreshable braille display system. It is a display that can extend and retract dots to form words or short sentences. It describes a microelectromechanical device.

'''Voice Assisted Text Reading System for Visually Impaired Persons Using TTS Method'''<ref>Sanjana, B., & RejinaParvin, J. (2016). Voice Assisted Text Reading System for Visually
Impaired Persons Using TTS Method. , 6 (3), 1–5. doi: 10.9790/4200-0603031523 </ref><br>
In this article the TTS method is elaborated. A finger mounted camera is used to capture the text image from the printed text and the captured image is analyzed using OCR. A predefined dataset is loaded in order to match the observed text with the captured image. Once it is matched the text is synthesized for producing speech output. MATLAB is used for performance analysis.

'''Eight-dot Braille'''<ref>Dixon, B. J. (2007). Eight-dot Braille. (September).</ref><br>
Elaboration on the use of 8 dot braille. It can be used to give more information per cell and it opens a way to produce a wider variety of symbols. It might be used to determine the users location.

'''FingerReader: A Wearable Device to Support Text-Reading on the Go'''<ref>Roy Shilkrot, J. H., Connie K. Liu, P. M., & Nanayakkara, S. (2014). FingerReader: A
Wearable Device to Support Text-Reading on the Go. In Proceedings of CHI ’14 Extended Abstracts on Human Factors in Computing Systems(Vi). doi: 10.1145/2559206.2581220</ref><br>
The article describes a finger worn device with a camera which allows the user to skim through a page. This camera records the text and sends the images to a computer chip to convert the text into speech. It uses fibration to correct the users movement.

'''Character recognition — A review'''<ref>Govindan, V. K., & Shivaprasad, A. P. (1990). Character recognition — A review. Pattern Recognition, 23 (7), 671–683. Retrieved from http://www.sciencedirect.com/science/article/pii/003132039090091X doi: 10.1016/0031-3203(90)90091-X</ref><br>
This article elaborates on some of the OCR techniques. Due to the age it might not be the most relevant information.

'''Design and Implementation of OCR to identify English Characters and Numbers'''<ref>Adhvaryu, R., Parikh, R., & Vora, K. (2018). Design and Implementation of OCR to
identify English Characters and Numbers. , 4 (2), 57–62.</ref><br>
This article describes the matrix matching method of OCR. According to the article there are two distinct methods for OCR: Matrix Matching in which the system compares the scanned characters with the library character matrices. This system works best when the characters to be scanned and the library characters have very little or no variation in style. Feature Extraction: Feature Extraction generally deals with the features of characters like shape, closed areas, diagonal lines, line interaction and curves. It is more effective and flexible methods as it has a wide scope to identify the same character with different shapes and dimensions. It also gives a short description of how a Matrix Matching OCR is build up.

'''Display of virtual braille dots by lateral skin deformation: feasibility study'''<ref>Lévesque, V., Pasquero, J., Hayward, V., & Legault, M. (2005a). Display of virtual braille
dots by lateral skin deformation: feasibility study. ACM Transactions on Applied Perception, 2 (2), 132–149. Retrieved from http://portal.acm.org/citation.cfm?doid=1060581.1060587 doi: 10.1145/1060581.1060587</ref><br>
This article describes an attempt to make a display for blind people in braille. It explains how lateral skin deformation works and how it can be applied to a keyboard.It gives a good guideline on how certain things work and how they can be used to benefit reading braille.

'''Display of Virtual Braille Dots by Lateral Skin Deformation: A Pilot Study.'''<ref>Pasquero, Jerome. Levesque, V. (2004). Display of Virtual Braille Dots by Lateral Skin
Deformation: A Pilot Study. Proceedings of Eurohaptics, 96–103.</ref><br>
This article addresses the differences, advantages and disadvantages between VR information delivery and normal plain paper braille. While Digital or VR based reading has its downsides its positive impacts on the process of blind people reading is uncontested. 

'''V-braille: haptic braille perception using atouch-screen and vibration on mobile phones'''<ref>Jayant, C., Acuario, C., Johnson, W., Hollier, J., & Ladner, R. (2010). V-braille: haptic braille perception using atouch-screen and vibration on mobile phones. Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility - ASSETS ’10 (January), 295. Retrieved from http://portal.acm.org/citation.cfm?doid=1878803.1878878 doi: 10.1145/1878803.1878878</ref><br>
This article is about a electronic based reader for blind people. Instead a device that is mechanical they use a display based on touch as well as vibration from the screen of a mobile phone for example. It's an interesting idea.

'''Slide Rule: Making mobile touch screens accessible to blind people using multi-touch interaction techniques'''<ref>Kane, S. K., Bigham, J. P., & Wobbrock, J. O. (2008). Slide Rule: Making mobile touch screens accessible to blind people using multi-touch interaction techniques. Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility (Assets ’08), 73–80. Retrieved from http://dl.acm.org/citation.cfm?id=1414487%5Cnhttp://dl.acm.org/citation.cfm?id=1414471.1414487 doi: 10.1145/1414471.1414487</ref><br>
This article is a bit different because it addresses an innovative way to interact via touch screen with a device. It explains about a new touch screen technologies that allows for simple movements of the fingers (hand gestures done on the screen) to be used to navigate menus. This i feel could be very useful considering the implementation of buttons is not something convenient when making devices for blind people. This could serve as the way we control the device.

'''MobiReader : A Wearable , Assistive Smartphone Peripheral for Reading Text'''<ref>Polanco II, M. (2015). MobiReader : A Wearable , Assistive Smartphone Peripheral for
Reading Text. MASSACHUSETTS INSTITUTE OF TECHNOLOGY June.</ref><br>
An article about a device called the MobiReader, a mobile device to assist blind people in reading. It is a camera and text to speech technology. While we aim for text to braille this article can help with the input of our device. The camera conversion system is something we could find helpful.

= Planning =

Below is a shown the planning. If a cell is colored green, it means that the task presented in the same row should be done in the week of the same column. In red are shown the milestones.

[[File:PlanningGroup2q1.PNG]]

= Research =

== Scanning technique ==

The device must have an effective way to scan or to photograph documents in order to be further processed. The images taken have to fulfill certain requirements if the OCR has to recognize the characters. A scan or photograph of a very high resolution would result in good recognition and an reliable conversion but it would also result in low processing speeds. It would also require a very good memory card to store the information. On the other hand if the scan or photograph is to low in resolution the OCR wont recognize the characters. It was found that the minimal amount of dpi (dots per inch) should allow the smallest feature, such as the dot on the i, to still contain at least 4 dots. This fact can be used to find the camera or scanner requirements but more on the way to scan has to be found. A lot of scanners have a lot of movable parts. These parts are very vulnerable if the device is shaken to much. This brought up the idea to not have something internally move, but allow the user to move the device to scan the paper. After some internet searches this type of technology proved widely available.

== Interview with Visio == 

We wanted to get a better impression of what our users would require and would appreciate from the sort of device we are aiming to make. We therefore contacted Visio, an organization which specializes in assisting blind or visually impaired people with performing their dialy tasks, and tries to make their lives as comfortable as possible. They provide advice and assistance to blind people, as well as develop new technologies which aim to imrpove the quality of life for blind people.<br> 

We sent an email to Visio expressing our interest in visiting their location in Eindhoven. We got a reply from Erik van Dijk, an employee of Visio, that it would be preferable for them if we first called. We did so, and what followed was a conversation of about 20 minutes in which Erik explained to us that what we were trying to do has already been done in quite a sophisticated manner, and is already being used by blind people, as well as being covered by insurance.<br>

He named a couple of products which were not exactly like our idea, but did more or less covered its entire range of functions, while also providing extra applications which make this device a lot more likely to see practical use than our own. One of such devices is called Optalec EasyLink12 Touch (see http://nl.optelec.com/producten/el000003-easylink12-touch.html), and can be fully integrated with a smartphone. It can present text from a phone in braille, or it can read it out loud depending on the preferences of the user. If someone were to take a picture of a piece of text with their phone, use OCR software on to convert the text in the image to characters that the phone can recognize, this device can do exactly the same as our own device.<br>

Erik went on further to explain to us the current challenges in developing technology for blind people. These included videogames for blind children and exploiting the possibilities of smartphones to increase the ease with which blind people could go shopping in a supermarket, as this, explained Erik, is currently quite a nightmare for them. He mentioned SeeingAI, VisionAI, iFarkle and Amazon Go as current points of interest.<br>

Following this interview, we decided to drop our current idea. The chances that we could make any significant improvement to the device that is currently used are slim, since it is already highly advanced and would require a group of specialists to make small improvements.

== Application development and Software ==

After extensive research on how Braille is written and the syntax and logic when translating from normal text to braille we came to the conclusion that the software of the device is fairly sophisticated to make from scratch. It requires a vast amount of pre- and post-checks around surrounding works in order to make the proper translation. In some cases in braille, certain words chains that are often used are shortened with their own symbol, but in other cases the whole word chain is translated in its full form. This created the need of a large database of word checks when translating.

If we want to have a high quality system that translates accurately we would not have enough time for the creation of such a program, given the time restrictions of the course and the required knowledge we would need to obtain.

There are a number of open source codes about translation from normal text to braille which can be used, however it is important to note that they are not very sophisticated in what they can translate and are also prone to making mistakes in the translation. It is important to note that slangs and specific speech patterns that can be found in texts, such as heavy dialects or languages (pirate slang, Scottish, Irish, Australian, different Chinese dialects etc.), are even harder to make translatable in braille, because they require their own pre- and post-checks that are very hard to make, as there are words and even whole sentences with no logical translation to braille.

Following the interview we decided to scrap the whole idea in general and focus something we can have a positive impact on, seeing that making an innovative improvement is going to be very hard for this topic.

== Braille display ==

In the research for the display multiple ways to portray the Braille were encountered. First of let's look at a purely mechanical way to display the Braille. The issue with this method is that to control every pin individually a big mechanical system is needed. This increases the thickness of the device, making it impractical to use, therefore this is not the best solution. A magnetic system could be a possible solution. The biggest benefit from this is that it is easy to understand and could be implemented by us. Another possible solution is the system that is discussed in 'Computerized microfluidic cell culture using elastomeric channels and Braille displays'. This system of computer-controlled microfluidic cells is a complicated system and more research needs to be done on this to understand it better and if a patent of this system exists. The conclusions from this research will also determine what system would be best for the device. Besides this the way to portray the Braille itself in the system also needs to be researched. This can be done with either simple pins or possible other ways.

= Solution = 
The core of the concept consists of the following subsystems:

* A scanner that directly contacts the surface which contains the text to be scanned.
* A small (single-board) computer that can run OCR software and control the physical braille pins.
* A refresh-able array of braille pins to present the scanned text to the user.

A first sketch of the device can be seen in figure 1 and figure 2.<br>

<div style="display: inline; width: 1300px; float: right;">
[[File:Braille_V1_A.png|600px|thumb|left|Figure 1 A first sketch of the concept showing two buttons on the bottom of the device to scroll through the text.]]
[[File:Braille_V1_B.png|600px|thumb|right|Figure 2 A first sketch of the concept showing an on/off button on the top of the device.]]
</div>
<br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>

There are two dark grey buttons on the bottom on the device used to flip through the pages of the scanned text. This is necessary since not very much text can be displayed on the screen at once and it also allows one to re-read some text that was scanned already. The top of the device contains a on/off button for the device. The text displayed on the device in figure 1 and figure 2 is actually the problem statement of this wiki in braille.

= Discussion = 

= Conclusion = 

=References=
<references />