PRE2020 4 Group3

Group members

Name	Student number
Eline Boom	1465872
Luuk van Dorst	1469789
Robin van de Hoef	1479679
Kyra Moviat	1349171
Jeroen Pullen	1477730
Veerle Uhl	1462229

Abstract

With the rise of technology comes new ways to implement technology into our daily lives. One of these implementations is technology in education. More specifically, we wanted to develop a technological tool that will help young children in their learning process of number sense. Your knowledge of counting and quantities is tested daily throughout your life. Not a day goes by where you do not have to count or decide the quantity of something, which it is such an important skill to learn from an early age. We wanted to make an application that teaches children number sense in a fun and easy way.

Introduction

Teaching children to count is very important. It helps with their development as well as being used in day-to-day life later on. The current curriculum used in most schools works well, but we feel that implementing technology into this curriculum would improve the learning of small children, both in school and at home. According to Hsin(bron), who studied the influence of children's use of technology on their learning, this technology does have a positive effect on learning. Hsin does provide us with a few conditions in which learning with technology is optimal. When children are allowed to work together and communicate with each other, technologies may not only improve knowledge, but social skills as well. These skills are most important for children, since interaction with others is needed for humans to survive. The second condition that needs to be met, is that adults need to be present to provide a safe climate and to encourage participation. This makes sure children stay motivated to learn and the adult can give guidance when instructions might be unclear. Our product is heavily influenced by these two conditions. What we designed is a web-based application that teaches children how to count and recognize shapes and color. It is targeted at children aged 3-5 that are at an average level of counting and shape / color recognition. It has a level-based design with three main levels. These levels then have several exercises each. The difficulty increases for each level, but stays the same within levels. OUr product is designed to be used in a classroom setting. It can be used on a whiteboard with a stylus or on a computer or laptop. When it is used on a whiteboard, children can interact and discuss with each other about the correct answer to the questions. This way, interaction and collaboration is promoted. When it is used in a classroom setting, the teacher can give extra help and information where needed, although the product is designed to need very little to none additional explanation.

Project Plan

Subjects: Child psychology, learning, image processing, software development.

Objectives: Create software to help children learn

Users: Children (of parents that want to provide a playful learning app)

State-of-the art: Alternative apps that help children either count, draw or learn colors individually. Not in a combined way, we feel the combination might improve the way to learn these things. Additionally, we hypothesize that explaining problems via a childs own drawn elements could improve their understanding.

Approach: Make up a model for how the software should work, make software in which one can draw, program software that can recognize multiple drawn objects/shapes, implement everything in a working framework according to the model.

Milestones: Drawing software, working framework, Working object recognition, implementation in task framework

Deliverables: Peer review, Wiki, Final Presentation, The software (?)

Planning

Task	Week 1	Week 2	Week 3	Week 4	Week 5	Week 6	Week 7	Week 8
1. Brainstorm Project ideas	X
2. Investigate Literature		X	X
3. Create Software Model		X	X
4. Create Software Framework			X	X
5. Create Drawing Software			X	X
6. Create Object Recognition Software					X	X
7. Implement Recognition Software in Framework							X	X
8. Work on Wiki	X	X	X	X	X	X	X	X
9. Work on Presentation						X	X	X

Subtasks

subtask	Done / to do by who
1.1 Find topics that are of interest and comply with technical skills of the group	Everyone
2.1 Investigate state of the art	Kyra
2.2 Investigate how children learn to count and learn colors and shapes	Veerle/Robin
2.3 Investigate current models or curricula that are used for learning	Veerle
2.4 Investigate the influence of online or "games" learning	Veerle/Robin
3.1 Meet with stakeholders to see what they want to see in an excercise	Jeroen
3.2 Figure out what kind of levels we want to make	Jeroen
3.3 Implement levels in the website	Jeroen / Eline
3.4 Make an instruction video for the levels	Kyra / Eline
4.1 Set up a website	Eline
4.2 Code and make a website layout, make it suitable for young children	Eline
5.1 Set up a canvas where users can draw	Eline
5.2 Find a way to use colors, a drawing tool and an eraser in the canvas	Eline
6.1 Find databases to train a neural network with	Luuk
6.2 Write code for a neural network	Luuk
6.3 Write code that can read a database and train a neural network	Luuk
6.4 Work on improving the accuracy of the neural network	Luuk
7.1 Make the drawing made convert into a png	Eline
7.2 Make the neural network compatible with a png	Luuk
7.3 Convert the drawing png so it cuts out every individual shape and recognizes color	Robin
8.1 Write down as much as we can on the wiki	Everyone
8.2 Transform the summary on the wiki into a good text with logical structure	-

Research

State Of The Art

There are already a lot of apps on the market that are geared towards educating children. Certainly apps that teach children how to count or how to draw separately. Combining the two is also done, but less frequently. One app and one website that combine learning to count and to draw are found on the internet. The first one is an app made by Bini Bambini and is called “123 tekenen tellen van nummers voor kinderen”. This game gives children very simple descriptions along with visual guides for drawing complex shapes. It then tells the child how many objects there are and lets the child draw the number. It does not have any AI to see what the user draws, it only gives an outline and a field in which to draw. Once the user stops drawing for a few seconds, the app goes on to the next step. The second one, a website, is called Kids Academy. This has the same general layout of the bambini app, only the Kids Academy focuses on more than just learning math. What stands out quite prominently in both apps is the design layout. It is clearly targeted towards kids with simple shapes, bright colors and audio directions instead of text. Our product keeps these general GUI attributes, but it is different in the level design. We combine drawing and counting in a different way. First, we start off with a level where children do not even need to draw and just need to recognize amounts, colors and shapes. Then, children need to draw (simple) shapes themselves, without guidelines, and also need to draw the correct amount eventually. This makes it so the child is more actively working on the amounts of objects, which will promote their learning.

How children learn

In order to develop an application that supports the learning of children aged 3-5, we first need to know how children's learning develops during their early years and what knowledge they already possess or still need to learn. From when they are born to around the age of 5, children develop “everyday mathematics”. These are informal ideas of more and less, taking away, shapes and sizes. These everyday mathematics can be surprisingly broad, complex and sophisticated. Apart from the everyday mathematics that children learn on their own, they also start learning mathematics in school. From the age of 2, children learn the language and grammar of counting. They memorize the first ten or so counting words, and then learn a set of rules to generate the higher numbers. When children are around 4 or 5 years old, they begin to develop metacognitive skills: They become increasingly aware of their own thinking and begin to express it in words. The hardest form of language for children to learn, which they thus only learn at a later date, is the special written symbolism of mathematics, like 5, +, - or =.

Understanding numbers involves more than saying a few counting words. It involves reasoning about numbers, making inferences and developing a mental number line. Children also need to mathematize: to conceive of problems in explicitly mathematical terms. They need to understand that the action of combining one bear with two others can be meaningfully interpreted in the terms of the mathematical principles of addition and the symbolism 1 + 2. This mathematizing is not very important for our product, as this only happens at a later age than the target group we are designing for. This means that we do not yet have to include problems of mathematization in our level design

The Number Worlds Curriculum

This curriculum is one of many out there that learns children what numbers are and how to use them. It provides a good understanding of the process a child goes to in order to learn how to count. Looking at 5-year-olds, there are several points of knowledge they need to know, relating to number sense. Five year olds know:

• that numbers indicate quantity and therefore, that numbers themselves have magnitude
• that the word “bigger” or “more” is sensible in this context
• that the numbers 7 and 9, like every other number from 1 to 10, occupy fixed positions in the counting sequence
• that 7 comes before 9 when you are counting up
• that numbers that come later in the sequence
• that are higher up - indicate larger quantities and therefore, that 9 is bigger (or more) than 7.
• that each counting number up in the sequence corresponds precisely to an increase of one unit in the size of a set. This enables children to use the counting numbers alone, without the need for real objects, to solve quantitative problems involving the joining of two sets. In doing so, it transforms mathematics from something that can only be done out there (e.g. by manipulating real objects) to something that can be done in their own heads, and under their own control.

As children get older, they progress through different knowledge they know: By the age of 4, children have constructed knowledge of counting and quantity. Sometime in kindergarten, children become able to integrate these knowledge networks. Around the age of 6-7, children connect this integrated knowledge network to the world of formal symbols. By the age of 8 or 9, most children become capable of expanding this knowledge network to deal with double-digit numbers and the base-ten system.

In the number worlds curriculum, there are 5 instructional principles that lie at the heart of the program:

1. Build upon children's current knowledge

2. Follow the natural developmental progression when selecting new knowledge to be taught

3. Teach computational fluency as well as conceptual understanding

4. Provide plenty of opportunity for hands-on exploration, problem-solving and communication

5. expose children to the major ways number is represented and talked about in developed societies

Lastly, the core of number sense forms a knowledge network that is called the central conceptual structure for number. This core is important for children for two reasons: 1) it enables children to make sense of quantitative problems across context, 2) it provides the foundation on which children's learning of more complex number concepts is built.

Stakeholders And OSL

3-4 years old: Easy levels, can count to 10, can manage color comparison, learning pen grip important, learn to count via touch, “which object is blue”? appropriate question, combination of tasks as described in goal level still to hard.

4-5 yeas old: Goal level, know all colors, can count, learn shapes important, number recognition important.

*What makes the software good?

- Rising level of difficulty (3-4 to 4-5 level ?)

- Possible on digital school board

- Possible with pen (learn pen grip)

- Returns performance data to authorized figure

- Test mode

- Feels like a game (reward system)

*What can kids learn from the software?

- Colors, Shapes, Counting & Drawing

- Pen grip

- Vocabulary (if drawing objects)

- Listening skills (if tasks given in audio)

- Number recognition (number on screen)

*Useful purposes

- Being able to use the software collaboratively (class setting, performing tasks together)

- Being able to use the software to test how far children are.

- Being able to have the kid perform tasks on their own, behind computer or tablet.

How We Use The Research

As a basis, we will use Number worlds curriculum and OSL curriculum. These have been tested and used by many schools and education institutes around the world and are proven to support learning of children.

Number words has 5 principles for learning, we used them as a framework to develop good exercises that cater to the needs of the children that want to learn. Principle 1 talks about building upon children’s current knowledge. We do this by slowly incrementing the difficulty of our levels. First, we ask children to simply move an object into a box, then when they have mastered this, they need to move a specific amount or specific color or shape of object into a box. This way, we slowly add more variables while building onto the knowledge children already have. Principle 2 is about following the natural developmental progression when selecting new knowledge to be taught. With this, we looked at what knowledge young children typically learn at what age. We also took into account the desired age for users and combined this to find what lessons work best. For example, we use squares and circles for objects instead of cars and dogs, since the latter is too far ahead in the developmental progress of knowledge for the small children. Principle 4 talks about providing opportunity for hands- on exploration and communication. We hope to do this by making our product work as well on a digiboard. This way, children and teachers can work together to solve problems and do exercises, promoting communication between them.

The OSL curriculum is one that does not just focus on math, or on younger children. It focuses on all subjects and children of all ages. For our application, we look at their learning materials for young children (ages 4-7) and specifically the math and number sense material. SLO works with "inhoudslijnen" that give an overview of everything the young child (until age 7) needs to be able to do and what knowledge they need to posess. Even though our app focuses on children ages 3-5, we can still use these "inhoudslijnen" to shape our excercises. Learning never stops, and everything we teach children at a young age, they do not need to learn at a higher age still. If we for example look at some of the learning goals of SLO, we can see that, at age 7, they want children to learn how to count to 20 and to also be able to count quantities of at least 20. At age 4 or 5, children using our app probably would not already be able to have knowledge on all these topics, as they are aimed at children aged 7, but we can give them a good push in the right direction. Since we know that children aged 7 need to know at least how to count to 20, then children aged 5 should not only be able to count to 5, but probably to 10 already. In conclusion, using the learning goals of SLO, we can modify them to set our own learning goals for users of the application. Based on these learning goals, we can then shape the levels of the game to teach the children just what we want to know.

OSL has learning goals for children. The learning goals important for our level design are on the drive file. Here is a summary of important findings: What we have now is doable for children aged 3-5 yeras old. What we might need to test is that it will not be too easy for 5 year olds, seeing as according to OSL, at that age they already learn additional things as well. What we have now as exercises follows the learning lines of children and will support their learning process. Nothing from the levels is too difficult, so it wont be the case that the level difficulty is above the level of the users. Related to language (listening/reading): reading is indeed too difficult for younger children. When you look at phase 1 of OSL, you can see it is mostly simple words and very very short sentences as well as looking at pictures. There is not a whole lot about listening, but according to the prephase (for ages 2-4) then children should be able to listen to and interpret explenations of exercises. We do have to keep in mind to only use very simple words

Software Model

Level Design

We separate two age groups, 3-4 years and 4-5 years old.

Within these groups, task complexity increases by means of two variables 'Level' and 'Difficulty'.

After completing age group 3-4, the child can be tested for the age group 4-5.

Level

The age group 3-4 will have 6 levels where the goal is to move certain figure(s) to the middle of the screen.

The age group 4-5 will have 6 levels where the goal is to draw certain figure(s) on the screen.

Levels age 3-4:

1 - All the same shapes, child must move 1 correct color

2 - All the same color, child must move 1 correct shape

3 - Altering shapes and colors, child must move 1 correct figure with corresponding color & shape

4 - All the same shapes, child must move # figures with corresponding color

5 - All the same color, child must move # figures with corresponding shape

6 - Altering shapes and colors, Child must move # figures with corresponding color & shape

Levels age 4-5

...

Difficulty

The child will have to succesfully move through all difficulties of a level before moving on to the next level.

The amount of difficulty has influence on the range of colors and amount of shapes that are available in the exercize.

The difficulty will go up or down depending on the childs success rate. This is an example of how a child can move trough the stages:

1.1 -> succes // 1.2 -> success // 1.3 -> failed // 1.2 -> failed // 1.1 -> success // 1.2 -> success // 1.3 -> success // 1.4 -> failed // 1.3 -> success // 1.4 -> success //

Level complete, move on to level 2.1

Whether the difficulty should go up or down after 1, 2 or 3 successive performances is yet to be determined.

Image Recognition

In order to recognise images drawn by children we have decided to use a neural network. In order to train this network we have settled upon a couple of usefull databases that contain drawings and letters, I will list the found databases below:

-MNIST dataset (http://yann.lecun.com/exdb/mnist/)

-Google quick draw dataset (https://github.com/googlecreativelab/quickdraw-dataset)

-(One other dataset that I decided to use but can't find the source for anymore as of this moment :( )

We have also created some code which can read a database, build a neural network, train a network and test the resulting network. However thus far the results from the neural network have been dissapointing, in total the succes rate of the neural network has been 0.0036 which is worse than random guessing (with 250 labels that would be 1/250 = 0.004) so optimisation is still necessary. The low accuracy rate can either be the result of using a too small dataset (each of the 250 labels have only 50 or so images in them) or because of wrong parameters used in the neural network or because we have not trained it long enough. Our current guess is that it's a mix of the first two and we should use less labels and more images to increase accuracy and add more internal nodes and layers to our neural network.

In order to solve these problems and move to a succesfull working prototype I suggest the following goals for next week:

1. Fix the accuracy rate and try to get a neural network to at least work with one of these datasets.

2. Try to find a way to extract the complete neural network from our code and find a way to implement it on our website.

3. Decide which labels are usefull and chose a final dataset which we train our network on.

Of these three the biggest challenge will probably be problem 1 since training the network takes a lot of time so we cannot really bruteforce our way to succes which means we should probably look at what previous work has been done in this subject.

---

UPDATE 16/06

The code is now adjusted so that it works with the google draw dataset. The reason for the switch to the google draw set is because it has a better support for simple shapes like squares and circles which we have decided to use for our final product. The downside of this is that the google dataset came in 4 different formats each of which has a different amount of preprocessing done to it which made it hard to make a informed decision about it. To make matters worse each dataset is stored in a different format which means that the part of the code that reads the input needs to be updated if we decide to switch again in the future. So I took some time to research the pros and cons of each dataset and I decided upon the 38x38 grayscale bitmap version of the original images. The reason for this choice is that the simple shapes will hopefully be easy to recognize even with this heavy downscaling and that the low image resolution will resolution in shorter training times and a less complex neural network. The sad part is that this dataset came in the .npy format which meant that it was completely incompatibly with 50% of the code that I already wrote which made me have to update that part again which took quite some time.

Platform

Designing the website

Link to website: https://group3use.glitch.me/

Link to code: https://glitch.com/edit/#!/group3use

Storyline throughout the levels

To make completing all the levels more appealing to children, a storyline will be shown through visuals in the levels and animated videos before them. The subject of helping animals was chosen because we have observed that it is a timeless subject. Things like tv-shows and apps geared towards children use animals very often.

The overarching storyline is about a group of animals (a horse, a monkey, a bunny and a tiger) that is moving from the city to the forest. The levels designed for 3-4 year olds are preceded by a video about the animals that explains the move and tells the children that they need help moving because there is a broad river around the forest without bridges. The levels itself are about building a bridge for each animal by moving the right shapes into the water. Upon completing the level the children are rewarded with a visual of the animal crossing the bridge, if possible.

The levels designed for 4-5 year olds will depend on the development of the program. If the levels will be randomized the storyline will continue with the animals wanting to decorate their houses with colorful shapes. If the levels will not be random the storyline will continue with the animals having to find food in the forest. The shapes the children draw will transform into food if they complete the level correctly.

Animating the videos

All the videos needed for the reward system of the drag and drop levels have been animated and added to the site. PowerPoint was used to make the animation. The storyline has been changed and is still changing to fit the game as it is now. Since we are still developing it, the story is also still changing and evolving.

Time Log

Eline Boom

Date	Description	Hours
22/04/21	Brainstorm meeting	1
29/04/21	Meeting	0.5
29/04/21	Setting up the website	3
30/04/21	Design of website	2
03/05/21	Meeting	1
03/05/21	Creating canvas function on website	1
03/05/21	Converting canvas to image	2
03/05/21	Colour pens and eraser	3
06/05/21	Colour pens and eraser	2
06/05/21	Switching between pen colours through button	2
06/05/21	switching between draw and erase through button	4
13/05/21	Researching possible ways to code Jeroen's game ideas	2
13/05/21	Coding different geometric figures in css	0.5
14/05/21	Start working on the actual game	2
17/05/21	Difficulty level 3, randomization of figures	3
19/05/21	Finishing all level difficulties + reward when correct	7
20/05/21	working together with kyra on animations	8
	Total Hours:	44

Luuk van Dorst

Date	Description	Hours
22/04/21	Brainstorm meeting	1
29/04/21	Meeting	0.5
29/04/21	Searching for viable datasets	2
30/04/21	Researching neural networks	3
02/05/21	Making a prototype neural network	4
03/05/21	Meeting	1
08/05/21	install and get tenserflow working (took longer than expected)	1
08/05/21	Found some additional datasets	1
08/05/21	Finished neural network code	4
09/05/21	Did some research into how much nodes the neural network should have to increase performance	2
09/05/21	Tested some parameters to increase performance	4
10/05/21	edited wiki	1
14/05/21	Did a couple of tensorflow tutorials to understand the library better	2
14/05/21	Modified neural network to work with a different dataset	3
15/05/21	Adjusted parameters and tested the neural network to see if performance improved	5
16/05/21	Did some research to see how performance can be improved in the neural network	4
16/05/21	Edited the wiki	1
	Total Hours:	10.5 + 12 + 15 = 37.5

Kyra Moviat

Date	Description	Hours
22/04/21	Research topics	5
22/04/21	Brainstorm meeting	1
29/04/21	Meeting	0.5
02/05/21	Add tables to wiki and update layout	1.5
02/05/21	Research state of the art	1.5
03/05/21	Meeting	1
09/05/21	Writing about state of the art	1.5
09/05/21	Updating wiki	0.5
09/05/21	Researching methods children learn counting	2
10/05/21	Meeting	1
11/05/21	Updating the wiki to only have 1 logging system instead of 2	0.5
11/05/21	Working on script videos that will accompany the levels	2
11/05/21	Thinking of ideas for name of the app	0.5
12/05/21	Working on script and asking for feedback on it	1
12/05/21	Updating wiki	1
17/05/21	Meeting	1
20/05/21	Working together with Eline, animating videos to fit the levels	8
20/05/21	Updating wiki	1
22/05/21	Animating introduction video drag and drop levels	8
	Total Hours:	38.5

Robin van de Hoef

Date	Description	Hours
22/04/21	Brainstorm meeting	1
29/04/21	Meeting	0.5
03/05/21	Meeting	1
07/05/21	Searching for suitable literature	2
09/05/21	Searching for and researching literature	4
10/05/21	Meeting	1
14/05/21	Installing and testing python image libraries	1
15/05/21	Color recognition	2
16/05/21	Image separation	4
10	Increasing code performance	2
11	.
	Total Hours:	x

Jeroen Pullen

Date	Description	Hours
21/04/21	Research project ideas	1
22/04/21	Brainstorm meeting	1
26/04/21	Prepare poject planning	1
27/04/21	Research state of the art	1
28/04/21	Updating Wiki	1
29/04/21	Contacting stakeholders	1
29/04/21	Meeting	0.5
29/04/21	Preparing stakeholder questions	1
30/04/21	Interview first stakeholder	1.5
03/05/21	Meeting	1
03/05/21	Prepare second stakeholder meeting	1
03/05/21	Interview second stakeholder	1.5
04/05/21	Work out model using interview data	2
05/05/21	Work out model by researching learning paths	3
05/05/21	Work on Wiki	1
08/05/21	Work on Wiki	1
09/05/21	Work out complete model based on combined information	2
10/05/21	Meeting	1
11/05/21	Create Level Generator	3
12/05/21	Create Level Generator	2
13/05/21	Create Level Generator	3
16/05/21	Discuss Level Design with new stakeholder	1
16/05/21	Redefine level design according to advice & SLO	2
17/05/21	Finalize Level Generator according to redefined levels	3
19/05/21	Watch HTML tutorials	2
20/05/21	Watch Python to HTML tutorials	2
22/05/21	Turn Python Level Generator into Javascript	2
22/05/21	Update wiki on level design	1
	Total Hours:	43.5

Veerle Uhl

Date	Description	Hours
22/04/21	Brainstorm meeting	1
29/04/21	Meeting	1
29/04/21	Literary research	1.5
30/04/21	Literary research	1
03/05/21	Meeting	1
03/05/21	Literary research on how children learn	2
09/05/21	Literary research on the Number World model	2
09/05/21	Work on wiki, update the research and timetables	1
10/05/21	Meeting	1
10/05/21	Write a more detailed planning on the wiki	1
12/05/21	Write something about the use of research	1.5
16/05/21	Write something about our use of OSL and number worlds	2
17/05/21	Meeting	1
17/05/21	Looking into "inhoudslijnen" of OSL and level design	2
19/05/21	Editing wiki texts	5
19/05/21	writing wiki texts: level and website design	2
	Total Hours:	25