Verslag group 14

Return PRE2017 3 Groep14

Introduction

Slim In Rekenen

For the course Robots Everywhere from the Technical University of Eindhoven, a project about some form of robotics needed to be made. The only explanation given was that robotics has to be part of the project in some way and that a product needed to be delivered at the end. Because of the huge amount of freedom in this course, for this particular project it was decided to create a smart quizzing program for in class. The program can be used in groups 3 and 4 from schools with a traditional educational tactics, all research is focused on these schools in the Netherlands. Simple mathematics is the subject that this quiz will cover, so addition, subtraction, multiplying, and division. The purpose of the program is easily identifying students that need some extra teacher attention, while stimulating all students on an individual level. The way in which this is done is by finding the child’s knowledge boundary based on his or her answers to the previous questions. This way, the child will get questions that are still challenging enough to be interesting, but not too hard that the pupil will give up. As a quizzing program will be started at the same time for a whole class, but all pupils soon have an individual test, it is a perfect combination of individualism while working as a class so nobody feels left out. All questions in the test are multiple choice to make it easier for the children to input their answers. The teachers will be able to see from the results which children have mastered the subjects sufficiently and are ahead of the program, but also which pupils lack behind. Because these results will be visible very quickly, the teacher knows sooner which children will need some extra attention to catch up. This extra attention can then be given when the teacher sees fit. To test whether or not teachers think the program could make an actual difference in their teaching programs, an questionnaire was send to 50? elementary school teachers. The questionnaire focused on their opinions and whether they had tips to keep in mind while designing the program. An important question is whether or not they think the program will help them plan their time better, as this is our main goal?. The following report is an elaborate documentation about the program, the questionnaire and the results from this.

Problem Description

The purpose of our program is to quickly recognize which children need extra help on a specific topic and to customize the test to each child, such that they can train specific topics on their own level. Quickly recoginizing the children will help the teacher to immediately step in. For example, at the start of the year teachers do not know the weaknesses of the children, but our program can find them immediately after 1 test. Furthermore the personalised test ensures that the children will not lose interest, due to the questions being to easy, and ensures that they will not feel left out, due to always getting the questions wrong. These personalised questions are determined on the answers of previous questions.

Our program is now focussed on children in the third and fourth grade of the elementary school in the Netherlands using a tradional education system. The test consists of mathematical questions, that are divided by subject, for example multiplication tables or adding numbers up to 20. These choices were made to ensure an easy starting point for the program.

Objective

Develop a smart quiz program for a computer that can assess knowledge levels of the pupils, to give the teacher a better view on his pupils, and ask questions to the pupils on their personal boundary, so they learn effectively.

Users

The users of the program will be the pupils of the third and fourth grade of the elementary school in the Netherlands using a tradional education system, the parents and teacher of these pupils. The users will have the following requirements:

• The pupils want:
  • A program that stimulates their imagination
  • A test that remains interesting throughout the whole session
  • A test that does not make them feel dumb or left out

• The parents want:
  • A test that stimulates their children to learn and to perform the best they can.
  • Their children to be happy
  • Check their children's progress
  • The privacy of their children to be guaranteed

• The teacher wants:
  • A test that motivates the pupils to learn and to perform the best they can
  • The system to match the curriculum, meaning the teacher has to change the settings of each test
  • To check the progress of each pupil in an easy way
  • To divide his or her time in the best way over all the pupils

For the program we did not put our focus on privacy, because this is only a small part of our program. The program will not contain an interface for the parents, due to fact that the goal only incorporates the pupils and the teachers. However this could be added later on if desired. Almost all of the other the requirements will be implemented in our program by the use of the traditional didactic system.

Intelligent Quiz System

Given a set of questions, the quiz master will test the knowledge of a child, and help the child improve by asking the right questions at the right time. We will build an application that selects the next question to ask the child, based on the previous answers the child gave to previous questions. The quiz will find the knowledge level of the child and ask questions at the child's knowledge boundary so he can still learn from the question but will not be overwhelmed.

The quiz master has to:

• Find out the level of knowledge the child has, and ask questions that are on the 'edge' of a childs knowledge in order to improve their knowledge.
• Generate new questions.

In order to do so, we must:

• Define distance (or question similarity) between questions, which questions are of similar difficulty. So cluster questions based on their difficulty. Note that this will vary per child.
• Simulate the (increasing/decreasing) knowledge of different children. (To be able to train our app.)
• Construct a (large enough) data set to use parts of it for training and validation.
• Find out what the next right question would be. Our app should do this, based on the question similarity for a certain child.

Desk Research

Didactic Systems

In the book, het didactische werkvormenboek , a few important aspects of teaching are found.

• The best way for pupils to take in information is when they read and/or see it.
• When asking questions, there need to be both open questions and closed questions.
• Make sure pupils work with the whole class, but also occasionally in smaller groups or on their own.
• Switching between teaching tactics stimulates learning. An example could be switching between a game and simple sums.

For our project we chose to focus on a traditional education system instead of a constructive education system. This means that we will keep in mind that

• The teacher largely decided on the curriculum and order in which subjects are taught.
• The focus lies on teaching the whole class at once.
• There is a curriculum in which some subjects are more central than others.
• Learning is an individual activity.
• Pupil’s progress is checked by means of tests.

The Use of Computer Programs in Dutch Education

From a government study , the following things became clear:

• Separating the class in smaller groups in year 3 has a positive effect on the pupils’ performances as it causes more interaction.
• More than 90 percent of the teachers in primary education use computers to teach. According to them using ICT can contribute to a more efficient, effective and likeable education.

The next article shows us that education often does not seem to fit the natural, experimental learning process that children have. In games it is possible to naturally lean, which is why using games can be relevant for primary education. There would have to be a good balance between playing and effectively learning, to make sure that the advantages of learning in a fun way are not lost. This means the following things need to be taken into account when asking questions:

• The information should not be presented in a way that is too abstract, make sure it speaks to the children’s imagination.
• There needs to be a lot of repetition.
• Important information should be taught in more than one way.
• Take some time, it should not be taught too fast.
• Use realistic characters.
• Use animation, childlike dialogues, interaction and direct feedback.
• Do not punish children for making mistakes.

The following factors influence the quality of a course.

• From the international and Dutch literature it follows that all researched interventions that focused on the methods used during instructions and lectures were effective. It is nearly impossible to identify the elements that make a certain intervention work effectively, as several effective interventions might have opposite starting points. Furthermore there are not a lot studies that directly compare different methods. The resulting idea is that `it works to manipulate the methods used'. This is in line with the report of the KNAW (2009) that the Dutch research available does not give a clear view on the relation between the didactics and the skills when it comes to teaching arithmetics.
• Using technology for teaching arithmetics has a positive influence on the prestations of children, just as using real three dimensional objects. This follows from both international and Dutch research, for example the use of the programs Snappet and Rekentuin. It is not clear how much the use of the programs increased the available time to teach arithmetics, and thus this will need some further investigation.
• The use test results to improve the learning process has a positive relation with the arithmetic prestations. This can either be feedback to the teacher, for example by means of a digital system to track students, or direct feedback to the students per question that is done.
• Dividing students in different groups which each have their own level has a positive influence on the prestations of the individual students. Note that further investigation is required with respect to the implementation of this division. In The Netherlands, it is common to divide students in three different groups.

Individual or in the Class

• As we can see in the following article , the program is part of groupdifferentiation with individual learling lines. This means that there is quite some form of external regie but with focus on the individuals. Consequently, the teacher has some set goals and subjects on which the pupils need to work, and they will work on these subjects when the teacher tells them to. However, as the program will ask questions on their personal knowledge levels, the individuals are also taken into account. This is a perfect combination of working with the class while working on one’s own.

Working on Paper vs Working on an Electronic Device

• As indicated in the following atricle using computers instead of paper to administrate achievement tests in elementary schools is supported. Using computers has multiple advantages such as being able to use an adaptive technology, which is the goal for the finished program. On the other hand, it is possible that children experience computer anxiety resulting in lower achievements, but for this project it is assumed that this will not be the case as most children these days work with computers from very early on.

Articles About the Layout of the Test

How to implement feedback

• In this article it becomes clear that attributional feedback to children is useful as it is an effective way to promote rapid problem solving, self-efficacy, and achievement. This is probably because children have a sense of how well they are doing and attributional feedback helps to support these self-perceptions and validates their sense of efficacy. Because of this, the children will stay motivated to work on leading to a better performance in the end. The best way to do this is by giving them ability attributional feedback, so telling them they are either very smart, or not that good at math.
• Feedback in between The major findings evidenced in this research were the following:
  • Feedback was generally effective for learning, but more so on the lower level (identical) questions than on the higher level (reworded) ones;
  • Feedback information had greater impact in the absence of supporting text than with supporting text;

In this article they used several feedback methods such as knowledge of correct response (KCR), which identifies the correct response, and answer until correct (AUC). These two were shown to be the most effective.

When implementing these articles about feedback in our program the following decisions were made the KCR method with almost no supporting text except for the attributional feedback will be used.

How to Construct Arithmetical Questions

When constructing arithmetic questions it is important to know how students put numbers and operations on numbers into context. In the article it is explained what should be taken into consideration.

Mental Arithmetic

Mental arithmetic is insightfully doing calculations (arithmetic) with numbers, while the value of the numbers is kept in mind. While doing mental arithmetic one also makes use of ready knowledge, and properties of numbers, operations, and the underlying relations. There are three main methods for doing mental arithmetic:

• Step-by-step mental arithmetic: [math]\displaystyle{ 36 + 12 \rightarrow 36 + 10 \rightarrow 46 + 2 = 48 }[/math].
• Mental arithmetic by dividing the digits: [math]\displaystyle{ 36 + 12 \rightarrow 30 + 10 = 40 \rightarrow 6 + 2 = 8 \rightarrow 40 + 8 = 48 }[/math].
• Handy mental arithmetic, for example by compensation: [math]\displaystyle{ 73 + 29 \rightarrow 74 + 30 }[/math].

"Use of symbols in mathematical questions" In an article about the mathematical development of children it is mentioned that in order to get a correct numerical magnitude representation, children need to practice with symbolic numbers, so digits, but also with non-symbolic formats, like the herd questions in our program. Therefore we have to keep this in mind when implementing this in the layout of the questions our program.

From these articles we can conclude that we want to focus on learning in a likeable way, using a computer program. We will do this by asking simple maths questions using fun pictures of animals. This way we hope to intrinsically motivate the children to play with our program. Other things we will implement is that we want to make sure it will not go too fast and we can use child-friendly language.

Program

In this section, we describe the actual program that was developed. The overview of a typical test scenario is as follows. First, the teacher launches the teacher interface of the program and specifies the parameters of the current test, such as the types of operations used in the questions, the amount of time the students have to answer the questions, etc. Next, all pupils launch the pupil interface of the program and wait till a connection between the teacher- and pupil program is established. All pupils fill out their name, and are added to an overview presented in the teacher interface. The teacher clicks the 'next question' button and a question is presented on all pupil interfaces. All pupils answer this question by clicking on one of the answer buttons, and receive immediate feedback on their answer. The teacher then clicks the 'next question' button again, and the cycle is repeated, until all questions have been answered. Based on his previous answers on previous questions, the next question a pupil will receive will either be an easier or a more challenging question, that properly matches the pupils knowledge level. We will now get into the details of the program and the scenario outlined above.

Teacher interface is started

According to this website students from our target audience should be able to do the following things at the end of the school year:

• Group 3 can add and subtract from 0 to 20.
• Group 4 can multiply and add and subtract from 0 to 100.

For this reason exactly, we made sure the teacher can change the settings of the program to fit the needs of the current group. The settings of the current test are completely customizable to the current topic and current level of knowledge the teacher wants to test. We developed our program such that it can at least accomodate the testing needs of our target audients, i.e. addition, subtraction and multiplication can be tested from 0 to 100. However, while developing, we took care of the fact that the program can easily be extended to the testing needs of higher groups. We will expand on this later.

Teacher interface launched when the teacher starts a new test

When the teacher first starts a new test, the teacher interface is launched. A form will be shown on which they can adjust the settings for the upcoming test. This form is shown on the right, including all the settings customizable by the teacher and their standard setting. The standard settings shown on the right are set to accomodate our target audience of group 3, but can be easily customized to meet other testing needs.

First, the teacher chooses the numerical bounds between which the questions of the current test have to lie. Initially, questions lie between the range [0,20]. The teacher chooses whether he wants to test integer or decimal numbers. The standard setting is integer numbers. The fact that we included decimal numbers is an example of how we made the program extendible such that it can later be used to meet the testing needs of higher groups. Next, the teacher chooses the operations he wishes to test: addition, subtraction, multiplication and/or division, standardly set to addition and subtraction. The maximal amount of arguments per question, which is initially 2, can also be customized. Even though our target audience will always use the standerd setting of 2 arguments per question, we still included this option with the possible future testing needs of higher groups in mind. By including this option now, extending the program for the use of higher programs will be a lot easier, because we already kept this in mind. The teacher chooses the amount of questions in the upcoming test. The standard setting here is 20 questions per test. The teacher also initializes the amount of time the pupils have to answer the question (in seconds), standard setting is 60 seconds per question. Lastly, the teacher chooses whether the program should automatically move to the next question if all children have answered or not, and whether a timer that displays the remaining time to answer the question should be enabled or disabled. When clicking the 'start' button, the program generates a test based on the settings filled in by the teacher and starts searching for pupil programs that are trying to connect to the current test. We will expand on the connection process later.

Pupil interface is started

Waiting screen for children.

When the pupil interface is launched by the pupils, a waiting form will be displayed, as is shown on the right. At this point, the pupil program is trying to connect to a teacher program where a test was generated. We will expand on this connection process in the next subsection 'connection is established'. After the teacher has generated a quiz, i.e. after the teacher pressed the 'start' button on the initial teacher interface, this connection between the teacher and all pupil programs will be initiated. As soon as the connection has been established, a name form will be displayed to the pupils, as shown on the right.

Name form for pupils to enter their name and join the quiz

The pupils all fill in their name and click the "Start!" button. By clicking start, their name will appear in the teacher overview, which is not visible for the children. We will expand ont his overview later. An empty question form is now displayed to all children, until the teacher presses the 'next question' button on the teacher overview interface.

Connection is establised

TODO: dennis please fill in :)

Teacher overview during the quiz

Teacher overview with one pupil connected

After the teacher pressed the 'start' button on the teacher setup screen, a teacher overview form is displayed to the teacher. Every time a pupil program establishes a connection with the curren teacher program, the pupils name will appear in a new row on the teacher overview, as shown on the right. In this figure, one pupil called 'Dennis' is currently connected and able to take the test. The teacher overview has several other fields that give the information on how the pupils are doing. The first column tells the teacher whether the pupil has answered the current question yet. The second column tells the teacher, if the pupil has answered the current question, whether he answered correctly or not. The third column shows the current score of the pupil and the last column shows the level category of the pupil (see subsection 'score function'). This overview is an easy way for the teacher to see how all the pupils are doing, which pupils need extra attention and which pupils can move on to more challenging topics. We kept the code for this overview screen as generic as possible, so that adding additional columns at a later stage is easily done.

Questions

Currently, our tests can generate two different question types. Again, the code supporting these question types is generic, so adding additional question types at a later stage is easily done. As mentioned in our research, we found that order to get a correct numerical magnitude representation, children need to practice with symbolic numbers, so digits, but also with non-symbolic formats. Based on this finding, we developed two types of questions:

• Flock questions, where children can add or subtract flocks of pigs, cows, or chickens. These questions are a way for the pupils to practice with non-symbolic formats, by counting amounts of animals, and training with symbolic numbers, by transforming the amount they just counted into an actual numerical answer.
• Animals with small blackboards which have simple math questions on them. These questions are a way for the pupils to train their mathematical skills in a purely symbolic manner.

By combining these two different kinds of questions, the pupils both train and test test their numerical magnitude representation, as well as their understanding of basic symbols of mathematical representation. All questions are multiple choice to make it easy for the young children to answer the question.

The two type of questions. Left. A flock question. Right. A regular question.

Feedback

As soon as a pupils clicks on one of the answers of a question, the pupil will immediately receive feedback on their answer (rather than at the end of the test), as supported by our literature study. Because the pupils are most likely to be young and are only just learning to read, we use smiley faces to indicate to the children how they are doing. This is again supported by our literature study. A happy face is displayed to indicate they answered the question correctly; a sad face means they answered the question wrong. If a pupil answers a question wrong, we wanted to highlight the correct answer, without focussing on the fact that they answered wrong (as supported by our literature study on feedback

Feedback for the children. Left. This question was answered correctly. Right. This question was answered wrong.

Score

In the program, a score function is used to determine the current level of a pupil. In our program, we chose to work with 3 level categories: easy, average or hard. After each question that a pupil has answered, we use the score function to determine the level category of the next question the pupil will receive. This means pupils can receive questions of different level categories during the test; the pupils can shift between different level categories, based on their score up-until-now. The final level category that the child ends up in, determines whether the child needs extra help: children in the easy level probably need some extra attention from the teacher, while children in the hard level can probably move on to a more difficult topic.

The score function we used in our program is as follows:

[math]\displaystyle{ score = score+\frac{correct * current Time}{total Time} }[/math].

Here, 'correct' is a variable that is 1 if the pupil answered the previous question correctly, and 0 otherwise. 'currentTime' is the time it took the pupil to answer the previous question. 'totalTime' is a variable that is set to the total time the pupils had to answer the question. To determine the level of the next question a pupil will receive, this score is then compared with two bounadary variables. These two boundaries are:

[math]\displaystyle{ lowerBound= amountOfQuestions * 100 * \frac{1}{3} }[/math]

and

[math]\displaystyle{ upperBound= amountOfQuestions * 100 * \frac{2}{3} }[/math].

Here, 'amountOfQuestions' is the total amount of questions the pupils have already answered, and '100' is the maximum amount of points a pupil can receive per question. Therefore, amountOfQuestions * 100 is therefore the total amount of score a pupil could have earned up until now. If the pupil has a score lower than one third of this total score, i.e. his score is lower than the 'lowerBound' variable, the pupil is shifted to the easy level and his next question will be of this level category. If the pupil has a score higher than the upperBound variable, the pupil will be shifted to the high level and his next question will be of this level category.

Generating the next question

As explained above, based on their score, every pupil is in a certain level category: easy, average or hard. The next question this pupil receives should thus be of this level. We generate questions as follows. Questions of the 'easy' level category are generated by

If the child is in an easy level, the questions will become simpler by narrowing the range set by the teacher by... . If the child is in the hard level, it will broaden the range by ... . The score of a child is determined by the following function:

End of the quiz

After the last question has been answered, the program is automatically closed.

Results

Survey

The purpose of the survey will be the following:

• look if there is demand for our program
• look if the program can be implemented
• look if teacher will find it a fun new addition to their class
• important features the program should have according to teachers

Conclusion

Part 1

• relatie tussen onderzoek en hoe het eruit ziet en hoe het werkt( wat komt er uit het literatuur onderzoek en hebben we gebruikt in de praktijk)

Part 2

• conclusion of the enquete

Discusion

The following is a summation of topics that would be of interest in further research.

• Implementation and---more importantly---the comparison of different types of questions, e.g. reading the clock. This can be done in terms of plug-in forms in the program.
• Machine learning classification of pupils in (possibly more than three) categories.
• Validation of the score function and possibly incorporating other variables (such as answers that are 'almost' correct and the current score).
• Changing the question difficulty based on more question properties (e.g. operations or 'tricky' multiple choice questions) and measuring the effect of these changes.
• Implementation of the quiz on different platforms, as well as surveying the need for this.