PRE2017 3 Groep9 - Test Setup
Back to the PRE2017 3 Groep9
Test Setup
The test will follow the following procedure:
- Quick introduction to let the test subjects know what they are being tested on
- Test subjects start answering a questionnaire about themselves
- The test subjects will get 7 minutes to study
- After studying there will be a 5 minute break to get rid of the words in the short term memory
- The subjects will then take a test about the words that had to be studied
- After the test the subjects will have to answer another questionnaire in which we ask for feedback and to rate the study methods.
You might notice the 7 minutes time to study, which seems arbitrary, however it is based on some test that we did. For this test we took 15 random italian words (different from the words in our game, since we already know those), and gave ourselves 10 minutes to study them. Then we had the 5 min break and took a test. All of us scored near perfect scores, which we think could be a problem. We looked at the list of words we had and decided it was slightly easier than the random list we learned. This made us believe that 10 minutes to study this list might result in a big part of the test group getting perfect scores, which we should avoid in order to see the difference between our 2 test groups. Therefore we decided to slightly reduce the given time, with the hope that it won’t be too easy. The test subjects will be split into 2 groups,which are explained further down below. The estimate for our total subject size is 50 students from the TU/e. We will also try to find more test subjects from our former high schools, since the more test subjects the more accurate the results.
Group 1: Study using the game
This group will be told that they are playing a game in order to learn Italian words, this information will be given because if this learning method is going to be used in the future the people using this method will still be aware that they are learning Italian words when using it.
The questionnaire they have to answer beforehand will be the exact same as that for the second group. This questionnaire will focus on getting information about the participants. This will include age, how much experience with learning languages the participant has as well as what languages they already speak.
They will then start studying, during the studying the only tool they may use is the game. The game will be run on a computer. After this they will have their break and take the test, which is all the same as the other test group.
After the test they will get a more detailed questionnaire, which is slightly different from the other group. This questionnaire will be focussed on a combination of the enjoyment gotten from the game and the perceived quality of the learning method, compared to their opinion other methods. We also asked for feedback on our game.
Group 2: Study using pen and paper
This group will be told that they have to learn the list of words given to them and take a test about it afterwards.
The questionnaire they will answer beforehand is the exact same as that of group 1.
During the studying this group will get a list with the English words and their translations to Italian. They will also be allowed to use scrap paper and a pen to write things down. After this they will have their break and take the exact same test as that of group 1.
After the test they will get a similar questionnaire, which will focus on their rating of this method of studying and what other methods they might prefer. However because they didn’t play our game we could not ask for feedback or how they would rate it and thus creating a difference in questionnaires.
Pre-Test Questionnaire
The purpose of this questionnaire is to get a view on what level of language skills the test subject has. This is mostly to catch outliers, say a person speak 10 languages they will probably have an easier time learning the words, because they either already know them, in which case they will probably not be a test subject, or because they can recognize them from other languages. They will probably also simply be better at learning a language because of their experience. The other aspect of this test is the age, this is again to find outliers. We assume that because the test subjects will be mainly students they will fall into the same age category.
What the questionnaire looked like:
How many languages do you speak, and which ones?
Do you like learning a language?
What is your native languages?
What is your age?
Post-Test Questionnaire Group 1
These are the questions that the people that have studied using the game will have to answer:
Did you feel as if you were playing a game? Please explain why (not).
How did you enjoy this way of learning? Please explain why it was (not) enjoyable.
If this was available for every language and for more words would you use it? Please explain why you would (not) use it.
Would you play a similar game (without the learning aspects), just for fun?
On a scale of 1 to 10, how would you rate this way of studying?
Similarly how would you rate studying from a textbook?
Do you feel this way of studying would require more or less time, compared to book studying?
What way of studying do you normally use (e.g. WRTS, learning from a textbook with just a list of words and translations, or some other method)? And if this method is not learning from a textbook, how would you rate it on a scale of 1 to 10?
The first 3 questions are focussed on trying to measure the amount of fun the subject had playing the game. For these questions we expect to get answers about the way this method of learning was perceived. Most importantly we can see if we have succeeded in making it feel like playing a game, and less like studying. The 4th question is important to get context on the answers of the first questions, if a person does not enjoy playing this sort of game to begin with their evaluation of our game will probably be affected. Questions 5 and 6 are to get a direct comparison between the value that the test subject gives to this method of studying compared to pen and paper studying. The 7th question is there to get insight in how effective the test subject thinks this way of studying is. The last question is there for comparison to the answers the other group give to the same question. This will allow us to see if the average is in any way influenced by having played the game.
Post-Test Questionnaire Group 2
The other group will also have to answer a few questions, though less than group 1.
On a scale of 1 to 10, how would you rate this way of studying?
Do you prefer any other way of studying (e.g WRTS, or some other method), and if so how would you rate that method?
The first question is there so we can compare the answers of this group to the answers of the first group, which will allow us to see if using a new method of learning has affected their view of the old method. The second question is to get some context on whether the subject would use this specific pen an paper way of studying or prefers something slightly different, e.g WRTS, duolingo. The rating will show how much they prefer their own method of studying to the ordinary pen and paper method, which can then be compared to the ratings the other group gave for the game method.
Links to actual files used for testing
The test: https://www.overleaf.com/read/qzwhzndvbrjj
Questionnaire before test: https://www.overleaf.com/read/dnspkgbvwzpk
Post-Test Questionnaire Group 1: https://www.overleaf.com/read/dxgbfthjvvxb
Post-Test Questionnaire Group 2: https://www.overleaf.com/read/hftbdsxvbqmn