OGMA - Language Acquisition System using Virtual Reality

One of the ways of learning a new language is by going to another place and immersing yourself in that environment. Ogma is a Virtual Reality System in which you can immerse yourself without going anywhere else and learn a new language. 

 
 

1. PROJECT DETAILS

  • Project Name: Ogma - Language Acquisition System using Virtual Reality
  • Project Type: Virtual Reality UX Project
  • Project Theme: Language Learning
  • Team: 2 - An undergraduate researcher and myself
    • My Role: UX Research, Planning the entire project Workflow, Designing the 3D environment, Conducting Usability Studies
    • My Teammate's Role: Development, Helped is Usability Studies, Swedish Language expert inputs
  • Skills:
  • Tools Used: Unity, SPSS, 
  • Project Inspiration: In Fall 2015, my lab purchased the new Oculus DK2 and I wanted to use it for my Masters Thesis project. So I, along with an undergraduate researcher, decided to use VR in the context of language learning. That's how Ogma became a reality!
 

2. INTRODUCTION

2.1. MOTIVATION

        IMMERS  ION METHOD

      IMMERSION METHOD

Suppose you want to learn german. You decide to go and live in Germany for few months so that you can immerse yourself in the environment and not only learn german but also the culture. This is call 'Immersion Method of Learning a New Language'.

Immersion is proven to be one of the most effective ways to learn language. However, it isn’t always a feasible option, so less effective traditional learning methods are used. But these modern language learning methods are still dramatically worse than direct immersion. 

IDEA: Virtual reality could simulate this immersion to provide a more natural and effective language learning method.

"Could virtual reality be used to improve retention of new vocabulary via immersion?" 

2.2. PURPOSE

Virtual Reality is fully immersive computer simulated environment that gives user the feeling of actually being there. We were trying build an immersive virtual environment for language learning. With this, we wanted to introduce a new way to learn a language and test the feasibility of VR environments as training platforms.

There are various disadvantages of immersion method - Time, Money, Danger (Learning Syrian), Weather Conditions, Job!

2.3. RELATED WORK

We conducted literature review to find if similar research study have been done in the past. Virtual Reality was used as learning environment but they were all cave systems. We did not find any work using Head Mounted Display.

Language Acquisition using “Immersive Virtual Reality” was a NOVEL IDEA (in 2015-2016, now things are different)

2.4. HYPOTHESIS

“Immersive VR Systems are better and effective than traditional methods of second language acquisition”

To prove this thesis, we set on a journey using the Iterative User Centered Design Process. 

 

USER CENTERED DESIGN PROCESS

I followed the ITERATIVE User Centered Design Process for this project.

There were 4 Iterations, each consisting of Research, Design and Testing Phase and a Final User Study. 

Screen Shot 2018-08-11 at 10.56.30 AM.png
 

3. ITERATION #1

3.1. RESEARCH - INTERVIEWS

I interviewed 7 people, 4 student taking a language class, 2 working people who had been to other country for work and 1 person who had enrolled in a french class just as a hobby.

  • Why do people learn a new language?
  • How fast do they learn?
  • How existing language learning system work?
  • How VR can improve language learning experience?
  • Who are the audience/user/stakeholders of the system?

3.2. DESIGN - PERSONA

For better understanding of the users, I created personas.

3.3. DESIGN - BRAINSTORMING IDEAS TO SOLVE THIS PROBLEM

Based on the answers, brainstorming was done for initial system design. An INTERACTIVE VR environment in which the user can move around, interact with the world and learn new language words was planned. 

3.3. TESTING - INITIAL PILOT STUDY

Aim: To find out how many words would be appropriate for the final system.

Approach: We did our initial pilot study on 3 participants. 

My partner made a list of 10 Swedish words and asked the participants to learn it. Then after different interval of times we tested them. The list consisted of these words:

  • Apple - äpple
  • banana - banan
  • book - bok
  • bread - bröd
  • milk - mjölk
  • coffee - kaffe
  • pasta - pasta
  • water - vatten
  • lemon - citron
  • shoes – skor

Participant A was given this list and was asked to learn them in 5 minutes just by looking at the list. Then the participant used a vocabulary training website (vokabel.com) to practice. Participant A was tested on the words immediately after as well as a week later, and scored 9/10 and 5/10, respectively.

Participant B and C were also given the same list and were asked to learn them in 5 minutes by looking at the list. Participant B was tested immediately after and scored 4/10. Participant C was tested the next day and scored 8/10.

 

4. ITERATION #2

4.1. RESEARCH - ANALYSIS OF INITIAL PILOT STUDY

  • Participant A – Training, Test- Immediate & Week Later, Score – 9/10, 5/10
  • Participant B  – No Training, Test – Immediate, Score – 4/10
  • Participant C – No Training, Test – Next Day, 8/10

Observations: Participants had more difficulty with words that were different from English. For example, nobody got water (vatten) correct in the test. In addition, participants tend to remember more if they are given some kind of practice. Thus, we decided to use games in our system to help users learn these words.

What we learned: Considering the lack of training presented in this pilot study, we have decided that 10 words should be a reasonably sized lexicon to use in our study. We also learned that time plays a major role. There is a difference between holding words in memory and actually learning them. Therefore, it would be ideal for us to provide our assessment at least a week after the training.

4.2. DESIGN - IDEATION ON SYSTEM DESIGN

Based on the knowledge gained from pilot study, 2 phases of the system were decided - Learning Phase & Game Phase (for practice)

Screen Shot 2018-08-11 at 11.44.23 AM.png

4.3. DESIGN - LOW FIDELITY (LO-FI) PROTOTYPE

Low fidelity prototype was made to check the validity of our idea. A real room was simulated as our virtual environment. Objects had been labelled with Swedish text. 

4.4. TESTING - LO-FI PROTOTYPE

Wizard-of-oz technique was used for this user study. 3 participants were asked to explore the room. A soft music was on in the background. As the participants explored the room, they saw textual cues (other language) over some object. When they looked at it, the music stopped playing and they heard an audio cue (other language) saying the object’s name. After this exercise the participants were tested.

 

5. ITERATION #3

5.1. RESEARCH - ANALYSIS OF DATA GATHERED FROM LO-FI PROTOTYPE TESTING

All of the participants remembered the object’s name in the other language very well.  They liked the music as well. One of them asked "How will you make a person walk in VR?"

5.2. DESIGN - MEDIUM FIDELITY (MED-FI) PROTOTYPE

The question asked by one of the participants made me think about interaction. Interaction was difficult (Remember, this was late 2015 and VR was very new. The only easy way to interact with VR environment was by using the keyboard, which was not at all user friendly). We explored two options - Leap Motion and Myo Band.

Myo band  was decided to be used over leap motion as it was a better input for VR environment. Medium fidelity prototypes, also known as digital prototype was built to find the user’s reaction to VR environment. An open VR environment was created.

User Study: I conducted an informal user study with 3 participants. The users were asked to move in VE by wearing Myo band. They were asked to look at a cube and read the word displayed. We tested our hardware (Oculus and Myo) in this environment.

User Feedback: The users felt uncomfortable navigating in the VE wearing Myo band they complained that the gestures used for navigating were difficult to maintain in VE. This was an attempt to find the easy of interaction with VE.

Screen Shot 2018-08-11 at 1.39.13 PM.png
video Block
Double-click here to add a video by URL or embed code. Learn more

5.3. DESIGN - HIGH FIDELITY (HI-FI) PROTOTYPE

I created the 3D environment. It was an apartment having living room, bedroom, kitchen and balcony. The space was created so that the user can move around and interact with the objects. 10 objects were placed in the apartment. Myo interaction was also built.

5.4. TESTING - HI-FI PROTOTYPE

3 New Participants were asked to use the VR system.  Some notable user feedback were -

  • “Music is good, but should be instrument - no lyrics” The user liked the concept of music but suggested it should be instrumental as it will not interfere with the pronunciation of the words.

  • “Oculus is too heavy”. The DK2 headset was heavy and wired. This caused discomfort.

  • "Need for practice environment”. The user felt very uncomfortable directly dealing with the new VE without any practice. A need for practice environment became necessary.

  • “Objects don’t need English translations, but should be easily recognizable”

  • “Slower pronunciation” The pronunciation were fast for some users.

 

6. ITERATION #4

6.1. DESIGN - ADDITION OF VR PRACTICE ROOM

The following changes were done based on user feedbacks.

  • Instrumental Music was added as in the learning environment.
  • Practice Room was created. This was an empty room with a painting on the wall as seen in Fig 3-10
  • Easy to recognize without English translation – 10 Swedish objects were chosen.

6.2. TESTING - PILOT RUN BEFORE MAJOR USER STUDY

Final User Testing was done before Data collection. One participant tested the system. After using the system, a small assessment was done to check how many words he remembered.Below tables shows the performance of that participant.

 

7. EXPERIMENTAL STUDY

7.1. OVERVIEW OF THE SYSTEM

  • Phase 1: Learn how to interact with VR environment using Myo Band and Oculus rift in the practice environment.

  • Phase 2: Freely roam a virtual apartment space. When looking at a relevant object, text will be shown of what it is in the Swedish language along with audio cues.

  • Phase 3: Game session in which user is given the word and they must go find the object.

Final VR System was made as per user feedback. The interaction using Myo band was adjusted as well to keep the system user friendly.

7.2. USER STUDY DESIGN

I designed the user study (A/B Testing) in the following manner. 

  • There will be two groups of participants. One group will study the word using traditional flashcard method. Other group will learn words using our virtual reality system. (Reason: It was important to measure the effectiveness of VR system over traditional method to prove the hypothesis)
  • Then immediately after learning the words, the participants will give a test.
  • After one week, the participants will be called and asked to give the test again.
  • After completing the test, they will fill out the questionnaire & give informal feedback. 

7.3. CONDUCTING A/B TESTING

36 student volunteers were chosen as participants out of which nineteen participants successfully completed the user study. They were randomly placed in either the traditional or virtual reality group.

Traditional group participants were first given a list of ten Swedish words and their English equivalents and told to study them for up to five minutes. The words were read out loud to them, repeated upon request. Then, they were given flash cards corresponding to the words, and told to study them until they felt they were ready for the test.

Virtual reality group participants entered a virtual practice environment where they could become accustomed to navigating virtual reality. Then, they entered the virtual apartment and were allowed to explore freely for up to five minutes, looking at objects to see and hear the corresponding Swedish words. Afterward, they entered the second phase, in which the Swedish words were displayed and pronounced and participants had to point to the corresponding object, until each was complete.

Test: Both groups took a test in which they were given the list of English words and told to write and pronounce the corresponding Swedish words. For each word, participants were graded on both spelling and pronunciation with correct(5 points), almost correct(2.5 points), and incorrect(0 point). All participants came after a week, gave the same test and filled out the questionnaire.

Questionnaire:

  • Aptitude of language
  • Have they learned any language before?
  • Which other methods did they use for learning language?
  • How was it different from our method?
  • Enjoyability Rating
  • Effectiveness Rating
  • Other Feedback
 

8. EXPERIMENTAL RESULTS

7.1. OBJECTIVE RESULTS

We measured the scores from the initial test (the test given immediately after the study) & post test (the test given one week later). We also calculated the word memory retention. 

7.2. SUBJECTIVE RESULTS

Ratings were much higher for virtual reality than for the traditional method, with the majority of participants rating virtual reality as “Very Enjoyable” & "Very Effective".

7.3. IMPLICATIONS

Even though traditional method had higher initial scores, the retention of VR system was way higher than traditional. VR is better than traditional because “it allows them to visualize objects”.

 

9. LEARNING OUTCOME

9.1. HIGHLIGHTS OF OPEN ENDED FEEDBACK

  • “Music kept the environment interactive”
  • Dizzy, Fear of falling of the door
  • “Felt easy and natural to navigate the virtual environment”
  • “When we write the words, we actually travel around the apartment in our minds. Its impactful”
  • “Would love to have game/real scenario in VR to interact”
  • “Interesting but requires practice as its new method”
  • Flashcards boring; pronunciation hard; boring for many words

9.2. CONCLUSION

The findings show excellent potential for virtual reality in language acquisition.

 

10. PUBLICATION

I published this work in an international academic conference - PETRA in 2016. You can find the paper here.

 

11. CHERRY ON THE CAKE :)

I was awarded the 'Verizon Outstanding Masters Thesis Award' in 2016 for this project. Ogma became quite popular in my university as it was the first Virtual Reality System developed using Iterative User Centered Design Process.