How a Serious Game Improved Trainee Performance by 60%
Via: Thinking Worlds - Improving Navy Recruits‟ Performance: A Serious Games Study
Following my prior post Serious Games Help New Recruits On The Ship where I addressed the "Serious Game" conception, here is the story of how the Royal Navy used this "Serious Game" to transform training effectiveness on a crucial operational area on board ship – Engineering Safety Rounds Inspection.
Historically, this was the Maritime Warfare School’s worst performing course. Compared to classroom training, the immersive learning experience of Serious Games improved trainee capability, cut failure rates by 54% and reduced the needs for additional resource requirements.
A Performance Problem
What would you do if 13% of all new recruits were failing one of their most critical courses? How would you cope with a failure rate that was 200%-300% higher than most of your other core training subjects and the cost to retrain failed recruits kept on rising? Clearly, figures like these would be unsustainable in most training and performance programs and this was no different for the training officers of the Royal Navy’s Maritime Warfare School (MWS), where improving performance and reducing cost were constant operational goals.
Upon identifying these unfavorable statistics, the MWS began evaluating current training methods for the course in question - the Engineering Safety Rounds Inspection - and officers were able to identify several factors. Getting on board ship for the first time was a shock for new recruits and this, coupled with the close spaces and complex operational environment, proved too challenging for many new recruits to complete without further training.
A solution was needed to reduce this initial shock period, shorten the learning curve and provide a safe practice environment for recruits to carry out their Safety Rounds Inspections, prior to boarding the ship for the first time. It was also necessary that whatever solution was reached, training officers could still assess the recruit's technical aptitude and fault finding skills during the trial inspection.
It was decided that the best and most cost-effective solution was to create a Serious Game: a fully immersive, 3d virtual ship that would enable trainees to walk through it and perform duties as if they were actually on board.
Serious Game Requirements
In order to achieve this, the MWS, in conjunction with award-winning Serious Games development company Caspian Learning, began production of the virtual ship and met with subject matter experts from the specialist school to start building the course content.
Allow trainees the ability to role-play scenarios
Allow the trainees to explore an accurate representation of the ship - a Type 23 Royal Navy frigate.
The ability to have trainees observe an instructor walking through the scenarios whilst also having access to a separate self study mode.
The ability to interact with the environment.
The ability to re-use the assets (art content) for further applications (such as recruitment or experimentation).
What Was Done?
To meet MWS‟s objectives, Caspian Learning developed an immersive 3D interactive game titled Weapons Engineering Round - Immersive Learning Simulation with their Thinking Worlds™ technology.
Thinking Worlds™ is a globally unique engine that has a range of proven learning interactions and behaviors designed into it. The engine allowed Caspian Learning to develop and utilize high fidelity 3D environments, such as the exterior and interior of the Navy frigate in the MWS game. It also allowed highly interactive game challenges to be used to target core learning outcomes.
As if they were on board a real ship, the trainers lead the learners through many of the ship’s compartments – the Combined Radar Office, Gyro Room, Bridge, Mess, SCOT Office, 4.5 area, Ship Control Centre, FWD SIS, FWD 911, and CMR. Additionally, the trainers guide the recruits through a typical Engineering Safety Rounds Inspection that involves photo-realistic device and compartment checks. During each round, and in each compartment, trainers examine and elaborate on equipment. By the time the lesson is complete, recruits are familiar with the importance of their jobs and the critical nature of the machinery on the ship.
Going Solo
In the free play solo mode of the game, recruits immediately discover that on board the ship is a saboteur who is creating faults in the machinery and putting the ship’s crew in danger. The recruit must find and fix the errors, locate the saboteur, and disarm his bomb before time runs out. In doing so, the recruit explores the ship and its cramped noisy compartments, interacts with the equipment they will use on the job and assembles knowledge about the critical importance of the machines.
Key Performance Metric
A key performance metric for the UK Military is Brought Up To Standard, or BUTS rate for short. The Military rigorously measures human performance and for those recruits that fail to make the required standard, they go on to receive additional BUTS training. This demands a considerable investment in additional training resources, cost and time off the job.
It is the BUTS measurement that was used to determine the effectiveness of the Serious Game lesson in comparison to the traditional classroom lesson. The measurement was taken over 10 classes, throughout the term.
Serious Game Makes Serious Impact
Before an immersive learning simulation had been used to deliver this course, the pedagogical approach had been more traditional; delivered under normal classroom conditions. This meant that recruits spent a lot of time reading from text books and watching PowerPoint presentations.
These methods were not very engaging and far from realistic. The missing context made it difficult for new recruits to fully appreciate their responsibilities and the consequences of not doing their jobs properly. Despite the comprehensive classroom training, new recruits were still heavily susceptible to a shock factor when they boarded for the first time. BUTS rates were 200% - 300% higher compared to other courses.
After using the Weapons Engineering Rounds Serious Game to deliver the course content the BUTS failure rates dropped substantially by almost 60%. The 6% failure rate was now lower than the average for other courses.
As well as measuring the BUTS rates, a questionnaire was tailored specifically to the program and given to all of the recruits who took part in the trial period. Of these 116 trainees, 99 questionnaires were returned and the collated results of this survey can be found at the Caspian Learning website1. In general, the trainees found that the simulation was easy to use, relatively intuitive and did aid their understanding of the subject matter being taught.
Feedback from training officers corroborated the results of the questionnaire and the introduction of the serious game package and the overall reaction from MWS has been positive and already discussions are taking place for further content and course development.
Training officers were invited to comment on their own feelings to the course and a summary of some of their comments are below:
“The majority of the students find that the game facilitates their understanding and aids the learning process.”
“Less time is spent using PowerPoint, a welcome break for both instructors and trainees.”
“The simulation has been smoothly integrated into the overall lesson plan.” “Overall, Caspian Learning’s Serious Game has been well received and has enabled a fuller understanding of one of the most important roles the trainees will conduct once they join their ship.”
“The lab setup worked well allowing the instructor easy access to the student PCs as required.”
“The noise level in the class is much less than normal. This is due to the increased concentration levels and focus of the trainees on what they’re doing.”
Undoubtedly, the introduction of the Serious Games package into the MWS Engineering Safety Rounds Inspection course has generated a marked improvement in trainee capability and a greater understanding of the module. This is particularly evident in the reduction of BUTS activity necessary after the course. We can confidently say this is a direct result of the introduction of the game as no other element of the course had been altered.
The qualitative results support the quantitative BUTS data and the overall feeling from this collation of feedback is that the immersive learning simulation has enabled greater trainee understanding of the subject and has enhanced the way this subject is delivered.
Final Words
The use of Serious Games technology to deliver a key element of career training in the Military has proven successful. The avatar based Serious Games package delivered and deployed by Caspian Learning has enabled trainees with no previous experience of a Royal Navy warship to successfully conduct Weapons Engineering Safety Rounds in a safe, but effective, training environment.
The game picked up the Best Game Award at the eLearning Age Awards 2009. The simulation is designed to run.