Game Evaluation Report
Games have gradually moved into classrooms of all levels over the past few decades and are gaining acceptance in situations as vastly different as early childhood settings, universities and workplace learning environments. This article will explain why games belong in the classroom and evaluate a serious game, Planet Mechanic, as evidence of this value.
Before discussing the benefits of games in education, and evaluating their worth, the concept of what a game is must first be defined. McGonigal (2012) defines a game according to four key traits; a goal, rules, a feedback system and voluntary participation. This definition is useful, and can be applied to nearly all games, as diverse as billiards and bridge. Serious games, otherwise known as games for learning or educational games, are games that have been designed with a specific audience and specific learning outcomes in mind (Rodríguez-Aflecht, Hannula-Sormunen, McMullen, Jaakkola, & Lehtinen, 2017).
Voluntary participation is a difficult concept in educational settings, however recent research shows that the voluntary participation aspect has little impact on learning (Rodríguez-Aflecht, et al., 2017). Therefore, for the purposes of this evaluation, the voluntary participation aspect of games will be eliminated, which leaves the operational definition of a game, for this paper, to be an activity that contains a goal, rules and a feedback system.
Game Based Learning
Raph Koster (2013) says that all games require learning and, in fact, the fun is in the learning and therefore games being brought into the educational environment should not be surprising. Koster says that players play the game to learn how to play the game better, and when the learning stops, the enjoyment stops. In fact, research has shown that playing a game and failing is actually just as enjoyable as winning the game, possibly even more so (McGonigal, 2012).
Game Based Learning (GBL) should not supplant traditional teaching methods, but rather support them as a valuable way to teach and reinforce learning (Koster, 2013) and GBL has many benefits for students. GBL improves knowledge acquisition for students (Perrotta, Featherstone, Aston, & Houghton, 2013), improves problem-solving skills in both global and topic-specific senses (Tsekleves, Cosmas, & Aggoun, 2016) and aids in memory retention and retrieval (Jean, 2019). Some studies have shown that GBL increases learner motivation (Woo, 2014; Pereira de Aguiar, Winn, Cezarotto, Battaiola, & Varella Gomes, 2018), although this claim has been disputed (Rodríguez-Aflecht, et al., 2017). Students playing games based on subject content are able to see the concepts with fresh eyes (Johnson, Adams, Cummins, Estrada, Freeman, & Ludgate, 2012). Additionally, GBL that requires students to work in teams fosters the Twenty-First Century skills of teamwork and communication (Boikou, 2019). Game Based Learning is undoubtedly beneficial for students.
Serious Game Design Assessment Framework
The Serious Game Design Assessment Framework (SGDA Framework) was developed by researchers at the Massachusetts Institute of Technology’s Singapore-MIT GAMBIT Game Lab to fill a perceived gap in the market of evaluative methods used when assessing the worth of serious games (Mitgutsch & Alvarado, 2012). They reviewed the few existing models of evaluation and found them lacking due to either a narrow focus on the design aspect of the game, or the absence of evaluation regarding the game’s purpose and whether it is achieved. Subsequently the SGDA Framework was developed, which assesses a game on seven aspects: purpose, content, fiction and narrative, mechanics, aesthetics and graphics, framing and the game system.
Planet Mechanic (Filament Games, 2015) is an educational game designed to teach students in middle school about earth and space sciences. It contains fifteen levels of increasing difficulty, and takes about 30 mins to play, with the potential to extend, as the final level is free play. It is available on iOS and Android tablets as a standalone game and is available on iOS and Android tablets and web browsers in the education edition (Filament Games, n.d.) and therefore should be accessible to all schools, although the $2.99USD per student fee may put this app out of budgetary reach for some schools. These factors are important as technological requirements and finances are significant obstacles to teachers implementing game based learning (Katmada, Mavridis, & Tsiatsos, 2013).
For this evaluation, the standalone edition for iOS was tested on an iPad running iOS 12.1.1.
Planet Mechanic is a goal oriented game (Hickey, North & Nagy, 2019) that aims to teach students about earth and space sciences, specifically how a planet’s lunar cycles, atmosphere, revolutions, tilt and rotation impact temperature, time and seasons (Filament Learning, 2015). It aligns with the Australian Curriculum Year Seven Science program in the area of Earth and Space Sciences and Science Inquiry Skills (Australian Curriculum, Assessment and Reporting Authority, 2018). The gameplay involves the student as the planet mechanic with a control panel to adjust the settings of the planet on the screen, with a non-playing character (NPC) of an alien who makes requests on adjustments required to the planet in question. Clear learning outcomes, like the ones Planet Mechanic targets, are an important factor in ensuring the game meets its learning objectives (Doney, 2019), and a logical integration of the content into the gameplay makes the GBL meaningful (Perrotta, Featherstone, Aston, & Houghton, 2013).
The first level involves a planet with no moon, and the alien requesting that the planet be given tides so that a chocolate-bearing shipwreck may be retrieved. The solution is to give the planet a moon, reinforcing the learning that the moon controls the earths tides. Subsequent levels involve making the year shorter so that birthdays will happen more frequently, or creating a planet with specific climactic conditions.
Content and Information
The content and information aspect involves all in-game text, and data that appears on the screen (Mitgutsch & Alvarado, 2012). Planet Mechanic provides data on the alien’s latest requirements, and includes a heads up display (HUD) with the current settings of the planet and the impact those settings has on the climate (in displays for temperature, seasons, the length of days and years and so on, labelled “planetary details”). In addition, it provides factual information throughout the course of the game as to the “settings” that would apply to Earth. (Pereira de Aguiar, Winn, Cezarotto, Battaiola, & Varella Gomes, 2018).
The information on screen is easy to understand, factual and relevant to the task at hand, while providing information ‘just in time’, which helps students to understand and apply the new information well (Gee, 2005). It also provides a voice over (which can be switched off) that would enable students with poorer literacy to still participate in this GBL experience. The information could be improved, however, by including factual information about other planets in our solar system, and how their “settings” impact their climate. That said, it teaches everything it can before the player has a chance to be bored and stop playing and so is a good game, as defined by Koster (2013).
Game mechanics are the actions a player takes to have an impact on the game environment and, as such, are described using verbs (Sicart, 2008). It can also refer to the challenges faced, and the win state of the game (Pereira de Aguiar, Winn, Cezarotto, Battaiola, & Varella Gomes, 2018).
The game mechanics in Planet Mechanic are simple, most actions are completed by making adjustments to planet settings, via either a slider or an on-off switch and then submitting your planet for assessment. The other action taken is positioning the moon in the correct place to cause an eclipse.
The complexity of the game comes from the sometimes unexpected impact the changing settings have on the “planetary details”, enabling students to understand how these factors interact.
Fiction and Narrative
The story in Planet Mechanic is a short and simple linear narrative (Pereira de Aguiar, et al., 2018), but engaging enough to give the tasks meaning. A fantasy setting, such as Planet Mechanic’s space theme, can create deeper learning engagement and player immersion in the game experience (Doney, 2019). While the narrative is relatively shallow, it is sufficient for this game that could be completed in one or two lessons, as a more engaging narrative would take more time to develop. A longer game may then consume too much classroom time, and may not be seen as a suitable investment of time for teaching to a relatively small number of science outcomes.
Aesthetics and Graphics
Planet Mechanic has two dimensional cartoon-like graphics, with a simple graphical representation of a planet orbiting a sun, surrounded by sliders and the “planetary details” HUD. The cartoon-like images allow students to more easily put themselves in the shoes of the Planet Mechanic and suspend disbelief, something that would be hindered with more realistic graphics (McCloud, 1994). The simpler images also allow students to focus on the content and learning as complex animations can increase cognitive load (Woo, 2014). The music, an ambient track that evokes a sense of vast empty spaces, complements the theme of the game without being distracting
(Pereira de Aguiar, Winn, Cezarotto, Battaiola, & Varella Gomes, 2018).
Framing refers to the way that the game has been designed for a particular target group, in this case students in year 7, and whether the game and the play literacy required, are suitable for this type of student (Mitgutsch & Alvarado, 2012). The game is well designed for 11-13 year olds (although early levels could be played by younger players) and the learning curve for learning to play is minimal, as students do not require any prior experience with video games to quickly learn this game.
The framing aspect also includes whether the game is responsive to the players by increasing the difficulty as the game progresses, as this progression in difficulty enhances learner engagement (Djelil, et al. 2014). The game begins with the aforementioned tide challenge (see the section on Purpose), which can be completed in under a minute, but gets progressively more difficult over the fourteen key game levels.
In the instance of an incorrect response the game replies “this planet is broken” and the player is prompted to try again. This repeated predictable fail outcome does not meet the criteria for a good learning game, which requires a variable feedback system and a cost for failure (Koster, 2013). Fail states that are relevant but less predictable and boring enhance enjoyment of the game and encourage players to persevere even when they are struggling (McGonigal, 2012).
Each of these incorrect responses is counted and is viewable to the teacher (in the education edition) (Filament Learning, n.d.), and by the player themselves at the end of the game, but does not have an impact on gameplay. Failure to complete a level does not preclude a student from completing subsequent levels, although a player must return to the home screen to select a different level if they are not progressing through the levels sequentially.
After completion of each level, there is a multiple choice question generally relating to the changes made in that level to ascertain that the student understands what changes were necessary and why, rather than simply clicking everything until something worked. This second possible fail state encourages students to grasp the underlying knowledge rather than simply guess, and their results of these challenge questions are recorded.
Due to the format requiring a single correct answer, this game does not easily lend itself to replayability, although the free play level included at the end of the game allows room for experimentation and perhaps teacher- or student-set challenges for the class.
One way the framing in Planet Mechanic could be improved is by allowing teachers to customise the game for the benefit of their specific cohort of students, to better suit their prior learning and needs (Ak, 2012). Adding further advanced levels, including those with multiple solutions, would improve the learning benefits by helping students to feel like they have agency and their choices make a difference to the gameplay (Gee, 2005).
Coherence and cohesiveness
This aspect the evaluation examines how well the previous elements of the game work together to achieve the purpose of the game (Mitgutsch & Alvarado, 2012). Planet Mechanic is fairly simple in gameplay but it effectively demonstrates the principle of the Goldilocks Zone (National Aeronautics and Space Administration, n.d.) by allowing students to make adjustments to the planet that ultimately affect its liveability. The game is developed from a constructivism learning theory foundation (Pereira de Aguiar, et al., 2018) which allows students to experiment in a safe space to build their own learning. It shows the interconnectedness of planetary systems which teaches students the complex notion of systems thinking, the understanding that small actions in one area can have broad implications across multiple other areas (Molderez, & Ceulemans, 2018).
Overall, Planet Mechanic is sufficiently engaging and detailed for a game played across one to two classroom lessons. It quickly teaches students about the climactic impact that atmosphere, tilt, rotation, revolution and lunar cycles have on planets in a way that encourages systems thinking. It also has a minimal learning curve for teacher and student alike, allowing reluctant teachers to investigate Game Based Learning on a small scale. Improvements could be made in allowing further teacher customisation and having advanced levels with more variables and multiple solutions that would help students to feel they were active agents in the game. Additional work on the fail state of the game being less predictable (such as the planet wobbling off its axis, or the alien being upset that the snow has melted) would increase enjoyment and consequences for failing would encourage students to work hard to produce the right answer.
Game based learning is a valuable teaching method and should be incorporated into classrooms, bringing student benefits in not only content learning but transferrable skills such as systems thinking and team work. Planet Mechanic represents a low-investment game that allows teachers to experiment with incorporating game based learning into their classrooms. References
Ak, O. (2012). A game scale to evaluate educational computer games. Procedia-Social and Behavioral Sciences, 46, 2477-2481.
Australian Curriculum, Assessment and Reporting Authority (2018) Australian Curriculum: Science. Version 8.4. Retrieved from https://australiancurriculum.edu.au/f-10-curriculum/science/
Boikou, Α.G. (2019). Game based learning’s impact in learning achievement: a systematic review. Retrieved from https://dspace.lib.uom.gr/bitstream/2159/22815/4/BoikouAndromachiMsc2019.pdf
Djelil, F., Sanchez, E., Albouy-Kissi, B., Lavest, J., & Albouy-Kissi, A. (2014). Towards a learning game evaluation methodology in a training context: A literature review. Reading: Academic Conferences International Limited. Retrieved from https://search-proquest-com.ezproxy.csu.edu.au/docview/1674246431?accountid=10344
Doney, I. (2019). Research into effective gamification features to inform e-learning design. Research in Learning Technology, 27.
Gee, J. P. (2005). Learning by design: Good video games as learning machines. E-learning and Digital Media, 2(1), 5-16.
Hickey, S., North, C & Nagy, G. (2019) Games and Learner Engagement: Gamification, eLearning, and Virtual Reality. Participant Resources (AECT) [pdf file] Games and Learner Engagement webinar series, AECT Learner Engagement Division. Retrieved from https://osu.app.box.com/s/pf64cn3mmxal01tukw3fhx3g0pcxlvv7/file/430086752454
Jean, P. H. (2019). Brain-based and learning theories: Application of theories in the classroom. European Journal of Education Studies. 5(12). Retrieved from https://oapub.org/edu/index.php/ejes/index
Johnson, L., Adams, S., Cummins, M., Estrada, V., Freeman, A., & Ludgate, H. (2012). The NMC horizon report: 2012 K-12. The New Media Consortium, Austin, Texas. Retrieved from https://www.nmc.org/publication/nmccosn-horizon-report-2017-k-12-edition/
Katmada, A., Mavridis, A., & Tsiatsos, T. (2013). Game based learning in mathematics: Teachers’ support by a flexible tool. Reading: Academic Conferences International Limited. Retrieved from https://search-proquest-com.ezproxy.csu.edu.au/docview/1549957118?accountid=10344
Koster, R. (2013). Theory of fun for game design. Retrieved from https://ebookcentral.proquest.com
McCloud, S. (1994) Understanding comics: The invisible art. New York: Harper Perennial.
McGonigal, J. (2012). Reality is broken: Why games make us better and how they can change the world. Vintage: London.
Mitgutsch, K., & Alvarado, N. (2012, May). Purposeful by design?: a serious game design assessment framework. In Proceedings of the International Conference on the foundations of digital games (pp. 121-128). ACM.
Molderez, I., & Ceulemans, K. (2018). The power of art to foster systems thinking, one of the key competencies of education for sustainable development. Journal of Cleaner Production, 186, 758-770.
National Aeronautics and Space Administration (n.d.) The Goldilocks Zone. Retrieved from https://exoplanets.nasa.gov/resources/323/goldilocks-zone/
Pereira de Aguiar M., Winn B., Cezarotto M., Battaiola A.L., Varella Gomes P. (2018) Educational Digital Games: A Theoretical Framework About Design Models, Learning Theories and User Experience. In: Marcus A., Wang W. (eds) Design, User Experience, and Usability: Theory and Practice. DUXU 2018. Lecture Notes in Computer Science, vol 10918. Springer, Cham
Perrotta, C., Featherstone, G., Aston, H., & Houghton, E. (2013). Game-based learning: Latest evidence and future directions. Slough: NFER. Retrieved from https://www.nfer.ac.uk/nfer/publications/GAME01/GAME01.pdf
Rodríguez-Aflecht, G., Hannula-Sormunen, M., McMullen, J., Jaakkola, T., & Lehtinen, E. (2017). Voluntary vs Compulsory Playing Contexts: Motivational, Cognitive, and Game Experience Effects. Simulation & Gaming, 48(1), 36–55. https://doi.org/10.1177/1046878116673679
Sicart, M. (2008). Defining game mechanics. Game Studies, 8(2). Retrieved from http://gamestudies.org/
Tsekleves, E., Cosmas, J., & Aggoun, A. (2016). Benefits, barriers and guideline recommendations for the implementation of serious games in education for stakeholders and policymakers. British Journal of Educational Technology, 47(1), 164-183.
Woo, J. C. (2014). Digital game-based learning supports student motivation, cognitive success, and performance outcomes. Journal of Educational Technology & Society, 17(3), 291-307.