A very common question that Instructional Designers field from clients goes something like this: “The training plan looks great; now how will I be able to evaluate the training effectiveness?”
While this is a perfectly natural question – after all, why else would one buy training – it can be a particularly difficult one to answer.
For a variety of reasons, the ability of training organizations to measure how well employees are “getting” what the training sets out to provide is limited at best. To start with, while clients often want to know how the training is going, they tend to be unwilling to pay for it – post-project assessment line items are often the first to be cut when any training budget is presented.
Even when money does exist for training evaluation surveys and assessments, the realities of training project-timelines often make it impossible to follow one of the established models, anyway.
For instance, the vaunted Kirkpatrick model, which has been the gold standard for evaluating training effectiveness for over fifty years, requires a good deal of time to implement correctly across all four levels – the kind of time that rarely seems to exist in real-world corporate training.
So what does this mean for the lowly Instructional Designer (or his still-lowlier Project Manager) when faced with that scary question about “measuring training effectiveness?” Should we sit the client through a well-worn PowerPoint covering Kirkpatrick’s method, knowing that by the time the project is nearing its completion there will most likely be neither time nor budget for such frivolousness?
Or is this the moment to be a training hero and present an entirely new and radical solution, one that can magically combine the first three (and possibly all four) of Kirkpatrick’s levels into the act of training itself?
We’re talking about gaming, people, and it holds out the promise of obviating the need for almost all direct project assessment. Because when a game is well-designed (as yours surely will be), the learner is effectively assessing himself by his very progress.
To see what I mean, let’s take a look at the four levels of the Kirkpatrick method:
Now think of your favorite video-game. For me, something like Skyrim comes to mind. For those who haven’t played it, it’s a classic role-based video game where the player’s character must overcome a variety of situations by developing a set of skills, and then use those skills (often in novel ways) to advance to a higher level and a new quest. When I first got the game, I happily played it for weeks (to the annoyance of my girlfriend), advancing rapidly in skill-level and completing quest after quest.
So did Skyrim-as-training score well on the Kirkpatrick model? Do we need a series of questionnaires, tests, and charts to figure that out? Nope. The answers can all be found in my performance in the game. Level 1 is covered – I wouldn’t have kept playing for weeks and weeks if I didn’t think it was well-designed and engaging. Level 2 is clear, as well – in order to progress in the game (which I genuinely wanted to do), I needed to learn certain skills. On the flip side of that coin, my changed-behavior was clearly demonstrated by my ability to employ my new skills to advance through the quests. So we can check off Level 3, as well.
The Kirkpatrick level that may or may not be measurable simply through playing a game comes at Level 4. In my example, it’s certainly included – the goal of my learning was to “beat” the game, and I was able to accomplish that. In a business situation, it may not be so clear cut (measuring financial performance, for instance, might require careful real-life study). Still three of the four levels covered in the training itself aren’t bad!
While Skyrim is an excellent example of a game that fosters a desire for learning (so effectively, in fact, that a player-run wiki detailing facts, scenarios and strategies for the game has more than 100,000 articles), it is also not something that the average training budget and timeline could produce. Nonetheless, simpler games can also foster intense interest and spark self-directed learning.
An example that comes to mind from my own past is the strategic board game Diplomacy, which was designed by a graduate student at Harvard to mimic the strategic conflicts in Europe at the beginning of the 20th century.
This game requires little more than a map of Europe and some game pieces, but it is instantly engrossing and has inspired thousands of articles on game strategy. Compared to the millions of man-hours (and dollars) that went into making Skyrim, Diplomacy was the work of one guy and some paper, and yet it still succeeded in creating a dedicated group of players who by their very actions prove the effectiveness of the game’s design.
So the next time you’re faced with a need to demonstrate training effectiveness, consider attacking the problem head on with a training game. Not only will you give the learner a welcome break from the usual classroom and web-based training routine, you’ll actually receive clear evidence of how well your training is doing – all without a single test or questionnaire in sight!