Recent research by ASTD and REED Learning indicates that the top skills desired by Learning & Development departments are measuring and evaluating training.
Even though many Learning and Development organizations find it a challenge to prove training’s effect beyond how learners react to the training and whether they have learned the training content, senior management and business stakeholders are more and more interested in metrics that show the impact on the organization.
According to Donald L. Kirkpatrick’s revised “Four Levels of Evaluation” model, what we need to do is find out what success looks like in the eyes of these senior managers and stakeholders and let them define their expectations for the training program. Then we need to identify specific metrics to demonstrate and deliver on those expectations.
For those of you who are not familiar with the original Four Levels, this is what they are:
Kirkpatrick’s revision of the Four Levels starts with defining what the results should be and then goes backwards through the four levels in sequence, building a “chain of evidence” with data from all four of the levels. The “chain of evidence” supports the results, showing the value learning and reinforcement has provided to the business. Kirkpatrick calls this Return on Expectations, or ROE.
As a followup to the revised Four Levels model, James and Wendy Kirkpatrick say the “true,” or “complete,” Kirkpatrick model is really both a planning and evaluation tool. They distinguish between the development of the plan to build, deliver, and evaluate training programs from the actual collection of data for the “chain of evidence.”
What they have done is to present the model upside-down during the planning stage of the training program, starting with the desired Level 4 results. When the program is being executed and evaluation data is being collected from each level, the levels are followed in sequential order starting with Level 1.
James and Wendy’s model is divided into two parts:
This is James and Wendy’s diagrammatic representation of the “complete” model:
As I think the diagram is rather confusing, I am going to provide a step-by-step process instead. The model starts with identifying the business need and the connection between the business and learning. The first four steps encompass the actions needed to achieve this across levels, moving from Level 4 to Level 1.
- Determine what success will look like (Level 4. Results).
- Determine critical behaviors and organizational drivers required to achieve that success (Level 3. Behavior).
- Determine the required Knowledge, Skills and Abilities (KSAs; Level 2. Learning).
- Determine the learning environment and conditions that will support learning effectiveness and enjoyment (Level 1. Reaction).
- Design and build the learning programs to achieve the targeted outcomes.
- Deliver the learning and begin formal evaluating with Level 1: Reaction and Level 2: Learning.
- Initiate onging reinforcement and monitoring.
- Analyze findings for Levels 1 through 3 along the way, making adjustments to eliminate any problems and creating a backup plan if necessary.
- Gather final data/information for all four levels and prepare for presentation to the stakeholders.(li>
- Present findings to stakeholders starting with Level 1 through Level 4 and demonstrating that the overall learning inititative has successfully contributed to the intended results.
The “upside-down” planning model’s efficiency is evident in Steps 1 through 4—starting with the expected results and quantifying them, defining what behaviors you want the people to exhibit that will produce those results, determining what information or knowledge you will have to provide to get the intended behaviors, and determining what modality the training should be delivered in to get a positive response.
These are some minor concerns I have with the clarity of James and Wendy’s model:
James and Wendy have said that one of the problems with implementing the Four Levels has actually been that instructional designers have attempted to apply the four levels after a program has been developed and delivered. They say it is difficult, if not impossible, to create significant training value that way.
Interestingly, Step 7— “Initiate ongoing reinforcement and monitoring” — deals with what happens after learning events take place. This step actually can represent months or even years of effort. It is where 50% of learning effectiveness occurs and where 70% of the potential learning failures can be prevented if done properly. This is among the most critical steps in the model.
The final issue is, what is the most important take-away from the “complete” model? Is it (1) the upside- down / right-side-up model or (2) is it using the Four Levels as an integral part of every phase of the training program? In my opinion, the answer encompasses the two-pronged planning and data collection approach as well as consideration of all four levels from beginning to end—at every step in the training program design, execution, and measurement.
Barbara
Thank you for this effective explanation. I think this approach of starting with the results that you need rather than the learning is obviously the right way to think about it.
Another way of looking at this is what Dr Bruce C. Aarons A-model which re-works the concepts in a slightly different way – starting with Problem, then Performance then Program. Crucially he starts from the business problem, which the re-working of the Kirkpatrick model also does, but his way seems more together. See http://blog.questionmark.com/a-whistle-stop-tour-round-the-a-model for a brief description of the A-model
John
\”According to Donald L. Kirkpatrick’s revised \’Four Levels of Evaluation\’ model, what we need to do is find out what success looks like in the eyes of these senior managers and stakeholders and let them define their expectations for the training program.\”
This is a fine notion if your primary goal is making customers happy. If you want your training to have a business impact, however, you first have to educate the senior managers and stakeholders on what training can and cannot accomplish. It\’s unreasonable to expect these people to have a deep level of knowledge about training–that\’s why they hire us. There is much in the revision to admire, but I can\’t get behind the idea that we should allow our leaders to define success based on whatever notions they happen to hold about training.