I was asked by Wendy Kirkpatrick to remove the copyrighted Kirkpatrick diagrammatic model from my original blog post, How to Evaluate Learning: Kirkpatrick Model for the 21st Century. This revised post includes a step-by-step table as a replacement for that diagram. I think the table actually makes the process easier to understand.
Even though many Learning and Development organizations find it a challenge to prove training’s effect beyond how learners react to the training and whether they have learned the training content, senior management and business stakeholders are more and more interested in metrics that show the bottom line.
According to Donald L. Kirkpatrick’s Revised “Four Levels of Evaluation” model, what we need to do is find out what success looks like in the eyes of these senior managers and stakeholders and let them define their expectations for the training program. Then we need to identify specific metrics to demonstrate and deliver on those expectations.
For those of you who are not familiar with the original Four Levels, this is what they are:
- Level 1. Reaction: To what degree did the learners react favorably to the training experience?
- Level 2. Learning: To what degree did the learners acquire the intended knowledge, skills, and attitudes as a result of the training?
- Level 3. Behavior: To what degree did the learners apply what they learned back on the job?
- Level 4. Results: To what degree did the targeted outcomes occur as a result of the training experience and follow-up reinforcement?
Kirkpatrick’s revision of the Four Levels starts with defining what the results, or return on expectations should be. It then goes backwards through the four levels in sequence, building a “chain of evidence” with data from all four of the levels. The “chain of evidence” supports the results, showing the value learning and reinforcement has provided to the business. Kirkpatrick calls this Return on Expectations, or ROE.
Kirkpatrick Partners, The One and Only Kirkpatrick Company®—Donald, James, and Wendy Kirkpatrick—continue to carry on the work and revisions to the Kirkpatrick model. In the white paper, The Kirkpatrick Four Levels: A Fresh Look after 50 years 1959-2009, James and Wendy Kirkpatrick take the model into the 21st century.
James and Wendy say the “true,” or “complete,” Kirkpatrick model is really both a planning and evaluation tool. They distinguish between the development of the plan to build, deliver, and evaluate training programs from the actual collection of data for the “chain of evidence.”
What they have done is to present the model upside-down during the planning stage of the training program, starting with the desired Level 4 results. After the program is executed, and evaluation data is collected from each level, the levels are followed in sequential order starting with Level 1.
Their model is divided into two parts:
- Development of the planto build effective programs and evaluation methodology (starting with Level 4: Results and working backward to Level 1: Reaction)
- Collecting data for the chain of evidence (starting with Level 1: Reaction and working forward through Level 4: Results).
Because James and Wendy’s diagrammatic representation is unavailable for use outside the Kirkpatrick Partners programs, I have restructured the process components into a table format, following the order they use.
Rather than the complex diagram components, I use 11 steps to indicate which evaluation level is being addressed by which action. The model starts with planning the training program by identifying the business need as well as the connection between the business and learning. The model continues through a formal evaluation process, ongoing monitoring, and analysis and presentation of the findings.
The “upside-down” planning model’s efficiency is evident in Steps 1 through 4, as it:
- Starts with the expected results and quantifies them
- Defines the behaviors you want the people to exhibit to produce those results
- Determines the information or knowledge you need to provide to get the intended behaviors
- Determines the modality in which the training should be delivered to get a positive response.
It is in Step 10, however, where I have a minor concern about James and Wendy’s model. Step 10 is the final step to achieving ROE. It is where the findings are presented to the stakeholders as a chain of evidence, starting with Level 1 and moving step-by-step to create the “value story” of the training program through Level 4.
I question why the chain of evidence starts with Level 1 rather than Level 4. Wouldn’t the stakeholders want to start with the results and then move backward through the levels? If the end goal is to provide stakeholders with a return on their expectations, they would want to start with results and change of behaviors back on their jobs.
James and Wendy have said that one of the problems with implementing the Four Levels has actually been that instructional designers have attempted to apply the four levels after a program has been developed and delivered. They say it is difficult, if not impossible, to create significant training value that way.
Interestingly, Step 7— “Initiate ongoing reinforcement and monitoring” — deals with what happens after learning events take place. This step is among the most critical in the model. It actually can represent months or even years of effort. It is where 50% of learning effectiveness occurs and where 70% of the potential learning failures can be prevented if done properly.
The final issue is, what is the most important take-away from the “complete” model? Is it (1) the upside- down/right-side-up model or (2) is it using the Four Levels as an integral part of every phase of the training program? In my opinion, the answer encompasses the two-pronged planning and data collection approach as well as consideration of all four levels from beginning to end—at every step in the training program design, execution, and measurement.