Clicky

Social Learning Blog

How to Evaluate Learning: Kirkpatrick Model for the 21st Century


I was asked by Wendy Kirkpatrick to remove the copyrighted Kirkpatrick diagrammatic model from my original blog post, How to Evaluate Learning: Kirkpatrick Model for the 21st Century. This revised post includes a step-by-step table as a replacement for that diagram. I think the table actually makes the process easier to understand.

Even though many Learning and Development organizations find it a challenge to prove training’s effect beyond how learners react to the training and whether they have learned the training content, senior management and business stakeholders are more and more interested in metrics that show the bottom line.

According to Donald L. Kirkpatrick’s Revised “Four Levels of Evaluation” model, what we need to do is find out what success looks like in the eyes of these senior managers and stakeholders and let them define their expectations for the training program. Then we need to identify specific metrics to demonstrate and deliver on those expectations.

For those of you who are not familiar with the original Four Levels, this is what they are:

Kirkpatrick’s Four Levels of Training Evaluation

  • Level 1. Reaction: To what degree did the learners react favorably to the training experience?
  • Level 2. Learning: To what degree did the learners acquire the intended knowledge, skills, and attitudes as a result of the training?
  • Level 3. Behavior: To what degree did the learners apply what they learned back on the job?
  • Level 4. Results: To what degree did the targeted outcomes occur as a result of the training experience and follow-up reinforcement?

Kirkpatrick’s revision of the Four Levels starts with defining what the results, or return on expectations should be. It then goes backwards through the four levels in sequence, building a “chain of evidence” with data from all four of the levels. The “chain of evidence” supports the results, showing the value learning and reinforcement has provided to the business. Kirkpatrick calls this Return on Expectations, or ROE.

Kirkpatrick Partners, The One and Only Kirkpatrick Company®—Donald, James, and Wendy Kirkpatrick—continue to carry on the work and revisions to the Kirkpatrick model. In the white paper, The Kirkpatrick Four Levels: A Fresh Look after 50 years 1959-2009, James and Wendy Kirkpatrick take the model into the 21st century.

James and Wendy say the “true,” or “complete,” Kirkpatrick model is really both a planning and evaluation tool. They distinguish between the development of the plan to build, deliver, and evaluate training programs from the actual collection of data for the “chain of evidence.”

What they have done is to present the model upside-down during the planning stage of the training program, starting with the desired Level 4 results. After the program is executed, and evaluation data is collected from each level, the levels are followed in sequential order starting with Level 1.

Their model is divided into two parts:

  • Development of the plan to build effective programs and evaluation methodology (starting with Level 4: Results and working backward to Level 1: Reaction)
  • Collecting data for the chain of evidence (starting with Level 1: Reaction and working forward through Level 4: Results).

Because James and Wendy’s diagrammatic representation is unavailable for use outside the Kirkpatrick Partners programs, I have restructured the process components into a table format, following the order they use.

Rather than the complex diagram components, I use 11 steps to indicate which evaluation level is being addressed by which action. The model starts with planning the training program by identifying the business need as well as the connection between the business and learning. The model continues through a formal evaluation process, ongoing monitoring, and analysis and presentation of the findings.

The “upside-down” planning model’s efficiency is evident in Steps 1 through 4, as it:

  • Starts with the expected results and quantifies them
  • Defines the behaviors you want the people to exhibit to produce those results
  • Determines the information or knowledge you need to provide to get the intended behaviors
  • Determines the modality in which the training should be delivered to get a positive response.

It is in Step 10, however, where I have a minor concern about James and Wendy’s model. Step 10 is the final step to achieving ROE. It is where the findings are presented to the stakeholders as a chain of evidence, starting with Level 1 and moving step-by-step to create the “value story” of the training program through Level 4.

I question why the chain of evidence starts with Level 1 rather than Level 4. Wouldn’t the stakeholders want to start with the results and then move backward through the levels? If the end goal is to provide stakeholders with a return on their expectations, they would want to start with results and change of behaviors back on their jobs.

James and Wendy have said that one of the problems with implementing the Four Levels has actually been that instructional designers have attempted to apply the four levels after a program has been developed and delivered. They say it is difficult, if not impossible, to create significant training value that way.

Interestingly, Step 7— “Initiate ongoing reinforcement and monitoring” — deals with what happens after learning events take place. This step is among the most critical in the model. It actually can represent months or even years of effort. It is where 50% of learning effectiveness occurs and where 70% of the potential learning failures can be prevented if done properly.

The final issue is, what is the most important take-away from the “complete” model? Is it (1) the upside- down/right-side-up model or (2) is it using the Four Levels as an integral part of every phase of the training program? In my opinion, the answer encompasses the two-pronged planning and data collection approach as well as consideration of all four levels from beginning to end—at every step in the training program design, execution, and measurement.

Related Posts Plugin for WordPress, Blogger...
Share what you learned:
I have been in the instructional design and performance improvement field for over twenty years, helping clients find the right solutions and the right consultant fit for their projects.

5 Comments

  • Terry D. Everson

    December 10, 2012, 5:28 pm

    The beauty of being a true practitioner is working with a model that has been a time-tested standard and taking it to the next level. My complements on your edit of the Kirkpatrick work. We did the same thing years ago with Blanchard’s SLII work, taking the concepts to a more practical and operational level. Fun stuff.

  • Ge Lee

    December 11, 2012, 12:23 pm

    Begin with the end in mind?

  • Rich

    December 11, 2012, 2:16 pm

    Your chart is a great job aid for program design and the post succinctly makes a case I have had limited success in making where I work. I’ll be sure to share it!

    In addition to the gloss on step 7 there is an equal gloss on step 1. Whether the goal is to address a problem or an aspiration, we have to know if we are aiming at the right target. Senior stakeholders may know what success looks like in the aggregate but likely do not know what the specific successes are, many levels below, that contribute to that. Ascertainment and diagnosis of the root causes is half the battle. If stakeholders think employees need to know X, but the underlying cause is managers’ behavior Y, then chances of success are low.

  • LM

    January 9, 2013, 9:15 am

    ADDIE…with appropriate attention spent on the Analyze phase? Wait let me package that and put my name on it…then sell it. The deeper you get into the weeds, the newer things look…especially once you reemerge.

  • Chandra

    July 12, 2013, 1:21 am

    Kirkpatrick Partners teach a workshop on how to measure Return on Expectations (ROE), which they deem is a more “holistic” measure than ROI. JJ Phillips dismisses the concept of ROE. You reference ROE but don’t get into the subject. What are your thoughts on ROE? Is it quantifiable?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>