Evaluating Social Learning

It seems to me that right now evaluating social learning is a wide open field.  There are people looking at applying the Kirkpatrick model, there are people measuring the use of social learning tools, and there are people talking about something similar to Brinkerhoff’s Success Case Method.

In the spirit of my blog posts on Re-evaluating Evaluation and Revisiting Kirkpatrick, I decided to start with Don Clark―Big Dog, Little Dog blogger―and his take on using Kirkpatrick’s four levels to create and evaluate social learning.

Clark says that the real value of Kirkpatrick’s Four Level Evaluation model is that it lets us take a number of measurements throughout the life span of learning process in order to place a value on it.

Now, Clark is not actually working with the Kirkpatrick’s original four-level evaluation model.  He appears to be using a variation of the Don and James Kirkpatrick revised model. The revised Kirkpatrick model:

1. Starts with the results, letting the stakeholders determine what success looks like and define their expectations
2. Identifies specific metrics to demonstrate and deliver on those expectations
3. Builds a chain of evidence for the results using Levels 1 (Reaction), 2 (Learning), and 3 (Behavior)
4. And finishes again with results—Return on Expectations, or ROE.

Clark calls his variation a “Backwards Planning and Evaluation Model.”  He uses this model to actually build the learning program and then evaluates it by going through these levels in reverse order. It goes like this:

1. Results or impact – What is Our Goal?
2. Performance – What must the performers do to achieve the goal?
3. Learning – What must they learn to be able to perform?
4. Reaction – What needs to be done to engage the learners/performers?

So how does he apply the model to creating and evaluating social learning?

Level 1: Results or Impact- What is Our Goal?

Clark says identifying measurements to evaluate your results helps you zero in on the result or goal you are trying to achieve. He says the best way to start implementing social and informal learning processes is to begin with a focused project and then let demand drive additional initiatives.

As establishing new or improving learning/training processes typically begins with a problem, Clark provides this example:

A manager complains that when he reads the monthly project reports he finds that the same mistakes keep showing up throughout the organization. Because there is no central database where employees can look up information, none of them realizes that their colleagues have had the same problems and documented solutions.

The Training Department designs a central “Lessons Learned” database, establishes social media to connect the people within the organization, and determines the appropriate measurement to be counting the number of problems/mistakes in the project reports each month to see if they improve.

These are the steps for planning and evaluating this project:

It seems that what Clark is saying here is to select a project that solves a measurable problem.  The goal here is to reduce the number of repeated problems/mistakes by 90%. The measurement/evaluation process is to compute the number of problems/mistakes within a timed interval to see if they are reduced.

As an aside, Clark’s Level 1 aligns closely with the revised Kirkpatrick evaluation model.  Kirkpatrick’s Results level also includes defining the goal and determining the metrics.

Level 2: Performance – What must the performers do to achieve the goal?

To achieve the goal of reducing the number of repeated problems/ mistakes, Clark uses an assessment called an After Action Review (AAR).  Much like a retrospective, an AAR is conducted after a project or major activity.  It allows performers to understand what happened and why.  Clark incorporates it into a three-prong solution:

Measurement that involves capturing success stories has some links to the Success Case Method. In this case, interviews are used find out whether the social learning tools are actually helping the performers.  And performers can use tools to evaluate tools so the success stories can videotaped or posted on blogs to help other learners.

The measurement process should also incorporate evaluating the AARs.  That evaluation could include observing some of the AARs and the informal learning that takes place.  It could also include a follow-up survey of the participants to find out what they thought about the effectiveness of their AAR.

Level 3: Learning – What must learn to be able to perform?

Evaluation at this level does not address how well the employees have learned the “Lessons Learned” central database.   He says that measurement is captured in the performance evaluation.  Instead, this level provides performers with what they need to learn so they can use the social learning tools on the job.  Clark says that programs for learning these tools can be eLearning, classroom, social, or informal learning.

These are the steps for planning and evaluating the learning for a project:

What is confusing is that “Creating the Learning Process” is the same as in Level 2: Performance.  And with the exception of the success stories, measurement of what the learners are learning is not captured in that level.  Measuring frequency of use of the tools doesn’t seem to fall into that category.

The difference is that Level 2: Performance seems to address quantification, while Level 3: Learning deals with measuring proficiency and performance.

Level 4: Reaction - What needs to be done to engage the learners/performers?

Although this is the last level when reversing Kirkpatrick’s original four-level evaluation model, it is actually the foundation of the other three. Interestingly, Clark substitutes the word “engagement” for reaction here to get that point across.

The purpose of this level is to ensure the learners are on board with the learning and performance process.   Clark wasn’t to know how engaged the learners will be and how the new skills and knowledge they have learned throughout the course of the project will be of use to them.

To accomplish this, he says it is important to make sure a segment of learners and managers are included in the planning process:

If Clark had examined Don and James Kirkpatrick’s revised model carefully, he would have seen that Kirkpatrick’s Reaction level really is no longer just evaluating whether or not you like a course.  Although this level still measures course, content, instructor, and relevancy to the job, it also communicates a link between quality, process improvement, and action.  At the end of Level 1, Kirkpatrick now recommends a focus group for learners and managers to provide links to his Levels 2 and 3, Learning and Behavior.

What Clark has done with his backward planning and evaluation model, though, is very impressive.  I said that the Kirkpatrick model, measuring the use of social learning tools, and success cases are all being used to evaluate social learning.  It turns out that Clark’s “Using Kirkpatrick’s Four Levels to Create and Evaluate Informal and Social Learning Processes” includes them all.