eLearning Blog

Evaluating Social Learning

Evaluating social learning is a wide open field. There are people attempting to apply the Kirkpatrick model, people measuring the use of social learning tools, and people using something similar to Brinkerhoff’s Success Case Method.

Blogger Don Clark says that the real value of using Kirkpatrick's Four Level Evaluation model to evaluate social learning is that it lets us take a number of measurements throughout the life span of learning process in order to determine value.

Clark is using a variation of the Don and James Kirkpatrick revised model, which is:

  1. Start with the results, let stakeholders determine what success looks like and define their expectations
  2. Identify specific metrics to demonstrate and deliver on those expectations
  3. Build a chain of evidence for the results using Levels 1 (Reaction), 2 (Learning), and 3 (Behavior)
  4. Finish again with results—Return on Expectations, or ROE.

Clark calls his variation the “Backwards Planning and Evaluation Model.” He uses this model to actually build the learning program and evaluates it by going through the levels in reverse order, like this:

  1. Results or Impact - What is our goal?
  2. Performance - What must the performers do to achieve the goal?
  3. Learning - What must they learn to be able to perform?
  4. Reaction - What needs to be done to engage the learners/performers?

So how does he apply the model to creating and evaluating social learning?

Level 1: Results or Impact- What is our goal?

Tips_For_Evaluating_Social_Learning.png

Clark says identifying measurements to evaluate your results helps zero in on the goal you are trying to achieve. The best way to start implementing social and informal learning processes is to begin with a focused project and then let demand drive additional initiatives.

He provides this example:

A manager complains that when he reads the monthly project reports he finds that the same mistakes keep showing up throughout the organization. Because there is no central database where employees can look up information, none of them realizes that their colleagues have had the same problems and documented solutions.

The Training Department designs a central “Lessons Learned” database, establishes social media to connect the people within the organization, and determines the appropriate measurement to be counting the number of problems/mistakes in the project reports each month to see if they improve.

These are the steps for planning and evaluating this project:

Creating the
Learning Process
  • Set up a process like a central database that allows the employees to capture “Lessons Learned” so that others may also learn from them when similar problems arise.
  • Include a means of connecting the people in the organization with each other
  • Let people to tweet their problems to get help and tweet their solutions to help others.
Measuring the Learning Process
  • Reduce number of repeated problems/mistakes by 90%.

 

It seems that what Clark is saying here is to select a project that solves a measurable problem. The goal here is to reduce the number of repeated problems/mistakes by 90%. The measurement/evaluation process is to compute the number of problems/mistakes within a timed interval to see if they are reduced.

Level 2: Performance - What must the performers do to achieve the goal?

To achieve the goal of reducing the number of repeated problems/ mistakes, Clark uses an assessment called an After Action Review (AAR). Much like a retrospective, an AAR is conducted after a project or major activity. It allows performers to understand what happened and why. Clark incorporates it into a three-prong solution:

Creating the
Learning Process
  1. Identify and capture “Lessons Learned” in an After Action Review, or AAR.
  2. Post them on a wiki or SharePoint.
  3. Search the wiki or Tweet using Yammer or Twitter to see if there is a previous solution to their problem.
Measuring the Learning Process
  • Count the contributions daily, weekly, or monthly to the wiki to determine if expectations are being met:-Contributions to the wiki or SharePoint
    -Questions asked on Yammer or Twitter
  • Interview learners to capture success stories on using the tools.

 

Evaluating_Social_Learning-1.png

Measurement that involves capturing success stories has some links to the Success Case Method. In this case, interviews are used find out whether the social learning tools are actually helping the performers. And performers can use tools to evaluate tools so the success stories can videotaped or posted on blogs to help other learners.

The measurement process should also incorporate evaluating the AARs. That evaluation could include observing some of the AARs and the informal learning that takes place. It could also include a follow-up survey of the participants to find out what they thought about the effectiveness of their AAR.

Level 3: Learning - What must learn to be able to perform?

Evaluation at this level does not address how well the employees have learned the “Lessons Learned” central database. He says that measurement is captured in the performance evaluation. Instead, this level provides performers with what they need to learn so they can use the social learning tools on the job. Clark says that programs for learning these tools can be eLearning, classroom, social, or informal learning.

These are the steps for planning and evaluating the learning for a project:

Creating the
Learning Process
  • Perform an AAR.
  • Upload the captured “Lessons Learned” to a wiki.
  • Search and find documents in a wiki that are similar to their problem.
  • Microblog in Yammer.
Measuring the Learning Process
  • Measure AAR proficiency by using it in an eLearning program and performing an actual AAR in a classroom environment.
  • Measure proficiency in using the wiki and Yammer in their respective eLearning program (multiple choice) and by interacting (social learning) with the instructor and other learners on Yammer.

 

What is confusing is that “Creating the Learning Process” is the same as in Level 2: Performance. And with the exception of the success stories, measurement of what the learners are learning is not captured in that level. Measuring frequency of use of the tools doesn’t seem to fall into that category.

The difference is that Level 2: Performance seems to address quantification, while Level 3: Learning deals with measuring proficiency and performance.

Level 4: Reaction - What needs to be done to engage the learners/performers?

How_To_Evaluate_Social_Learning.png

Although this is the last level when reversing Kirkpatrick's original four-level evaluation model, it is actually the foundation of the other three. Interestingly, Clark substitutes the word “engagement” for reaction here to get that point across.

The purpose of this level is to ensure the learners are on board with the learning and performance process. Clark wasn’t to know how engaged the learners will be and how the new skills and knowledge they have learned throughout the course of the project will be of use to them.

To accomplish this, he says it is important to make sure a segment of learners and managers are included in the planning process:

Creating the
Learning Process
  • Learners are brought in on the planning/building process to ensure it meets their needs.
  • Managers meet with the learners on a one-on-one basis before they begin the learning process to make sure the program is relevant to their needs.
  • Instructional staff meets with the learners during the learning process to make sure it is meeting their needs.
  • Managers, with help from the learning department, meet with the performers to ensure the new process is not conflicting with their daily working environment.
Measuring the Learning Process
  • During the weekly project meetings, learner/performer engagement problems/roadblocks are the first item discussed and solved.

 

If Clark had examined Don and James Kirkpatrick’s revised model carefully, he would have seen that Kirkpatrick’s Reaction level really is no longer just evaluating whether or not you like a course. Although this level still measures course, content, instructor, and relevancy to the job, it also communicates a link between quality, process improvement, and action. At the end of Level 1, Kirkpatrick now recommends a focus group for learners and managers to provide links to his Levels 2 and 3, Learning and Behavior.

What Clark has done with his backward planning and evaluation model, though, is very impressive. I said that the Kirkpatrick model, measuring the use of social learning tools, and success cases are all being used to evaluate social learning. It turns out that Clark’s “Using Kirkpatrick’s Four Levels to Create and Evaluate Informal and Social Learning Processes” includes them all.