More on Re-evaluating Evaluation – Jack Phillips and ROI
I have been blogging a lot about Training Evaluation this year—mostly Kirkpatrick, but also Brinkerhoff and Scriven. I just realized that I haven’t included a single word about Jack Phillips, who introduced Return on Investment (ROI) as Level 5 to Kirkpatrick’s Four Levels of Evaluation.
My first exposure to Phillips’ ROI—although I didn’t realize it at the time—was through a colleague who introduced me to Kirkpatrick’s Four Levels. Her four levels were Level 1: Reaction, Level 2: Learning, Level 3: Behavior, and Level 4: ROI. And for a long time, that is what I thought the four levels were. Years later, I realized that she had skipped Kirkpatrick’s Level 4: Results and substituted Phillips’ ROI.
So what exactly is ROI and, as Level 5, what does it measure? First of all, it builds on Level 4: Results, which are the business results linked to the training program. According to Phillips, ROI compares these business results to the costs of the training program. Level 5 is developed by collecting Level 4 data, converting the data to monetary values, and comparing those values with the cost of the program in order to provide the return on training investment.
The actual Phillips Evaluation Model looks like this:
Phillips says that when a training program is implemented, it should create a chain of impact at several levels, ultimately ending in ROI. A chain of impact needs to occur as skills and knowledge (Level 2) are learned and applied on the job (Level 3) to produce business impact (Level 4).
At each level, measurements need to be taken to determine whether the business impact was actually caused by the program. If there is a negative ROI, it should be easy to identify which link in the chain was broken. For example, did the participants not learn (Level 2) or were they not able to successfully apply the new learning on the job (Level 3)?
Now, you should know that calculating ROI is not an easy process, but I found a great step-by-step example of Jack Phillip’s process for calculating ROI from the Leopard Learning Blog.
Step 1. Collect Pre-Program Data
The first step in the process is to collect pre-program data as a baseline measure so you can compare metrics before and after training. Here is an example that shows how this can work:
A company needed to reduce customer complaint numbers and costs. A decision was made to provide training to customer service personnel on how to handle the complaints. Data was collected from the previous six months to identify any trends and be able to compare post-program costs.
Step 2. Collect Post-Program Data
The next step is to collect data after the training program is completed. These are the major types of data sources that can be used:
There are two common methods to get the data:
- Extract from existing organization performance records
- Conduct follow-up questionnaires
Step 3. Isolating the Effects of the Learning Program
Of course, then you have to make sure that any results you are measuring are actually based on the training program. So you have to identify the key factors that might have contributed to the performance improvement, getting information from:
These are techniques that can be used to approximate the impact of the training program:
- Control groups
- Trend line analysis
- Forecasting methods
- Participant estimate of training’s impact
- Supervisor’s estimate of training’s impact
- Customer’s estimate of training’s impact
- Expert’s estimate of training
These techniques can isolate the business impact of training from other potential sources of business improvement such as employee bonus programs, competitive environment, marketing programs, and seasonal effects that temporarily contribute to business improvements.
Step 4. Convert Data to Monetary Benefits
After isolating the effect of the program, you can now convert the data that was collected to monetary values and compare it to the program costs. To do this, you need to place a value on each data item connected with the program. You should have collected hard (objective) data and soft (subjective data). This is what you need to do to convert the hard and soft data to monetary values:
Step 5. Tabulate Program Costs
After you collect the program data and convert it to monetary values, you need to calculate the costs of the training program. This involves monitoring all of the related costs of the program targeted for the ROI calculation:
- Specific costs related to the needs assessment and evaluation
- The cost to design and develop the program (prorated over the expected life of the program)
- The cost of all program materials provided to participants
- The cost for the instructors, including preparation and delivery time
- The cost of facilities for the training program
- Travel, lodging, and meal costs for participants
- Salaries plus employee benefits of participants who attend training
- Administrative and overhead costs of the training function, allocated in some convenient way
In the customer complaint example, the total costs were $15,200.
Step 6. Calculate the Return
At this point you have collected the data on the learning program benefits, converted them to monetary values, and tabulated all of the program costs. Now, you just need to put the values into the ROI formula. This is the formula:
In the customer complaints example, the net program benefits are: $24,615 (total value of improvement) – $15,200 (total program costs)/ $15,200 (total program costs) x 100. The result is an ROI of: 61.9%.
The ROI process is highly complex and sensitive. These are a few precautions:
- Only use the ROI process for programs where a needs assessment has been conducted.
- The ROI analysis should always include one or more strategies for isolating the effects of training.
- Use only the most reliable and credible sources when making estimates.
- Take a conservative approach to calculate benefits and costs.
- Be cautious in comparing the ROI in training and development with other financial returns.
- Include management in developing the return.
- Be cautious in approaching sensitive and controversial issues (that is, what is measurable and what isn’t).
Step 7. Identify Intangible Measures
The final step in measuring learning is identifying the intangible, non-monetary benefits associated with the training program. These are the benefits that it is impossible or too costly to quantify.
For some programs—such as interpersonal skills training, team development, leadership, and management development—intangible, non-monetary benefits are extremely valuable and may carry as much influence as the hard data items.
These are the four possibilities for identifying intangible benefits:
- During the needs assessment
- Through discussions with the client or sponsor
- During the attempt to convert the data to monetary values
- During a follow-up evaluation
Typical intangible variables include such items as:
Phillips says, “In practice, every project or program, regardless of its nature, scope, and content will have intangible measures associated with it. The challenge is to efficiently identify and report them.”
Evaluation of ROI Evaluation
Evaluating ROI is extremely complicated and time consuming. Phillips does say that organizations should consider measuring learning and implementing ROI impact studies very selectively on only 5 to 10% of their training program. He acknowledges that otherwise it becomes incredibly expensive and resource intensive.
He recommends that training measurement and evaluation focus on an individual program or a few tightly integrated courses. For organizations implementing the ROI concept for the first time, he suggests that only one course should be selected for a calculation as part of the ROI learning curve. In the final analysis, the selection of programs for ROI calculations should yield a level of sampling where top management is comfortable in its assessment of the training and development functions.
More and more companies are becoming interested in metrics. My question is whether stakeholders will realize the overall benefit of the ROI evaluation to accept the time commitment required of their training staff. In that regard, I am still a Brinkerhoff’s Success Case fan.