When we consider training evaluation, Kirkpatrick's Four Levels of Evaluation tends to be the go-to method. This method ends the evaluation process with "Results." Did the targeted outcomes actually occur? Jack Phillips first introduced ROI to this model, as Level 5. Calcuating ROI in training has long been an elusive quest. Philips suggests that it can be done by compiling the Level 4 data, converting it to monetary values and then comparing those values with the cost of creating the program itself.
The Phillips Evaluation Model looks like this:
Phillips suggests that when a training program is implemented, a chain of impact should occur as skills and knowledge (Level 2) are learned and applied on the job (Level 3) to produce business impact (Level 4), ultimately ending in ROI.
At each level, measurements are taken to determine whether the business impact was actually a result of the program. When negative ROI exists, it should be easy to identify which link in the chain was broken. For example, did the participants not learn (Level 2) or were they not able to successfully apply the new learning on the job (Level 3)?
Calculating ROI is not an easy process, but here's a step-by-step example of Jack Phillip’s process for calculating ROI from the Leopard Learning Blog.
Here's how it works:
Step 1. Collect Pre-Program Data
The first step is to collect pre-program data as a baseline measure in order to compare metrics before and after training. Here's how this might work:
A company needed to reduce customer complaint numbers and costs. A decision was made to provide elearning to customer service personnel on how to handle the complaints. Data was collected from the previous six months to identify any trends and to compare post-program costs.
Step 2. Collect Post-Program Data
The next step is to collect data once training is completed. Data sources used could be:
- Organizational Performance Records
- Participants’ Subordinates
- Team/Peer Group
- Participants’ Supervisors
- Internal/External Groups
There are two common methods of data gathering: extracting it from existing organization performance records or conducting followup questionnaires.
Step 3. Isolate the Effects of the Learning Program
Then determine whether results discovered are actually due to the training program. Identify key factors that might have contributed to the performance improvement, gathering information from:
- Client leadership
- Program analyst and developers
- Program participants
- Participants’ supervisors
These are techniques used to approximate the impact of the training program:
- Control groups
- Trend line analysis
- Forecasting methods
- Participant estimate of training’s impact
- Supervisor’s estimate of training’s impact
- Customer’s estimate of training’s impact
- Expert’s estimate of training
These techniques can isolate the business impact of training from other potential sources of business improvement such as employee bonus programs, competitive environment, marketing programs, and seasonal effects that temporarily contribute to business improvements.
Step 4. Convert Data to Monetary Benefits
After isolating the effect of the program, you can now convert the data to monetary values and compare it to the program costs. To do this, you need to place a value on each data item connected with the program. You should have collected hard (objective) data and soft (subjective data). Here's how to convert it:
Step 5. Tabulate Program Costs
After you collect the program data and convert it to monetary values, calculate the costs of the training program, including:
- Specific costs related to the needs assessment and evaluation
- The cost to design and develop the program
- The cost of program materials provided to participants
- The cost for instructors, including preparation and delivery time
- The cost of facilities for the training program
- Travel, lodging, and meal costs for participants
- Salaries plus employee benefits of participants who attend training
- Administrative and overhead costs of the training function, allocated in some convenient way
In the customer complaint example, the total costs were $15,200.
Step 6. Calculate the Return
Then, insert the values into the following ROI formula:
In the customer complaints example, the net program benefits are: $24,615 (total value of improvement) - $15,200 (total program costs)/ $15,200 (total program costs) x 100. The result is an ROI of: 61.9%.
The ROI process is highly complex and sensitive. These are a few precautions:
- Only use the ROI process for programs where a needs assessment has been conducted.
- The ROI analysis should always include one or more strategies for isolating the effects of training.
- Use only the most reliable and credible sources when making estimates.
- Take a conservative approach to calculate benefits and costs.
- Be cautious in comparing the ROI in training and development with other financial returns.
- Include management in developing the return.
- Be cautious in approaching sensitive and controversial issues (that is, what is measurable and what isn’t).
Step 7. Identify Intangible Measures
The final step is identifying the intangible, non-monetary benefits associated with the training program. These are the benefits that it is impossible or too costly to quantify.
For some programs—such as interpersonal skills training, team development, leadership, and management development—intangible, non-monetary benefits are extremely valuable and may carry as much influence as the hard data items.
Identifying intangible benefits can be done during the needs assessment, through discussions with the client or sponsor, during the attempt to convert the data to monetary values, or during a follow-up evaluation.
Typical intangible variables include such items as:
Phillips says, “In practice, every project or program, regardless of its nature, scope, and content will have intangible measures associated with it. The challenge is to efficiently identify and report them.”
Evaluating ROI is complicated and time consuming. Phillips suggests that organizations should implement ROI impact studies selectively, on only 5 to 10% of their training program, since it is both expensive and resource intensive.
He recommends that training measurement and evaluation focus on an individual program or a few tightly integrated courses. For organizations implementing the ROI concept for the first time, only one course should be selected for a calculation as part of the ROI learning curve. In the final analysis, the selection of programs for ROI calculations should yield a level of sampling where top management is comfortable in its assessment of the training and development functions.
As companies become more interested in metrics, hopefully stakeholders will realize the overall benefit of the ROI evaluation and accept the time commitment required of their training staff.