What metrics are you using to measure workforce effectiveness today?

[Steve Bonadio]

Hi Folks. First discussion post in this new forum as we gear up for my webinar on March 12 to discuss Workforce Analytics. Thought I'd get the discussion started.

So, what are your favorite metrics? Many of my company's (Softscape) customers are starting to look beyond traditional HR metrics like time-to-hire and retention. One area that's getting a lot of attention is the impact of learning on performance. What are you seeing, and how is your organization embracing more "meaningful" workforce metrics?

[Robert Zotti]

At Stevens Institute of Technology, I'm in the process of gathering data for all of our graduate courses, including number of students in each class, outcomes (grades), end-of course student surveys, and the use of real-time web conferencing or recorded lectures. I'm particulaly interested in any differences between our online courses and our conventional classroom courses. I'm digging though 3 years of this data, and hope to compare it with other schools as well.

[Tim Martin]

At Intel, we're looking at workforce performance in terms of factory performance. We have 2 primary groups for our technical training - technicians and engineers. Technicians are responsible for equipment operation and maintenance, so their performance is linked to equipment performance. We have processes and tools to evaluate technician proficiency based on speed (time to complete scheduled maintenance), quality (through post-maintenance monitors), and effectiveness (time to next equipment failure). Engineers are responsible for trouble-shooting and technology transfers, so their performance is related to problem solving and knowledge transfer. This has been more difficult to measure through factory metrics, so more subjective follow-up surveys have been used. It is the challenge of measuring tacit vs implicit knowledge.
We are moving further from traditional training or HR measures but closer to meaningful business measures.

[Susan Chai]

At Kaiser Permanente, I am directly involved with gathering and analyzing key training metrics. To measure learning application and follow-through, we use the Fort Hill Company's tool, Friday5s. Using this tool we gather Kirkpatrick level 3 evaluations. Once the follow-through period ends we identify key course participants and conduct one on one interviews to gather ROI data. From these one on one sessions we develop both monitary and non monitary results that we use to present to Senior Learning Leaders and Senior Leaders at Kaiser.

[Steve Bonadio]

Very interesting comments. I'm seeing a much bigger push - due in part I would imagine to the economy - by HR orgs to better link their efforts to bottom-line business outcomes and develop more quantitative measures for success. Looks like both Intel and Kaiser are thinking along the same lines.

But I'm also finding that developing concrete financially-oriented metrics remains elusive for most companies. So how are you doing it? Is it engaging the CFO’s office and its analytic staff to align HR activities to actual corporate costs and revenues? What other strategies are you employing to expose the financial impact of HR?

[Susan Chai]

At Kaiser Permanente in the National Human Resources Learning and Development organization, we have blended the best of Phillips and Brinkerhoff to develop an internal process. Basically, it consists of several steps. The first step is analyzing participant's follow-through activities (we are using Fort Hill's Friday5's tool to gather this data.) After we review each course and the participant's response, we determine the level of follow-through and sort the participants into three groups, High, Mid and Low. We conduct interviews with participants from each group. After our interview process we use key metrics to relate the data findings to ROI (monetary) and non monetary findings. The next step is to create the Follow-Through report where we summarize our key findings. This report is passed to the senior leaders. At this point, I am not sure if it is making its way to our CFO's office. Sending the data to the CFO and aligning improvements throughout the organization (via corporate goal aligment) that ultimately improve safety, morale, quality and reduce costs (Hoshin Kanri) would be the ultimate goal of our metrics process. However, we are relatively young in our metrics process.

[Tim Martin]

Our technical training structure for manufacturing consists of a central group reporting to technology development and site groups reporting to the factory managers. This aligns us directly to business strategies and needs. We work closely with Finance to validate bottom-line results and with Operations to address performance issues (people and equipment capabilities).

Three years ago, consultants were brought in for a sweeping evaluation of efficiency across the entire corporation. They followed the money and one conclusion that they came to is that technical training had the opportunity to drive critical improvements and lower costs. Since then we have implemented a number of programs and measured our progress. Our key learning was to be bolder and act as our own consultants going forward. These aggressive actions also better prepared us for the current environment.

[Susan Chai]

Hi Tim, did you retrain your trainers to act as the consultants?
Tim Martin said:

Our technical training structure for manufacturing consists of a central group reporting to technology development and site groups reporting to the factory managers. This aligns us directly to business strategies and needs. We work closely with Finance to validate bottom-line results and with Operations to address performance issues (people and equipment capabilities).

Three years ago, consultants were brought in for a sweeping evaluation of efficiency across the entire corporation. They followed the money and one conclusion that they came to is that technical training had the opportunity to drive critical improvements and lower costs. Since then we have implemented a number of programs and measured our progress. Our key learning was to be bolder and act as our own consultants going forward. These aggressive actions also better prepared us for the current environment.

[Jay Krupp]

At Newmarket International, I have been active for the past 9 years measuring the impact of our educational offerings to our customers through technical support call volume. For example, when we release a new Delphi eLearning course, I measure the number of "how do I calls" into technical support relating to content in the course. I can easily see if a customer purchased the course, so I compare those that purchased and those that did not to the call volume. If we see an uptick in HDI calls on a certain area of our software, we may create an eLearning course on that topic. We also use this same technique to revise our instructional design in existing courses. While the impact we see internally is a decrease in support calls - which shortens answer/hold times, the increase in effeciency is really at our customers facility. Their sales people are selling more effectively, not fighting with how do I things in the technology.

[Art Drake]

At AIG, we are revamping the performance management model for evaluating our project management effectiveness. At the Global Program Management Office, rather than focusing on the "who" (the individual project/program manager), we are establishing clear and qualtifiable metrics around the "what" (the outcomes/performance of both individual projects and the larger project portfolio). These metrics are linked to key PM competencies and/or stated policies. So, when we do not see the desired outcome or performance, we first focus on the project (organizatinal impact) and then assess down to individual PM and PMO performance to identify potentail gaps. This approach has allowed us to focus our "learning" strategy and effort on outcomes. The PMs still have their personal annual review, but we can now enhance that discussion with performance metrics. Having been a PM, its not about about shooting the messenger, but rather helping the messenge to become more bulletproof.