Simplifying Evaluation
by David Ripley, PhD

We all know that evaluation of performance improvement interventions is important. We need to be able to answer questions like, “Did it work?” “Is it still working?” “If not, why not?” and “How can it be improved?”

Figure 1. The Model: Relationship of Key Evaluation Elements

The Nine Steps
The Planning Phase

NOTE: Steps 1-3 above should be an iterative process that results in a clear purpose and key questions, with key stakeholder buy-in.

NOTE: As was the case with steps 1-3, steps 4-6 need to be considered together. One good way to help stay on track is to make a chart (as shown below) that lists, for each key question, the data required to answer it, how it will be collected, and how it will be analyzed. You are not ready to go further until you can answer those questions.

Data Collection and Analysis

The Execution Phase

To simplify evaluations, stay focused on the key elements, establish a clear purpose, and identify your key questions, and you will have taken a big step toward success. Working on the front-end with key stakeholders will go a long way to gaining their support and helping to pave the way for cooperation in the rest of the organization.

Before gaining his doctorate in HRD and becoming an academic, David Ripley, PhD, spent over 20 years in HR management in Abu Dhabi, Australia, England, Canada, and the United States. He has worked at project, branch office, division, and corporate levels. David is a member of the Management Department faculty at the University of Canterbury in New Zealand and the online adjunct graduate faculty of Boise State University for whom he teaches program evaluation. He may be reached at david.ripley@canterbury.ac.nz.

return to top

Advertisement  Would you like to advertise in this space? Contact marketing@ispi.org return to top

TrendSpotters: The Impact Evaluation Process
by Carol Haig, CPT, and Roger Addison, CPT, EdD

Ingrid Guerra-López, PhD, specializes in performance measurement and evaluation and is our guest this month. Currently an assistant professor in the Instructional Technology program at Wayne State University in Michigan, as well as an associate research professor with the Sonoran Institute of Technology in Sonora, Mexico, Ingrid, iguerra@wayne.edu, recently founded Intelligence Gathering Systems, a consulting firm focused on performance measurement and tracking. Ingrid joins us to expand upon her brief article in March PerformanceXpress and adds the Impact Evaluation Process to the TrendSpotters Open Toolkit.

Genesis of This Process
Ingrid tells us that many of the evaluation models, tools, and applications she has found fail to clearly and directly emphasize results at the various organizational levels, and they do not logically align the functions and steps in the evaluation process. In response, the Impact Evaluation Process evolved from Ingrid’s work in both applied evaluation projects and research. The various levels of results and objectives in the process are derived from Roger Kaufman’s Organizational Elements Model. Generally, performance improvement specialists do not like to do evaluation because they are then accountable if the solution was unsuccessful. Clients have similar fears and use cost and time as an excuse for not evaluating the impact of their decisions. Many evaluation models and tools are filled with intimidating jargon and are particularly discouraging to new users. The Impact Evaluation Process seeks to overcome these obstacles and to promote a “healthier” approach to performance evaluation. Specifically, the Impact Evaluation Process:

• Begins by asking decision makers what they expect from the evaluation, what decisions must be made with the data, and to what results these are tied
• Makes visible the impact of an implemented solution on the organization’s performance and beyond
• Creates a culture of performance tracking in which decisions are based on data relevant to the desired impact decision makers have specified
• Positions evaluation as a decision-making tool

1. Identify Key Stakeholders and Expectations: who will affect or be affected by the evaluation findings and what their expectations are
2. Determine Key Decisions and Objectives: what decisions must be made with the evaluation findings
3. Derive Measurable Indicators: identify key performance indicators and gather data to answer the evaluation questions
4. Identify Data Sources: determine where you will find the data about key performance indicators
5. Select Data Collection Methods: choose appropriate instruments to gather data
6. Perform Data Analysis: select logical approaches to analyze the data found
7. Give Recommendations for Improvement: present recommendations for continuous performance improvement based on the evaluation results

If steps 1, 2, and 3 of the Impact Evaluation Process look familiar it is because they are derived from the steps in gap analysis.

How to Use the Process
As performance improvement practitioners know, an evaluation should be completed for every step in project development. Begin evaluation planning alongside solution selection and development. This helps clarify what we will measure and how we will measure it so that we will know if our solution is working.

Success Story
Ingrid has used the Impact Evaluation Process in various customized forms over the last few years. However, a project she did to evaluate a sales tool is a good example of the processing action, and it illustrates the importance of careful selection of stakeholders.

A client created and implemented a web-based sales tool intended to maximize the sales force’s productivity. Key stakeholders were the sales manager who chose the sales tool, the salespeople, staff members in related departments, and customers. The objective of the evaluation was to determine if the investment in this sales tool was worthwhile. Worth was determined by the following: an increase in sales, a decrease in customer complaints, if profit changes matched the changes in sales production, if customers thought the new tool better met their needs, and other indicators.

Generally, the data showed that customer satisfaction ratings were higher, quoting errors were reduced, and there was an increase in sales volume. However, the sales manager reported that industry-wide sales had also grown during this project’s period, which likely impacted the sales increase. Two decisions were made from this evaluation:

• There was sufficient evidence to support the continued investment in the new tool until further data proved otherwise.
• The measurements taken at this point could serve as a second baseline and be used in a continuous improvement framework.

Postscript: The client has continued to monitor the key performance indicators, using the data to make decisions about various performance issues.

Advice to Users of the Impact Evaluation Process
This process is a guide to thinking, planning, and conducting a performance evaluation. Build your plan around your client’s decision-making needs, and focus on the evaluation objective. Let the process guide your actions while factoring in feedback from the environment. Make the modifications needed to reach the stated objective as you move through the project so you do not discover at the end that you took a wrong turn earlier.

Links to the Performance Technology Landscape
The Impact Evaluation Process supports these principles of Performance Technology:

Application Exercise
Ingrid suggests that the Impact Evaluation Process lends itself to planning an evaluation for a new project as well as revisiting a past project. Choose a project, follow steps 1-7, and see what you can uncover.

And Another Thing…
As we mentioned earlier, Ingrid is affiliated with the Sonoran Institute of Technology. In a special program for 20 doctoral students where the students develop and manage projects for local communities, Ingrid is leading the applied research and evaluation efforts. TrendSpotters hopes to bring you a special interview spotlighting this program in a future issue.

To review past contributions to the TrendSpotters Open Toolkit and find all the models and tools featured in this column, click here.

If you have an HPT model or tool that supports you in your performance improvement activities, contact Carol Haig, CPT, at carolhaig@earthlink.net or http://home.mindspring.com/%7Ecarolhaig, or Roger Addison, CPT, EdD, at roger@ispi.org.

return to top

McGregor Meets Gilbert
by Fred Nickols, CPT

I recently attended an ISPI chapter presentation in which considerable attention was placed on Thomas F. Gilbert’s distinction between two factors affecting performance: (1) the performer’s repertory of behavior and (2) environmental supports. As Gilbert (1978) put it:

“For any given accomplishment, a deficiency in performance always has as its immediate cause a deficiency in a behavior repertory (P), or in the environment that supports the repertory (E), or in both.” (p. 76)

Gilbert further observed of such deficiencies in performance:

“But its ultimate cause will be found in a deficiency of the management system (M).” (p. 76)

Earlier, Douglas McGregor (1967) drew essentially the same distinction when he wrote:

“…the performance P of an individual at work in an industrial organization is a function of certain characteristics of the individual I, including his knowledge, skills, motivation, attitudes and certain aspects of the environmental situation E, including the nature of his job, the rewards associated with his performance, and the leadership provided him.” (p. 5)

McGregor went on to express his remarks in a now-famous notational form, which states simply that performance is a function of individual and environmental variables:

P = f (I a,b,c,d…E m,n,o,p…)

There are some clear implications of the distinctions drawn by McGregor and Gilbert:

1. Training is suited for addressing some individual variables, most notably deficiencies of skill or knowledge.
2. Other individual variables such as capacity, attitudes, and motives are not readily affected via training. Instead, managerial actions tied to recruiting, selection, placement, and incentives are more effective.
3. Environmental variables are addressed via interventions such as communicating expectations, ensuring properly designed processes, and ensuring the availability of required tools and materials.
4. It is typically the case that both individual and environmental variables must be addressed to realize improvements in performance in a particular role or job.
5. Training, therefore, must almost always be accompanied by other interventions to realize improvements in performance.
6. Yet, trainers are regularly expected to realize improvements in performance through training alone. Rarely will that prove to be the case. Moreover, rarely do trainers have the authority to address the environmental factors affecting performance. They can draw these to management’s attention but, as Gilbert indicates, the ultimate responsibility is management’s. As the diagram below illustrates, trainers can take some of the responsibility for learning but management must do the managing.

Figure 1. Responsibilities for Learning

References
Bennis, W. & McGregor, C. (1967). The professional manager, Douglas McGregor. New York: McGraw-Hill.

Gilbert, T.F. (1978). Human competence: Engineering worthy performance. New York: McGraw-Hill. [2007 republished version available through ISPI’s online bookstore.]

Fred Nickols, CPT, is a longtime ISPI member and a frequent contributor to its publications. He maintains a website at www.nickols.us, where many of his articles and papers are available. Fred may be reached at nickols@att.net.

return to top

Advertisement  Would you like to advertise in this space? Contact marketing@ispi.org return to top

Performance Technology Makeovers
by Allison Rossett, CPT, PhD, Column Editor

This past semester I taught a graduate class about performance technology for two classes—one on campus at San Diego State University; the other at a distance. The final assignment, dubbed PT Makeover, asked graduate students to choose a past project that would benefit from some serious PT “magic.” Over the next months, you will get to enjoy how my students reworked their efforts.

As an assignment for a graduate-level course in Educational Technology at SDSU, we were asked to describe a past initiative or project that we would now handle differently with our new performance technology knowledge.

Some time ago I worked as the trainer for a call center. Like many call centers, we had a high attrition rate. We were trying to increase our number of representatives, but it seemed that our new-hire classes (which were repeated every two to three weeks) seemed to only replace people.

The following are two conversations I wish I had with the call center manager and supervisor regarding the high attrition rate. The first describes the issue and the thoughts that were tossed about to fix it. The second describes how I would handle the problem now. Names have been changed.

Peggy: I’m glad you stopped by. Carol’s managed to find 18 new hires for your next class.

Marjorie: Eighteen? I’m not sure how I’m going to handle that many people.

Peggy: We’re pretty desperate. We’re down 12 people right now.

Marjorie: I just finished a new-hire training class.

Peggy: Well Carol hired 12, 10 showed up, 9 finished the class, and now only 7 are still here. I told Carol to step up the job advertising and if we need 10 people to hire 20. We’ve got to get people who are serious about being here.

Sharon: I think we need to stop paying people for the training period too. That new telemarketing company in town is doing that.

Peggy: Yeah, I’m working on a proposal for that.

Marjorie: It seems like we are losing people faster than I can get them through training. Why are they leaving?

Sharon: Well, some are being let go because of demerits. Celia was let go last week. She got sick, was late getting back from breaks a few times, then one of her kids got sick. It’s a shame that it all happened in a month, but those are the rules.

Marjorie: What about the people who are leaving voluntarily?

Peggy: Trina left because she couldn’t wait six months for health care benefits.

Sharon: Jackie left because she was stressed by some of the conversations she was hearing.

Marjorie: Sounds like we definitely have an attrition problem. I don’t think hiring twice as many people as what we need or not paying people for time in training is the answer. Let me think about this and get back to you.

Days later…

Marjorie: I have a few ideas about the new-hire attrition rate we were discussing.

Peggy: Let’s hear it.

Marjorie: First, I think we should try to find the right people for this job and not just getting warm bodies that can type. We need to be more selective. I talked to Carol about her hiring procedure. I asked her what screening she does. The only determinates she has for making a job offer is if they can pass the typing test, have a high school diploma, and how they do in the interview—which she was very vague about. I also did some research. According to this article, call centers have a high turnover rate, but there are some things we can do to improve the hiring process. I think we should start with #5—look at our top performers and determine what it is about them that makes them successful, then come up with interview questions that will help us find similar traits in interviewees. I think we could benefit from #9 as well—being honest about the job, including things like the wait time for benefits, the strict attendance and tardiness policy, and the types of conversations we may hear.

Peggy: I think that’s doable. Keep going.

Marjorie: Second, we need to look at the things that are making people leave. I’d like to have a focus group with a few reps to get some input on what they think the problem is and what would help keep good employees. Basically ask them what they’d do if they were running the place and see if we can incorporate some of their ideas.

Based on what we learn from the focus group, we can look at some current policies and see if there are better ways to get the same results. For instance, I know the reason we have the demerit system is to ensure we have enough people on the floor when we suddenly get busy, but is it the best way?

And maybe there is something we need to change or add to the training I do. Let’s be open to what we may need to change.

Finally, we should calculate what our current attrition rate is, set a goal for what rate would be acceptable, and then celebrate once we get there by having a steak!

References
Foy, D. (2005). Improving agent performance by improving agent selection. Call Center Magazine, 18(11), 38-41.

Morgan, D.L. (1998). The focus group guidebook. Thousand Oaks, CA: SAGE Publications.

Rossett, A. (1999). First thing fast: A handbook for performance analysis. San Francisco: Pfeiffer.

Marjorie Old is a graduate student at San Diego State University and plans to complete her MA in educational technology this month. She works as a clinical instructional designer for ResMed in Poway, CA. She may be reached at marjorieold@hotmail.com.

return to top

CPT News from Around the World

New CPTs!
Last month we listed new CPTs. Unfortunately, one name was omitted. Please welcome Pooja Singh Mehta to the CPT ranks. You can find the complete list at www.certifiedpt.org. We now have over 1,000 CPTs in 23 countries.

If you are interested in becoming a CPT, the next deadline for submitting your application is June 15. Some excellent examples of applications are now on the website along with a self-assessment guide, the reviewers’ standardized checklist, and the application form.

ISPI’s Continuing Outreach Results

ANSI Update
ISPI’s application to begin the process of getting the CPT ANSI accredited was accepted. We are now in phase two of the accreditation process. During this phase, ANSI auditors will review our application and the accompanying documents. Phase three is an on-site audit, during which two of ANSI’s auditors interview staff and review the certification procedures.

ANSI accreditation is program focused. It will look primarily at ISPI’s management system used to administer and maintain the CPT. To get ANSI accredited, ISPI must demonstrate that the CPT certification is balanced and transparent and was built through voluntary consensus, and its assessment methodology follows a due process. If you want to learn more about Standard 17024, the standard which we are working to meet to achieve ANSI accreditation, you can find it at www.ansi.org.

Monthly CPT Calls
Starting this past January, Keith Pew, ISPI’s Director of Sales and Marketing, and Judy Hale, ISPI’s Director of Certification, have been conducting monthly phone calls with people interested in learning more about the CPT. Between 30 and 40 people have called in each month. Here are the answers to the more frequently asked questions:

What are the top reasons for becoming a CPT?

• It is valued by employers.
• CPTs become part of an exclusive group.
• CPTs are known as people who can produce meaningful results under real workplace conditions.
• CPTs have committed to a Code of Ethics.
• CPTs are recognized as people who partner with others and add value in the process.
• CPTs are gaining global recognition for the profession of performance improvement.
• CPTs’ work undergoes peer scrutiny.

What do employers say about the CPT?

• The standards provide us with a benchmark. We now know how good our people are and what they need to be even more effective.
• The standards let us deploy teams on major initiatives with greater confidence.
• Having our people gain the CPT distinguishes us from our competitors.
• We look for the CPT when hiring. It increases the odds that the person can get the results we want and in ways we think appropriate.
• The standards make sense. They balance idealism with pragmatism.

What do individuals say about the CPT?

• I’m proud of being a CPT!
• The standards set me apart from others competing for contracts and jobs.
• Getting it was hard, but worth it! I learned a lot from the process.
• I like being part of a top-notch group!
• My boss’s opinion of me went up when he read my application and realized what I had contributed to the company.

What makes the CPT different?

• It is the only performance- and competency-based credential for our profession. Other credentials are knowledge based.
• It has internal and external integrity.
• Performance and external integrity are assessed through client attestations that the work was of value, considered workplace constraints, followed a systematic process, and was accomplished through collaboration with stakeholders.
• Competency and internal integrity are assessed through a peer review of work descriptions using a standardized checklist.
• Other credentials may have internal integrity as determined by a review of a work product.
• The standards on which it is based were developed mostly by employers—the people who hire us and rely on our services. Other credentials are usually developed by practitioners and academics.
• It is based on demonstrated repeatable performance. Candidates must demonstrate they have satisfied all 10 standards at least three times. Other credentials may require only one example.
• It is international. Other credentials are unique to the United States.

What is required to become a CPT?

• A minimum of three years of experience.
• A commitment to the Code of Ethics.
• A description of three to seven projects demonstrating your ability to satisfy each of the 10 standards at least three times.
• Attestations by employers or clients that the work descriptions you submit accurately reflect the results achieved and how the work was done, and that you did the work.
• Peer review of your descriptions to confirm that you met the standards. Trained reviewers are CPTs who use a standardized checklist to evaluate your work.
• A commitment to recertify every three years by completing a minimum amount of professional development (that may be augmented by volunteer service).

What does it cost?

• $995 for members and $1,195 for non-members.
• If you attend the two-day CPT workshop, your certification fee is pre-paid.

If you know someone who wants to learn more about the CPT, contact Keith@ispi.org to find out when the next monthly call is scheduled.

Your Story
If you have results to share that you think others would value, send them to judy@ispi.org.

return to top

Project Proven Tools and Techniques for ISD: Curriculum Architecture Modules, Events, and Paths
by Guy W. Wallace, CPT, Representative of the Instructional Systems ProComm

The fifth topic that we are covering in this 12-part series is Curriculum Architecture Modules, Events, and Paths. These are instructional design outputs constructed by a facilitated design team, which is a subset of the analysis team, using the four types of analysis data covered in the first four articles in this series:

• Target Audience data
• Performance Model data
• K/S Matrices data
• Existing T&D Assessments data

Using these will make the instructional modules, events, and paths performance based. Curriculum architecture modules, events, and paths are products of a higher-level ISD than that which the ADDIE model typically represents.

A Curriculum Architecture Path is a learning continuum for a job or a job family. The path guides the learner in a lockstep fashion, when and where necessary, or in a flexible manner, when and wherever possible. The path lays out the modular T&D events, deployed in blended and unblended ways. Each T&D event is composed of T&D modules, which are a temporary-design device that I use at this level of ISD.

Later in my ADDIE level of ISD, it is replaced by detailed designs for lessons and their instructional activities (covered next month).

So, modules add up to events, which are presented on a flexible or rigid (or mixed) sequence on a path. Just as chapters add up to books, which are presented on a flexible or rigid (or mixed) sequence on a reading list.

Figure 1. Example T&D Path

A path might be done for each of the primary target audiences, or they might be severed by one path. The path identifies each T&D event by title and provides high-level information, such as whether the events are deemed mandatory, recommended, or elective.

The path itself is often configured into learning phases to facilitate planning focus. Other paths are organized by stages or phases from the major performance cycle for that job, such as the phases of new product development providing the main structure of the path for brand managers. The main thing is that it guides the learners (the performers) and their management in developing an Individual T&D Plan (often referred to elsewhere as an IDP) to support the performance requirements of that individual, accounting for both his or her incoming K/Ss and the specific assignments for the planning period.

The T&D Events
Each T&D event on the path either exists currently, or does not. A Curriculum Architecture Design Document presents the T&D event definitions (Defs) for all of the T&D that exists on that Path.

For those T&D events on the path that do not exist—the gaps—we put a place-marker, a T&D event-specification on the path; it represents gap T&D that is currently being accomplished via on-the-job-training with no structure. It, therefore, still should be considered in planning and then accomplished. As priorities and resources allow, the path will be updated as T&D is built or bought to fill the gaps.

The T&D Modules
The design document includes the gaps at that level, and their T&D modules at the next level of object-design. That is where most of the analysis data ended up—all of the Performance Model data, the K/S data, and some of the existing T&D assessment data. Plus, additional data is added during and after the design process with the design team.

Summary and Close
T&D paths help provide a “visual” of the performance-based learning continuum. The path identifies the T&D events in flexible, rigid, or mixed sequences, for planning guidance, as appropriate for the target audience(s) for that path.

If the T&D path is like a reading list, then the events are the books, and T&D modules are the chapters. All performance based!

Guy W. Wallace, CPT, has been an external ISD and HPT consultant since 1982, is the president of EPPIC Inc., has been a member of ISPI since 1979, is a past president of ISPI, is the author of lean-ISD, and is a recipient of an ISPI 2002 Award of Excellence. He may be reached via guy.wallace@eppic.biz and related resources may be obtained at his website: www.eppic.biz.

return to top

Is ISPI Coming to Your City?

Here’s the schedule of ISPI’s upcoming professional development programs. Visit www.ispi.org for more details or to register.

Certification Workshop: Preparing for the CPT

• July 24-25, 2007: Baltimore, MD
• September 25-26, 2007: Phoenix, AZ

HPT Institute, Principles and Practices of Performance Improvement

• July 24-26, 2007: Baltimore, MD
• September 24-26, 2007: Phoenix, AZ

Fall Conference and Symposium: Achieving Business Results through Performance

• September 27-29, 2007: Phoenix, AZ
• One-Day Workshops: September 26, 2007

Online Certificate Program

• June 13-26, 2007: ROI and Beyond
• July 18-31, 2007: Leadership and Coalition-Building
• August 8-21, 2007: Managing Projects with In-house and Outsourced Teams
• September 26-October 9, 2007: Designing & Modeling Communication Systems and Infrastructures
• October 17-30, 2007: Evaluating Emerging Technologies
• November 7-20, 2007: Needs Analysis and Performance Consulting

Performance Professional Workshop Series

November 6-7, 2007: New Orleans, LA

• Introduction to Serious Performance Consulting, Geary Rummler, CPT, PhD
• Managing Mentoring Processes for Measured Results, Margo Murray, CPT

NEW! November 8-9, 2007: New Orleans, LA

• Six Boxes™ Performance Improvement and Introduction to FluencyBuilding™,
  Carl Binder, CPT, PhD
• Organizational Performance: Focus on Results, Donald Tosti, CPT, PhD

return to top

High Performance Marketing
Successful Marketing as a Performance Consultant: Three Mistakes to Avoid
by Richard F. Gerson, CPT, PhD, CMC

Marketing is both an art and a science. In many cases, it also involves risk taking and gambling. Quite often, performance consultants are unaware as to why their marketing has either succeeded or failed. What we do know, though, is that there are three huge mistakes every marketer makes. Once you overcome these mistakes, you will find all your marketing to be more effective.

Mistake #1: No Unique Selling Proposition (USP)
Every business and performance consultant must have one. A USP tells a client how you are different from a competitor, what makes you special, and why the client should hire you to do training. Having a USP is critical for both external and internal performance consultants. It sets you apart, provides an irresistible offer to a potential client, and motivates the client to learn more about how you can help him or her.

The simplest way to develop a USP is to find out what you do better than everyone else, and then describe it. For example, can you create a customized intervention program positively, absolutely overnight? Or, are you so dependable that like a good neighbor, you will always be there? You get the idea.

Your USP defines you and separates you from the crowd.

Mistake #2: Failure to Test Your Marketing
Too many marketers come up with what they think is a great marketing or advertising idea and then run headlong to implement it. This can be a direct mail campaign, an email campaign, print advertising, radio, television, or anything else you can think of. Performance consultants, just like other marketers, are also guilty of this.

Before you roll out a marketing program and spend your hard-earned money, test the program on a small number of prospects. Find out how they respond to your programs and offers before implementing the larger campaign. If you get a large number of positive responses, you are on the right track. If the prospects are indifferent to your marketing messages, then something must be changed before a larger roll-out occurs. If there is one thing you should remember about your marketing, it is to TEST, TEST, TEST.

Mistake #3: Treating All Customers and Clients Alike
This mistake actually has two parts. The first is that we think we actually know who our customers really are. The second is that we treat every customer the same.

You will never totally know all your customers. Your goal is to learn as much as you can about your customers and develop your interventions so they are specific to each client's needs. This leads directly to the fact that you must treat each client differently because clients all have different needs and requirements. One size does not fit all. That is why you have to learn as much as you can about your customers, develop customized approaches for dealing with them, and then create and deliver programs that solve their problems.

Other Marketing Mistakes
There are many other marketing mistakes a performance consultant can make. Some of these include not marketing all the time, forgetting to sell and deliver what marketing promises, neglecting to maintain marketing or advertising efforts that are working simply because you are personally tired of them, and many more. We have a hard enough time convincing people about what we do and why it is valuable for them. Do not compound this with any of these marketing mistakes. Avoid the problem by:

1. Creating a powerful USP
2. Testing all your marketing and advertising efforts and messages for effectiveness
3. Treating each prospect, client, and customer as different from all others

Richard Gerson, CPT, PhD, is president of Gerson Goodson Inc., a performance consulting firm located in Clearwater, Florida. He is the author of 22 books, with six books on marketing, sales, and customer service, along with several other books on performance management and performance consulting. He may be reached at richard.gerson@richgerson.com.

return to top

Listening to Learners: Systematic Feedback Creates a Cascade of Positive Effects
by Garry DeRose, PhD, and Lora L. Snow

In the traditional classroom, all information flows from the instructor to the learner. However, if you shift focus from the teacher to the student, interesting and valuable things begin to happen. Gathering feedback from learners can set in motion a cascade of positive events from ongoing upgrades in the curriculum to better learner motivation and morale and, ultimately, more effective performance.

Acquiring feedback should be done systematically throughout each training cycle. Even before classes begin, find out how prepared each individual is to understand and use the information he or she will receive. That will guide your presentation of the material, indicating where extra emphasis, coaching, or an e-module may be needed.

Second, draw out learners about real conditions where they work, including obstacles they may encounter. This feedback can help the training address real-world issues and reduce or eliminate barriers.

Next, as the learning proceeds, use exercises to measure progress in gaining skills. This can show you where curriculum changes or additions may get the information across more effectively.

These are among the steps we have taken in a successful corporate intervention that has turned out more than 800 skilled project managers in little more than five years.

As a result of feedback, project management learners now bring current projects right into the classroom and move the work ahead while they are acquiring skills. We also urge collaboration with classroom peers from other units and disciplines, and this cross-feedback can become a key element in the learning process.

The learners are also encouraged to develop individual growth plans and take additional courses as needed. The feedback process even extends into the post-course period in follow-up consultations with the instructor.

Feedback about the organization has also been important, including information about supports such as internal subject matter experts and specialized software. In addition, we have identified organization obstacles such as inadequate planning time and mid-project personnel changes.

Results of our five-year program have been highly positive. For example, corporate objectives were altered to clarify project priorities, a new course segment on Managing Without Authority was added, and more emphasis was given to post-course coaching and e-learning.

Feedback has led to steady incremental improvement in the curriculum, enhancing its effectiveness and contributing to its unusual popularity. The program has also added clarity and prestige to the project management career ladder, raised morale among project managers, and built enthusiasm for such assignments by demonstrating that leadership pays attention to issues raised by learners.

Focusing on learners brings yet another important benefit. As learners take a more proactive role in the training, they tend to become more involved in achieving unit and corporate goals and often return to their units motivated to identify needed improvements and bring them to leadership.

This case history has involved only project management training, but it seems quite likely that the principles it demonstrates may also be applicable to those learning other skills.

Garry J. DeRose is director and CEO of the Corning Center For Learning (CCFL). He holds a BA from Montclair (NJ) State College, an MBA from Syracuse University, and an MA and PhD from University of Wisconsin. Lora L. Snow is director of development at CCFL. She provided essential research and other assistance in the preparation of this article.

return to top

ISPI Lifetime Membership: Benefits for a Lifetime

Have you considered ISPI’s Lifetime Membership? If you have not, here is why you should. Lifetime Membership is straightforward and simple and makes the most sense—you, the member, make a one-time, lump-sum payment to ISPI and become a member for life and never pay dues again. Any member can apply; however, to receive a discount, the member must be in good standing with continuous years of membership. See the chart below.

SAVE Up to 33% on a Lifetime Membership

The amount of the payment depends on the member's tenure with ISPI. Sign up now! Become a Lifetime Member and NEVER PAY DUES AGAIN; plus, retain all member benefits, avoid future dues increases, receive lifetime discounts on ISPI products and services, never miss a member benefit because of a lapse in payment, and contribute to building your performance improvement community of the future. For complete details on the lifetime membership, click here, or download a copy of the brochure by clicking here.

ISPI congratulates its Lifetime Members:

return to top

Call for Proposals:
Evidence-Based Practice Initiative

With the growth of the performance technology field, the practice of performance technology is outstripping its theory and research base (Stolovitch, 2000). The misbalance is cause for concern. Brethower (2000) states that the connections between theory, research, and practice, which require data collection specific to relevant questions, adds value to HPT. Clark and Estes (2000) state that the evidence strongly suggests that too many of our popular performance technology solutions do not work, and hence we do not have an effective technology. The authors suggest that ISPI must support the development of new science-based, authentic performance technology, by using solutions that are firmly based on scientific evidence.

Of course, there is also a flip side. Not only is it important to identify practices supported by evidence and those that have been shown to be ineffective, but also those practices that have not yet been tested. The absence of evidence proves nothing, other than, perhaps, “we don’t know.” Just because there is little or no evidence, does not mean the practice is ineffective. Reciprocally, just because a technique has received considerable “research press” does not necessarily make it better than techniques that have not been tested.

It is important that both HPT researchers and practitioners be fully informed about best available evidence, the strengths and limitations of different sources of evidence, and the strengths and limitations of our expertise, as applied to each situational-specific context. Recently, the American Psychological Association (APA) convened a Presidential Task Force (PTF) to examine sources of evidence and the integration of evidence and practice. The APA-PTF (2005) found that best practices reflect the integration of multiple sources of evidence, everything from research, to practitioner’s experiences, to situational context. Hence, we must not only draw from the entire range of research approaches, but even more widely, from the range of evidence sources.

Furthermore, decisions or practices based upon what we know today will become outdated. Because our evidence-base is dynamic, our practices and decisions change as we bring new evidence to light. Because what we designate as evidence will determine (DeAngelis, 2005): (1) our practices and thereby the outcomes we effect; (2) what is researched; (3) what is funded; and (4) what we teach in academic curricula.

It is critical that we acknowledge the reciprocal linkages between evidence and practice. We need common processes for evidence-based practices; guidelines for making evidence-based decisions, and the infrastructure to accumulate, mine, analyze, and synthesize evidence across all sources. Edwards, Dattilio, and Bromley (2004) not only provide a compelling case for case-based reports as an integral source of evidence, but also offer recommendations for closing the rigor gap, such as collaboration between practitioners and researchers, systematically documenting and summarizing case information, and relating cases to theory to build case law or grounded theory. Furthermore, standards of reporting that maximize the utility of case-based reports as reliable and valid sources of evidence are needed. Currently, cases tend to be reported without regard to rigor of format, organization, or content. Such haphazard reporting prohibits the accumulation, analysis, and synthesis of this potential goldmine of evidence.

While the goals of practitioners may seem different than those of researchers, there is considerable overlap in purpose and methodology. Both practitioners and researchers are (or should be) concerned with answering important questions in the context of solving important problems (Guerra-López, 2007). To do this, both practitioners and researchers must depend on the best available evidence (Thomas, 2006). Although practitioners may not realize that research is feasible in their practices, Foshay (2000) affirms that it is possible. Both quantitative and qualitative methods are useful to the practice of performance technology. Reciprocally, there is room available for researchers’ efforts to incorporate real-world applications.

Purpose of Proposed Research
The ISPI Research Committee, in collaboration with the ISPI Science and Research Professional Community, is calling for proposals that efficiently and effectively target the issues discussed above. Proposed research should study both research and practices so as to better explicate researcher and practitioner behaviors and protocols with regard to the use of evidence as integral to performance improvement efforts. The deadline to submit is June 8, 2007. Click here to download the guidelines.

return to top

Performance Marketplace

Performance Marketplace is a convenient way to exchange information of interest to the performance improvement community. Take a few moments each month to scan the listings for important new events, publications, services, and employment opportunities. To post information for our readers, contact ISPI Director of Marketing, Keith Pew at keithp@ispi.org or 301.587.8570.

return to top

ISPI Membership: Join or Renew Today!

Are you working to improve workplace performance? Then ISPI membership is your key to professional development through education, certification, networking, and professional affinity programs.

If you are already a member, we thank you for your support. If you have been considering membership or are about to renew, there is no better time to join ISPI. To apply for membership or renew, simply click here.

return to top

Newsletter Submission Guidelines

ISPI is looking for Human Performance Technology (HPT) articles (approximately 500 words and not previously published) for PerformanceXpress that bridge the gap from research to practice (please, no product or service promotion is permitted). Below are a few examples of the article formats that can be used:

• Short “I wish I had thought of that” Articles
• Practical Application Articles
• The Application of HPT
• Success Stories

In addition to the article, please include a short bio (2-3 lines) and a contact e-mail address. All submissions should be sent to april@ispi.org. Each article will be reviewed by one of ISPI’s on-staff HPT experts, and the author will be contacted if it is accepted for publication. If you have any further questions, please contact april@ispi.org.

return to top

About PerformanceXpress

Feel free to forward ISPI’s PerformanceXpress newsletter to your colleagues or anyone you think may benefit from the information. If you are reading someone else’s PerformanceXpress, send your complete contact information to april@ispi.org, and you will be added to the PerformanceXpress emailing list.

PerformanceXpress is an ISPI member benefit designed to build community, stimulate discussion, and keep you informed of the Society’s activities and events. This newsletter is published monthly and will be emailed to you at the beginning of each month.

If you have any questions or comments, please contact April Davis, ISPI’s Associate Executive Director, at april@ispi.org.

ISPI
1400 Spring Street, Suite 260
Silver Spring, MD 20910 USA
Phone: 301.587.8570
Fax: 301.587.8573
info@ispi.org
http://www.ispi.org

return to top

turn to top