Evaluation must be linked to the strategic plan and changed when the objectives or related activities change. Evaluation is not a one-time effort, but an ongoing process by which all the critical elements of an organization are routinely and systematically reviewed. Evaluation efforts should translate the concepts in the strategic plan to all levels of the organization in terms they understand and must be an acknowledged part of the management process.
Principle II: Know why you are conducting evaluations.
The ultimate purpose of evaluation efforts is to focus the organization on results. There may be several reasons for evaluating a specific program or process and some of these reasons may be at cross-purposes. One shouldn't expect that one type of evaluation will serve all of one's needs. Reasons for doing an evaluation may include:
Principle III: Evaluate systems, not their parts.
This principle applies to both units and individuals. Important processes often require the cooperation of many units. Evaluation efforts should not be limited by organizational boundaries. Similarly, most programs require the cooperation of a variety of individual workers. Individuals have responsibility for only part of the performance of a program and great care should be taken when making inferences about an individual who is part of a larger organization. A unit can not be evaluated by summarizing the performance evaluations of its members.
Principle IV: Don't take the "value" out of evaluation.
"Evaluation" implies that we will place a value on something. "Measurement" is a method by which we assign a numerical value to a property of something. Neither evaluation nor measurement can be completely objective, but evaluation always requires judgment. Performance measures provide an indicator of how something is performing, but someone must judge whether the news is good or bad.
Principle V: Build measurement into all processes.
Minimize the cost and disruption caused by evaluation activities by building measurement into the system. Evaluation as an add-on activity is difficult to sustain.
Principle VI: Be careful what you measure.
Improving one outcome may have adverse impacts on other, equally desirable, outcomes. Don't focus on the measures themselves; numerical goals are often achieved, even when improvement is not. Instead, work to develop a process and culture for using and revising measures to assist employees in focusing on improvement.
Principle VII: Performance measures can not stand alone.
Performance measures (including benchmark data) should be linked to organizational objectives and be part of an overall evaluation plan. These measures should translate organizational goals into concrete outcomes that directly relate to the jobs of individuals (don't use a measure just because it is easy to obtain). Performance measures usually do not have sufficient information to provide diagnostic information and must be supported by other types of evaluation.
Principle VIII: Every measure of performance has its shortcomings.
Different measures provide different perspectives and has different sources of bias. Use multiple sources of information and a variety of methods to evaluate performance (but don't over do it either). Important decisions shouldn't depend on one piece of information.
Principle IX: You can't measure everything all the time.
Evaluation costs time and money. Periodically evaluate all programs and processes, but not at the same time. During an evaluation, identify areas to evaluate that are important and that provide a balanced view (organizational goals, resources, client satisfaction, internal processes). Aim to improve the things which will make a difference and measure what employees can translate into direct corrective action. Sample when data collection is costly, intrusive, or not continually needed.
Principle X: Different audiences need different kinds of information at different times.
Evaluation findings should be routinely reported to all important audiences. The higher the audience in an organizational hierarchy, the greater the need for summarized and overview information (but not to the point that it obscures problems). The timeliness of evaluation information has a great influence on whether it will be used or not (if it isn't used, why bother?) and it has become increasingly important to produce information quickly.
Principle XI: Credibility is crucial.
Perceptions of self-interest can sabotage the credibility of evaluation efforts. Take care to assure that data can not be easily manipulated, and the analyses and evaluators are as fair as possible.
Principle XII: Plan the evaluation with key staff and clients.
The greater the participation in the process of creating an evaluation system, the greater the resulting performance change, and the greater the ease of implementation of future changes based upon the findings. If a process being evaluated crosses conventional organizational boundaries, be sure to include all affected units in the planning effort.
An evaluation plan documents what you plan to do and why. The elements of a plan include the objective of the process or unit, activities designed to accomplish the objective, questions to be answered by the evaluation, methods used to collect data, who will collect the data, who will analyze the data and how, how the information will be reported, who will receive a report on the outcomes of the evaluation, how the information will be used in follow-up activities, and the resources required in the evaluation effort. The plan doesn't need to be elaborate; it should, however, be suffiently descriptive to enable people to understand what will be done. A brief outline of questions that should be answered in each part of the evaluation plan are shown in the "Evaluation Plan Checklist" on the next page.
Principle XIV: Evaluate the evaluation.
Like any other key organizational activity, the evaluation system should be examined for its usefulness and cost effectiveness. Compare the purposes and intended uses of the evaluation findings with what actually happened. Were plans changed or performance improved? Routinely check with the employees to see if the evaluation system is working as intended (acceptance of the measurement and evaluation process is essential to their success as a performance improvement tools). A complete and effective evaluation system will require years of incremental work to achieve.
Principle XV: Evaluation doesn't change systems, feedback and reward do.
Industrial and organizational psychologists have long recognized that when rewards are based on the measurement of an outcome, employees work to maximize that measure. But be careful to make a commitment to improvement-not to measurement.
If you would like to contact me, please feel free to send e-mail to randy@cats.ucsc.edu.
Return to Randy Nelson's Vita
Return to Institutional Research and Policy Studies' Home Page