PLANNING FOR ASSESSMENT AT UC SANTA CRUZ

ABSTRACT

This paper provides an overview of the assessment movement and examines the implications for the University of California, Santa Cruz. It was written for campus administrators who are interested in gaining a broad overview of assessment as well as those who have the responsibility for evaluating campus effectiveness. The paper discusses The paper provides a selective bibliography and an appendix that lists the assessment-related accreditation standards that pertain to UCSC.

Planning for Assessment at UC Santa Cruz

The purpose of this paper is to provide an overview of the assessment movement and examine its implications for the University of California, Santa Cruz (UCSC). This paper was written for campus administrators who are interested in gaining a broad overview of assessment as well as those who have the responsibility for evaluating campus effectiveness. It is hoped that this paper will serve as a focal point for campus discussions on planning for assessment.

The paper is divided into six sections. The introduction provides a description of what is meant by the term "assessment," its purpose, what is assessed at colleges and universities, and how assessments are conducted. The second section briefly traces the history of "the assessment movement." Next, the development of assessment in the State of California is described, and assessment activities at UCSC are outlined in the fourth section. The fifth section speculates about the future of the assessment movement. The last section discusses issues that will affect the development of assessment at Santa Cruz and suggests a course of action for the campus. A summary of the assessment-related accreditation standards that pertain to UCSC is listed in the appendix.

Introduction

What is assessment?

In the current jargon of education, assessment is a broad term related to the evaluation of educational effectiveness. In higher education, assessment includes activities such as studies of potential students and non-matriculants; placement and basic skills testing; surveying the educational goals and needs of new and continuing students; learning why students drop-out or transfer; evaluating the need for and effectiveness of student services; teaching and curriculum evaluation; surveys of the community; and surveys of the alumni. Although the primary focus of assessment has been the undergraduate student, evaluation activities now examine graduate students, faculty, and staff.

What is the purpose of assessment?

Assessment activities can be classified by their purpose into two broad types: (a) studies where the information is used to improve a process (formative evaluation), and (b) studies designed to describe final outcomes (summative evaluation). The following table further refines this dichotomy by including the unit of analysis (individual or group). 
                             Purpose of the Assessment                    
  Unit of
 Analysis      Improvement (Formative)       Demonstration (Summative)    

Individual        Diagnosis/Feedback        Certification/"Gatekeeping"   

   Group        Evaluation/Self-study          Accountability/Quality     
                                                     Assurance

The same evaluation question would be approached differently depending on the purpose of the evaluation and unit of analysis. Examples of typical assessment questions for each type of evaluation are listed below.

Diagnosis/Feedback

The purpose of diagnosis/feedback assessments are to provide information to an individual to help improve a process. Examples at a university might include: Evaluation/Self-study

These assessments provide information about group performance and are designed to improve a process. Examples at a university might include:

Certification/"Gatekeeping"

This type of assessment is used to certify that an individual has met a specified criterion. Examples at a university might include: Accountability/Quality Assurance

Assessments of this type are used to certify group performances. Examples at a university might include:

What is typically assessed at colleges and universities?

What is assessed is heavily influenced by the reason assessment is undertaken. For campuses in states with strict assessment guidelines, the main purpose may be to comply with state regulations. For campuses with a mandate to assess, but with the freedom to choose how and what, there is wide variation. Generally, assessment activities can be classified into six broad categories: student knowledge and skills, student values and attitudes, personal growth and adjustment, academic progress, organizational and administrative effectiveness, and teaching effectiveness. For each broad area, the evaluation questions typically examined and the problems associated with their assessment are briefly discussed below.

Student knowledge and skills

This area of assessment is what most people think of when the term is used. It includes evaluation of areas as disparate as entering skills (for the purpose of placement), college-level basic skills, cognitive skills (e.g., critical or diversive thinking), and classroom learning. With the exception of placement testing, evaluation of these areas are often summative, that is, the purpose is to document what has been learned rather than to improve an educational process.

Student values and attitudes

Student attitudes toward diversity and their responsibility as citizens are frequently assessed. As most campuses have broad, general statements of intent to develop well-rounded citizens in their formal mission, it would seem appropriate to evaluate the extent to which this is accomplished. Campuses have found that evaluation of such goals are difficult for three reasons. First, it is difficult to define (and thus, quantify) what is meant by these broad statements. Secondly, there usually is no single course or set of courses in which "citizenship" or related characteristics are an outcome, thus the desired change must necessarily be viewed as the product of the "college experience." As this experience is far from uniform (even at small institutions), it is difficult to evaluate what a student's experience has been and identify why there was a change. Lastly, it is difficult to isolate the effects of maturation and incoming traits from that which is acquired in college (evaluation of the treatment effect). Characteristics such as tolerance for the views of others or appreciation of other cultures are equally problematic. While each represent values that are considered essential in an educated person, no student (and therefore no institution) can be held accountable for what are held to be private views.

Personal growth and adjustment

Several prominent theorists in student development emphasize the evolution of self-identity. It is one of the major developmental tasks of late adolescence and of particular importance for college students from non-traditional groups (e.g., students-of-color, re-entry students, disabled students, students from families of low socio-economic status). Some campuses have elected to assess self-identity and other developmental characteristics, but they face the same problems (cited above) as assessing attitudes. An alternative to evaluating the development of individual students is to implement and evaluate programs that research has found to be effective in aiding student's personal development.

Academic progress

Central to the thinking in assessment several years ago was the notion of "value-added" evaluation. Value-added evaluation attempts to document how much a student learns in a course or a college career after taking into account prior knowledge. This pretest/posttest approach to assessment, while intuitively appealing, has fallen into disfavor among practitioners. The assumption of simple, linear growth is being replaced by a more complex, multidimensional model of student development. When value-added methods are used to evaluate the skills of graduates, defining which skills and knowledge all graduates should possess is difficult and highly political. Once agreed upon, measuring those skills presents yet another set of thorny technical problems. As a method for viewing the totality of effects of college, the few standardized instruments that purport to measure such skills do not correlate highly with student achievement in courses designed to teach those skills. While some states allocate funding based on gains on these tests, this simplistic approach does not take into account the complexities of higher education or the associated problems of measurement. In an alternative approach, New Jersey is currently constructing discipline-based exams that will be administered to all graduates. Other states have shown great interest in these tests. The utility of the new tests and new legislative mandates may keep the idea of value-added assessment alive during the next decade. In the final analysis, such measures can be useful as part of an integrated assessment effort, but they can not be the sole criterion.

Organizational and administrative effectiveness

While student learning and research productivity usually head the list of institutional objectives, the effectiveness of administrative and student service support has also become a focus of assessment. State legislatures, in particular, are increasingly interested in the effectiveness (especially cost effectiveness) of institutions of higher education. Demonstrating and improving the efficiency of non-academic units has also become more salient as accrediting organizations ask institutions to view assessment in terms of campuswide goals rather than just academic goals. Quality of service delivery, work climate, and administrative efficiency are examples of the broad issues that are examined. Instead of treating academic goals as separate, evaluators have recently been investigating how organizational effectiveness influences achievement of academic objectives.

Teaching effectiveness

Most campuses systematically collect student evaluations of teaching. This information is reported to the instructor and is often used in the tenure and promotion process. While the value of this practice is sometimes questioned, it often provides the only information available on teaching effectiveness. Recently there has been more interest in peer evaluation of teaching, but its use is not widespread. The assessment movement has also increased the interest in "classroom research," that is, activities conducted by an instructor to collect information on student learning and teaching effectiveness. These activities are usually of small scale, undocumented, and undertaken solely for the use of the instructor.

How are they assessed?

When the assessment movement began, traditional paper and pencil methods were most common. Typically, these were multiple-choice tests and self-report questionnaires. As the movement evolves, other non-traditional evaluation methods are more widely used. These include qualitative methods, such as naturalistic observation and interviews, and unobtrusive methods, such as the use of institutional records. Capstone courses, use of student portfolios, and ethnographies (observational studies of student cultures) are now being used to supplement the more traditional methods and provide a better picture of the multiple realities of higher education.

After reviewing what information is typically collected and used in assessment, Eva Baker (UCLA) found that the indicator is often confused with the criterion. By this she means that the measures used to evaluate learning have become more important than what students actually learn (this is an important problem in primary and secondary education where student test performance is sometimes used to determine teacher pay). A well-designed evaluation that uses quantitative and qualitative measures would help to address this problem. Qualitative and quantitative methods can complement each other; each can provide information that adds to the understanding provided by the other.

A related problem is the use of standardized evaluation measures for local uses. There is an increasing number of assessment instruments being marketed by testing organizations. Most off-the-shelf assessment measures are summative and cover a wide spectrum of issues. Unfortunately, they usually are too general to focus of the needs of an specific institution and offer too little information about any topic to be useful for improving specific courses. As a result, many institutions are developing their own assessment instruments.

Campuses that monitor students' development require a mechanism for collecting data several times during student's careers. Most universities have entering students take surveys and placement/diagnostic exams as part of orientation activities. Surveys of graduates are also standard procedure on most campuses. To gather information at other times, some campuses have "assessment days" which are devoted to campuswide testing (student motivation to perform well is a problem). Other campuses collect information from a carefully selected group of student volunteers. Longitudinal data bases are used to store the information. No matter how the data is collected, monitoring and analyzing student development requires a significant commitment of resources.

A Brief History of "The Assessment Movement"

Assessment of student learning was an integral part of higher education in the early years of the republic. For example, Harvard University required intensive oral examinations of all its senior students by a board of professors in all subject areas before a student was allowed to graduate. At the turn of the century, the University of Michigan administered very lengthy tests over a wide range of subjects to all of its students to determine if students were learning more as they progressed through college (they did). More recently (1970s), Alverno College and Northeast Missouri State University made student assessment an integral part of their curriculum.

The roots of the national assessment movement in public schools began with several national reports on the quality of elementary and secondary education by the U.S. Department of Education in 1983. Four nationwide policy reports on higher education released in 1984-85 included: Access to Quality Undergraduate Education (Southern Regional Education Board, 1985) which identified skill deficiencies of entering freshmen and led to statewide testing in New Jersey and widespread testing programs in community colleges; Involvement in Learning (National Institute of Education, 1984) which focused on the importance of feedback in improving performance of students and colleges; and Integrity in the College Curriculum (Association of American Colleges, 1985) and To Reclaim a Legacy (Bennett, 1984) which argued that existing institutions are not accountable for their products and called for intensive, integrative demonstrations of student knowledge to complete and certify the process of undergraduate education.

Governmental organizations added to the call for change. The National Governors Conference asked for colleges and universities to develop comprehensive programs to measure student learning. By 1987, two-thirds of state education boards had assessment programs planned or implemented (currently, there are 41). In 1987, the federal government revised the guidelines for the Council on Postsecondary Education, which oversees all regional accreditation organizations, by proposing that accreditation focus on student achievement. Some regional accreditation agencies already had introduced assessment standards (some as early as 1985), and by 1989 all agencies had them.

States have continued to be involved in assessment. In the Indiana and Ohio regents systems, no new programs are approved without provisions for their assessment. Several states, through legislative mandate, require public universities to measure undergraduate achievement with specific instruments at specific times in students' careers. Missouri allocates funding to institutions based on sophomores' improvement over their standardized entrance scores. Tennessee sets aside 5.45% of its educational and general budgets to reward campuses that demonstrate effectiveness. Florida requires all students to pass a state-developed test to earn an AA degree or enroll in upper-division coursework (Texas and Georgia require similar tests).

Assessment in California

Statewide Initiatives

In 1984, the State Legislature commissioned a review of the 1960 Master Plan for Higher Education in California. Among other things, the original plan established the differential missions of the three segments of higher education in the State (UC, CSU, and CC). The 1988 report suggested minor revisions of the 1960 plan, and added a new section on quality of instruction. Specifically, it directs each campus to establish clear curricular goals, match the curricula to them, and evaluate whether the goals are being attained. It further instructs faculty and administrators from the University of California to take the lead in mandating assessment activities through accreditation agencies.

Activity in the State Legislature has centered around bills authored by Assemblyman Tom Hayden. In 1986, Assembly Concurrent Resolution 141 directed the California Postsecondary Education Commission (CPEC) to study "talent development, valued-added, and performance-based budgeting to measuring and improving the quality of education." The CPEC report, Funding Excellence in California Higher Education, recommended six principles to guide State policy:
  1. Incentives to promote quality should be supplementary to institutions' base budget and based on the State's entire system of higher education;
  2. Value-added assessments are of greatest value when linked with other data about students' educational experience;
  3. Funding incentives should reflect the multiple missions of the different segments of higher education;
  4. The primary objective of assessment should be to improve the teaching-learning process and that responsibility should lie with the faculty;
  5. Incentive funding should be phased in over time and reviewed periodically;
  6. The incentive strategy must recognize that assessment of learning is but one characteristic of an effective institution -- it is also important to consider student services, minority recruitment, financial aid, etc.
In 1987, Assembly Bill 2016, sponsored by Assemblyman Hayden, was signed into law. The bill directed CPEC to "develop and present options" for "measuring and implementing talent development or value added approaches to higher education," and "an incentive funding approach designed to develop appropriate methods of assessing the teaching and learning process." The CPEC report, Beyond Assessment: Enhancing the Learning and Development of California's Changing Student Population, recommended that the Governor and Legislature establish a challenge grants program to support initiatives for improving teaching and learning in higher education, including the development of institutional assessment plans.

Assemblyman Hayden continues to introduce new assessment/accountability legislation. Recent efforts have failed due to the budget crisis, but legislative analysts in the UC Office of State Government Relations expect that the bills will eventually be passed.

Community Colleges

Assembly Bill 1725 (1988) requires the Board of Governors of community colleges to report on the development of accountability systems. Systemwide implementation is to commence in 1991-92 and be complete in three years. The work of developing an accountability system was divided into conceptual and operational phases. The conceptual phase is complete and the mandated report outlining the basis of a community college accountability system has been submitted to the Governor and Legislature. Implementation continues with an operational pilot test of the proposed model. The 1990-91 State budget provided $375,000 in local assistance for a pilot test. Grants have been awarded to four districts to produce prototype accountability reports and strategies for improving the quality of student outcomes. The pilot test covers an eighteen month period that began in January, 1991. Major features of the pilot program are:

The California State University

The CSU worked with its campuses to develop a systemwide assessment policy. After two years of consultation with faculty, administrators, and students, the advisory committee proposed a set of principles that have been adopted by the board of trustees. The final report, Student Outcomes Assessment in the California State University: A Report to the Chancellor (1989), outlines the broad principles to be followed that apply regardless of differences in campus governance, mission or political climate. The principles include:

Each campus is charged with developing an assessment plan that is tailored to its institutional objectives. The CSU system has provided funding to support pilot and demonstration projects.

The University of California

In response to a legislative request for a description of progress toward the "adoption and implementation of comprehensive outcomes assessment mechanism," the Office of the President published Report to the Legislature on University of California Progress in Comprehensive Outcomes Assessment: A Response to Supplemental Language to the 1987-88 Budget Act (1988). The report describes how student learning is evaluated in classrooms (e.g., examinations, written assignments) and across campuses (e.g., surveys of employers, graduate schools, and alumni; performance on admissions tests). It details how program effectiveness is assured through regular reviews of the curriculum, teaching effectiveness, and teaching assistant training; and it describes assorted instructional improvement programs. The report also describes the reports and procedures it uses to assure accountability (e.g., annual reports to CPEC, special reports to the legislature, accrediting agencies).

The Academic Senate of the University of California assembled a committee of faculty from across the UC to examine the implications of assessment for the university. In 1988, the Report of the Task Force on Outcomes Assessment was completed. After an examination of strategies for improving quality including testing, expansion of student data bases, improvement of faculty instruction, and use of student evaluations, the report concluded that: In 1990, the UC Office of the President awarded grants to campuses to assess student achievement in lower-division courses. The grants, funded by State lottery monies, were to provide a maximum of $45,000 over a three-year period. The grants were dispersed for the first year, but funding for the subsequent years are, as yet, uncertain.

Western Association of Schools and Colleges

In the Western Association of Schools and Colleges' (WASC) 1988 revision of its accreditation standards, a series of standards on institutional and program quality and effectiveness were introduced. Standard 2.C states the central theme:
The institution has developed the means for developing how well, and in what ways, it is accomplishing its purposes as the basis for broad-based continuous planning and evaluation.
The standard emphasizes institution-wide assessment which includes, but is not limited to, evaluation of student learning. The standard states that assessment should be continuous and integral to the planning process. Standards addressing institutional purposes and planning, educational programs, and student services, also contain assessment requirements. A complete listing of WASC's assessment-related standards are listed in the appendix.

WASC's expectations for institutions that are currently going through the accreditation process include: WASC representatives summarize these criteria by stating that they are looking for "a culture of evidence," that is, the use of assessment information (evidence) should be part of a campus's institutional culture. Of the CSU campuses that have been reviewed by WASC over the past several years, half have received action recommendations for not having met these expectations.

Professional Organizations

Professional organizations throughout higher education have focused more of their activities on assessment. Most notedly, the American Association for Higher Education (AAHE) has sponsored an annual national conference on assessment for the past six years. Other organizations, including the Association for Institutional Research, the American Educational Research Association, and the Society of College and University Planners, have included symposia, papers, and training on assessment. Two years ago, the California Association for Institutional Research, an organization of researchers from all segments of higher education in California, devoted its yearly conference to assessment.

Assessment at UC Santa Cruz

There are a wide range of assessment activities already being conducted at UC Santa Cruz. The following descriptions provide an overview, but are, by no means, complete. To date, there has not been a campuswide summary of assessment.

Assessment in the Boards

Assessment functions in the academic boards include senior comprehensive requirements, narrative evaluations, and program reviews. Information on board-initiated studies on student learning is not centrally collected.

Program reviews are conducted every five years. In preparation for the review, each board receives a charge from the divisional dean based on a standard set of evaluation criteria. The evaluation criteria include the board's curriculum, research, students, budget, facilities, and faculty. The criteria concerning student learning include:

Each review includes at least three external experts in the discipline. To help ensure that the recommendations are carefully considered, the dean submits a progress report to several administrative bodies three years after the review is completed.

Evaluation of Student Services

As the assessment movement has matured, there has been greater interest in evaluating the contribution of student services units. At Santa Cruz, each student service unit undergoes a formal evaluation approximately every five years. A broad survey to evaluate awareness, use, and satisfaction with all student services is conducted every three or four years. In addition, several units collect feedback from students on a regular basis (e.g., counseling, career services) and other units conduct periodic surveys to assist with long-range planning (e.g., housing, financial aid, disabled student services).

Committee Research

Standing committees have conducted several campuswide surveys during the past few years. The Academic Senate's Committee on Educational Progress routinely examines curricular issues. Most recently, the committee examined the scope and quality of the colleges' core courses. With the assistance of the Office of Analysis and Planning, other standing committees have conducted studies on the child care needs of students, faculty, and staff; and on the educational and social climate for gay, lesbian, and bisexual students.

Lower-Division Education Assessment

In 1990, the UC Office of the President awarded grants to be used for evaluation of lower-division education. Santa Cruz's grant was used to fund research in mathematics, world cultures, and writing. Faculty committees were formed in each area to guide the research and staffing was provided by the Office of Analysis and Planning. Although the grant was intended to cover three years, the tentative nature of the funding source led committees to adopt discrete, one-year goals.

Each committee set its own research agenda. To determine whether the quantitative general education requirement was sufficient to guarantee student preparation for upper-division work, the math committee examined how students of different majors met the quantitative requirement. The writing committee focused its efforts on evaluating board-sponsored lower-division courses. Specifically, the committee worked to document the type of writing instruction that is currently being offered in the boards (e.g., the nature of the writing instruction; the amount and type of writing required; the amount, quality, and frequency of feedback; student's need for additional tutoring; and the training of the teaching assistants). The committee on world cultures centered their efforts on a new two-term course in world cultures. The committee worked to identify goals that could be used to evaluate the success of the course.

Subject A Exam

All students must demonstrate a minimum level of writing proficiency prior to enrolling for their fourth term. High school seniors applying to the UC are required to take the Subject A exam if they have not demonstrated sufficient proficiency on other admissions tests. Students not passing the high school exam are required to pass the exam on campus. Students who fail the exam are enrolled in special sections of their core courses or other, specially-designated courses. The Subject A exams are administered and scored by staff in the Writing Program.

Administrative Assessment Activities

The Office of Analysis and Planning conducts a wide range of studies to assist in campus planning. Periodic studies of students include: One-time studies of students have included: a survey on the educational and social atmosphere; Reasons Freshpersons Come To UC Santa Cruz; and Personal Characteristics of Santa Cruz Freshpersons. To assist in monitoring student progress, the office has developed several longitudinal data bases to track students throughout their UCSC career. Other information reported by Analysis and Planning include faculty workload, facilities utilization, and course enrollments.

The Future of Assessment

Based on the history of other movements in higher education, the accountability movement in primary and secondary education, predictions of policy experts on educational trends in the states, and the success of the assessment movement to date, the outlook for assessment in the U.S. and California is discussed below.

Nationwide

A short, one item test:

In the future, assessment will:

The best answer is probably "all of the above," although many institutions are having difficulty demonstrating that their assessment programs have been useful.

At the present time, the assessment movement appears to be leveling off. The initial sound and fury are now well in the past, and many states and campuses are stepping back to evaluate what has been achieved. The most critical issue facing the assessment movement now appears to be effectiveness. The impetus behind the movement was for colleges and universities to demonstrate the how well they were educating students and improve the educational process. In many cases, assessment has not been able to adequately satisfy these accountability demands. Nationwide, critics have seen no improvement in the quality of higher education and are publicly questioning whether campuses should be evaluating themselves. In spite of the serious deficiencies of state-mandated assessment, public policy experts believe that the inability to demonstrate improvement will lead to more legislated programs. If higher education is to head off this legislation, it must be more effective in publicly demonstrating its effectiveness.

In a study by Ewell of states that mandate assessment, assessment was deemed successful at about fifteen percent of the campuses polled. For another fifteen percent, institutions were "stonewalling." For the remaining seventy percent, the results were mixed. In a discussion at this year's AAHE conference, leaders in the assessment movement said that no large, public research university had made a high-level commitment to assessment. Reasons for this are easily found. These institutions are often well-established and prestigious -- unless a state mandates assessment, accreditation agencies have little power to induce change. Research is often rewarded more than teaching, especially for untenured faculty -- unless the reward system is modified, faculty are unlikely to invest their time in the improvement and evaluation of learning. In addition, research on teaching and learning in the discipline is often not viewed as a legitimate area for scholarly research.

The role of regional accreditation agencies will continue to play a critical role in the development of the assessment movement. If institutions find that a cursory hand waving, augmented by favorable survey findings, are sufficient to satisfy the accreditation requirements, meaningful assessment will be left to institutional discretion. If these agencies place assessment in a central role in the accreditation process (as WASC officials anticipate), interest in assessment may continue. Accountability in other areas of higher education, such as the number of courses taught by faculty, graduation rates, or the utility of research, may also be thrust into public awareness.

More sophisticated evaluation designs will also emerge. There will be more research on the net effect of a curriculum and the total educational environment. There will be more research on how students' attitudes and expectations affect learning and the role of the institution in changing those attitudes. Colleges will begin to assess patterns of student development rather than rely on average performance. Finally, there will be more work to move assessment from an activity that is outside the classroom to one that is an integral part of the classroom.

California

Assessment accreditation requirements have been phased in by WASC to give institutions sufficient time to develop programs. The current expectations were cited earlier in this paper. In three to five years, WASC expects that campuses will: A critical element in WASC's assessment of an institution's assessment efforts will be whether the information is used. As UC Santa Cruz will have an accreditation review in four years, these are the broad criteria on which the campus will be evaluated.

There have been several attempts in the legislature to mandate assessment in the University of California. Because the University receives a significant portion of the State's resources, accountability will remain an important issue. Thus, it will be important for the University to be able to demonstrate its effectiveness. Significant initiatives by the University may be required to avoid legislative intervention.

Analysis and Discussion

The following section examines some of the issues that effect how assessments are conducted at UCSC and a plan for action is proposed.

Campus Assessment Issues

Decisions on several key issues will shape what will be attempted and what will be successful at this campus. These issues include: Since there is a limited amount of assessment already taking place, the decision to be made is where this campus wishes to be on the compliance/adoption continuum. The question can be rephrased as: Will this campus continue to perform evaluations only when they are judged necessary or will evaluation become an integral part of all teaching, planning, and administrative functions? If campus efforts are perceived to be aimed mainly at meeting external demands (i.e., WASC), it will be difficult to obtain the participation of faculty and unit heads, and more difficult to obtain useful information. The costs associated with moving forward probably would not justify the external rewards. In 1988, an internal report on assessment by the Analytical Studies Office at UC San Diego concluded that there was enough legislative activity to warrant careful monitoring, but an active development effort was unnecessary. Assessment efforts at Santa Cruz are equivalent to those at San Diego, however there are two important differences: (1) over the past three years, WASC has clearly defined its expectations for research on institutional outcomes; and (2) UCSC's movement toward decentralized governance would benefit from new methods to assure internal accountability. Although both are important, the decision to move ahead must depend mainly on the utility of assessment to the institution and not on external requirements. The conclusions of the San Diego report were probably reasonable at the time, but differences in external demands and internal needs suggest that UCSC should begin to make plans for a wider range of activities.

Who will oversee assessment activities?

At the University of Kentucky at Knoxville, a campus at the forefront in assessment, a new Assistant Vice Chancellor for Assessment was created. More typically, the responsibility falls to the Academic Vice Chancellor, Vice Chancellor for Undergraduate Education, Director of Assessment or Director of Institutional Research. These people are usually responsible for providing departments and committees with guidance, resources, and technical support. Policy issues are often discussed within universitywide committees on assessment.

The success of an assessment program is heavily dependent upon the advocacy of legitimate authority. Without visible support from the highest levels of the administration, broad-based assessment programs falter in the face of faculty resistance.

What resources will be available to support assessment?

The resources assigned to the assessment effort usually reflect the level of institutional commitment. The actual work of assessment is usually assigned to a Director or Coordinator. While some campuses have created a new office, many more campuses have added a position within existing units (often in the office of institutional research). Campuses concerned mainly with compliance have added the workload to existing offices without providing additional staffing. At Santa Cruz, some centralized assessment activities have been assumed by the Office of Analysis and Planning.

During the early years of assessment, grants were available from the federal and state governments. In particular, the Fund for the Improvement of Postsecondary Education (FIPSE) awarded a large number of grants for assessment projects. Now that assessment is no longer viewed as innovative, these sources are less likely to be available (although assessment remains high on FIPSE's list of priorities for the upcoming year). Units that used external funding to develop their programs are now scrambling to secure institutional support. While some states have provided resources to fund mandated programs, most states do not provide the resources to support the additional workload. The UC Office of the President provided small grants to assist campuses to begin assessment of lower-division education, but the funding source (State lottery) is erratic, and probably temporary.

More important than supporting a centralized staff is securing the participation of the faculty. To ensure that faculty are actively involved in research on learning, one campus has a faculty member in each department conduct research on teaching in the discipline. Another college requires candidates for tenure to describe the results of a project designed to improve their teaching. Yet another college requires every graduate student to conduct a study on teaching in their discipline. While these may not prove practical at Santa Cruz, other mechanisms are possible. The resolution on the colleges that recently passed a vote of the Academic Senate committed each faculty member to teach a course for their college or perform related service every three years. It might be worth exploring if assessment or teaching improvement activities could be used to satisfy this requirement (the resolution states the service should be in support of undergraduate education in a board or college). Other possibilities include the use of release time or sabbaticals to support assessment.

For most units, academic or administrative, technical assistance with evaluation design and analysis will be required. Workshops, conferences, and educational materials on assessment can meet some of these needs. Enlisting the help of faculty and staff with training in evaluation may be needed when more specific expertise is required. Creation of research assistantships to assist with assessment is a common strategy (students in education, sociology, or psychology would be the most likely candidates).

How should assessment be implemented?

For those whose experience with evaluation has been threatening, there will be resistance. In our culture, evaluation tends to be summative, and many people are not familiar with using evaluation to improve a process. While no one will argue against improving quality, some would argue that it already exists and no additional efforts are required. Administrative staff might argue that assessment is a non-essential function in an environment where resources are extremely limited. Unit managers may feel evaluation is an insult to their professional expertise and implies that their supervisor believes their unit is doing a poor job. Some faculty will feel that systematic evaluation of student learning is a violation of academic freedom, and, on a more personal level, a threat to their autonomy. Other faculty will feel threatened by being evaluated on something they were not trained to do (teach). A key to overcoming these objections will be a strong, credible advocate; education about the purpose of assessment; demonstrations of goodwill; and, a great deal of patience.

Although the motivation to assess will probably arise from external sources, all evaluation activities should be directed by the administrators and faculty who will use the results. This is a critical requirement for the results to be used for planning or improvement. Advice from outsiders on institutional priorities and evaluation strategies should be encouraged, but evaluations for the purpose of improvement should not be controlled by people outside of the unit being studied.

The most sensible way to make assessment of learning useful is to make it part of each board's five-year review. Centralized research may be appropriate in some areas (e.g., general education), but boards and departments must be the center of activities within the disciplines. The evaluation standards for the five-year review already outline broad areas for student assessment, however these standards do not require evidence of teaching effectiveness or student learning. If each board were required to provide evidence of the skills and competencies of their graduates and show how that information was used to improve the program, then the campus would comply with WASC requirements (this approach is used at UC Santa Barbara and WASC officials were favorably impressed during their recent accreditation review).

Since many academic assessment activities will center on teaching and learning, involvement of the new Teaching Center will be essential. The Center's director should be involved with any training on assessment for faculty or TAs, and be part of the planning and commission of campuswide assessment studies.

How will the effectiveness of assessment be evaluated?

A campus committee is probably the best group to examine whether assessment efforts effectively address the campus's long-range goals. The committee's charge would be to educate the campus about assessment, review assessment strategies, and update the assessment plan when needed. In addition, the committee would be responsible for documenting the use of assessment information for the accreditation purposes. A yearly survey on projects, results, and utilization will meet this need.

What should be assessed at UCSC?

Central to all assessments are institutional goals. Before any efforts to assess effectiveness should be undertaken, decisions must be made about what the institution is trying to accomplish. Where campuswide assessment is a new activity, it is common to start with a revision of the campus's mission statement. The new statement differs from earlier versions in that it attempts to articulate goals that are amenable to evaluation. These statements need not be overly specific, but every department should be able to demonstrate how its activities support the campuswide goals. It is important that the mission statement also address the relative value of the goals. As all goals are not equally important, their relative importance should guide what should be evaluated. Based on these institutional guidelines, academic departments could integrate the goals into their curriculum plans (including statements on pedagogy) for all degree programs and administrative units could relate their mission to campuswide objectives.

Although institutional goals are an important starting place, assessment should also be targeted toward the most exigent issues facing the campus. A campus assessment committee which includes planners and decision-makers from all areas of the campus could address this need. Evaluating the effect of budget reductions (e.g., class size, reduced student services, student academic progress, financial aid, facilities maintenance) is a good example of such an issue.

Assessment should not be viewed as serving a traditional "quality control" function. The University should not be in the business of identifying "defective units," that is, students who are not well-educated. Instead, the emphasis should be placed on continual improvement of educational processes, that is, improving teaching, the curriculum, advising, and student services.

What should be done at UC Santa Cruz?

After considering the changes in external demands, and the direction and impetus for organizational change, the need for two projects are clear: An assessment plan is a WASC requirement. The process of developing a plan will invite campus dialog and guide current efforts. The plan should make explicit the campus's assessment philosophy; define oversight and support responsibilities; review external requirements; specify expectations for academic, student services, and administrative units on what should be evaluated, when, and by whom; identify areas in the infrastructure that need to be developed or modified to support these activities; and suggest implementation timelines and strategies. Specifically, the assessment plan should reflect the uses of data to support the decentralized decision-making processes in the colleges and divisions.

A survey of current practices will be a necessary first step to support the planning process. Comparing the plan to existing practice will identify the areas where assessment activities should be developed. These two projects should provide the information and planning to guide assessment project development and resource planning over the next five years.

Both steps are necessary to coordinate existing assessment activities. For example, the projects funded by grants from by the Office of the President would have benefited from having a clear set of institutional objectives. While the projects were worthwhile, they may have been more useful if the long- and short-term goals for lower-division education had been clearly delineated. Without knowledge of previous or planned projects and a long-term strategy, research to support planning can only be haphazard at best.

Selected References on Assessment

California Postsecondary Education Commission. (1987) Funding excellence in California higher education. Sacramento, CA: Author

California Postsecondary Education Commission. (1988) Beyond assessment: Enhancing the learning and development of California's changing student population. Sacramento, CA: Author

California State University. (1989). Student outcomes assessment in the California State University: A report to the Chancellor.

Evans-Layng, M. (1988). Here come the judges: The assessment movement and its potential impact on UCSD (Environmental Scanning Occasional Paper No. 3). Unpublished manuscript, University of California, San Diego, Campus Planning Office, San Diego, CA.

Ewell, P. (1988). Outcomes, assessment, and academic improvement: In search of usable knowledge. In Smart, J. (Ed.), Higher education: Handbook of theory and research (Vol. IV). New York: Agathon Press, Inc.

Ewell, P. (1991). To capture the ineffable: New forms of assessment in higher education. In Grant, G. (Ed.), Review of research in education (Vol. 17). Washington, D.C.: American Educational Research Association.

Henderson, R. (1988). Report of the task force on outcomes assessment. Berkeley, CA: Assembly of the Academic Senate -- Academic Council, University of California.

Jacobi, M., Astin, A., & Ayala, F. (1987). College student outcomes assessment: A talent development perspective. ASHE-ERIC Higher Education Report No. 7. Washington, D.C.: Association for the Study of Higher Education.

Nichols, J. (1988). Institutional effectiveness and outcomes assessment implementation on campus: A practitioner's handbook. New York: Agathon Press, Inc.

University of California. (1988). Report to the legislature on University of California progress in comprehensive outcomes assessment: A response to supplemental language to the 1987-88 budget act. Berkeley, CA: Office of the President, University of California.

Western Association of Schools and Colleges. (1991). Achieving institutional effectiveness through assessment: A resource manual to support WASC institutions. Oakland, CA: Author.

Appendix

WESTERN ASSOCIATION OF SCHOOLS AND COLLEGES
COMMISSION FOR SENIOR COLLEGES AND UNIVERSITIES

Standards Related to Assessment

STANDARD 2:

INSTITUTIONAL PURPOSES, PLANNING, AND EFFECTIVENESS

Standard 2.A, Clarity of Purposes: The institution is guided by clearly stated purposes that define its character, are appropriate for higher education, and are consistent with Commission standards.

2.A.1: The statement of purposes identifies the broadly based educational objectives the institution seeks to fulfill. In implementing its purposes, the institution has defined the constituencies it intends to serve as well as the parameters under which educational programs can be offered and resources allocated.

Standard 2.B, Institutional Planning: The institution is engaged in ongoing planning to achieve its avowed purposes. Through the planning process, the institution frames questions, seeks answers, analyzes itself, and revises its purposes, policies, and procedures accordingly.

2.B.2: In the planning process, internal and external environmental factors are taken into consideration, institutional data are integrated, and the collection of new data is stimulated.

2.B.3: The planning process identifies and establishes priorities in addressing them.

2.B.6: Appropriate evaluation mechanisms for all major components of the institution are utilized in planning.

Standard 2.C, Institutional Effectiveness: The institution has developed the means for evaluating how well, and in what ways, it is accomplishing its purposes as the basis for broad-based, continuous planning and evaluation.

2.C.1: As an essential element in planning and evaluating institutional effectiveness, institutional research is conducted. Research assesses such elements as instructional programs, research functions, and the co-curricular environment.

2.C.2: Procedures and measures used to evaluate instructional programs may include: changes in students' academic achievement; peer evaluation of educational programs; structured interviews with students and graduates; changes in students' values as measured by standardized instruments or self-reported behavior patterns; pre- and post-testing of students; surveys of recent graduates; surveys of employers of graduates; student scores on standardized examinations or locally constructed examinations; performance of graduates in graduate school; performance of graduates of professional programs on licensure examinations; or the placement of graduates of occupational programs in positions related to their fields of preparation.

2.C.3: Institutions with research and public service missions assess the extent to which support is provided and activities undertaken to accomplish these aspects of their mission.

Suggested Supporting Documentation STANDARD 4:

EDUCATIONAL PROGRAMS

STANDARD 4.A, General Requirements: The achievement and maintenance of quality programs is the primary responsibility of every accredited institution; hence, the evaluation of educational programs and their continuous improvement is an ongoing responsibility. As it analyzes its goals and discovers how conditions and needs change, the institution continually redefines for itself the elements that will result in programs of high quality.

4.A.5: In each field of study, degree objectives are clearly specified: the subject matter to be covered; the intellectual skills and learning methods to be acquired; the affective and creative capabilities to be developed; and, if relevant, the specific career-preparation practices to be mastered.

4.A.6: Efforts are undertaken to develop and implement ways to measure the educational effectiveness of programs.

4.A.8: Courses and programs are planned both for optimal learning and accessible scheduling. Programs offered in concentrated or abbreviated time frames are designed to ensure that courses requiring development of analytical skills allow sufficient time to permit reflective analysis of the material. Where such institutional formats are employed, the institution is under a particular obligation to meet the expectations of 4.A.6.

Standard 4.B, Undergraduate Programs: The undergraduate program is designed to give students a substantial, coherent, and articulated exposure to the major, broad domains of higher education.

Undergraduate studies as a whole have a clear rationale, and the three segments articulate in such ways to reflect that rationale. The undergraduate program, as a whole, is characterized by clarity and order, characteristics which are visible in model curricula in official bulletins, and also in student records, of actual programs pursued.

4.B.2: Undergraduate programs ensure, among other outcomes: (a) competence in written and oral communication; (b) quantitative skills; and (c) the habit of critical analysis of data and argument. In addition to these basic abilities and habits of mind, goals also include an appreciation of cultural diversity.

4.B.4: The general education segment of the undergraduate program is based on a rationale that is clearly articulated, informs the design of all courses, and provides the criteria by which the appropriateness of each courses to the general education segment is evaluated.

4.B.5: In setting the pattern for general education, an institution specifies the minimum number of general education units to be required for all undergraduate students working toward any Bachelor's degree. Where exceptions are provided to special groups of students (e.g., through double counting), they are clearly stated and justified.

4.B.6: General education is integrated with the entire undergraduate program and includes offerings at the upper division level.

4.B.7: The general education program ensures adequate breadth for all students pursuing the Bachelor's degree. Offerings are included that focus on the subject matters and methodologies of the humanities, the natural sciences (including mathematics), and the social sciences; the program may also include courses that focus on the interrelationships between subject matters in these three major disciplinary fields.

4.B.9: The equivalent of two years of study toward the baccalaureate degree will be in the general education and unrestricted electives even if this extends the basic program. Institutions which offer programs that do not meet this standard bear the burden of proof that the tripartite goals of the baccalaureate are otherwise met.

Standard 4.C, Graduate Degree: Graduate and professional programs offer conceptually well designed programs of study which are guided by appropriate and well defined educational objectives, and are supported by sufficient resources to ensure a sophisticated mastery of a complex field of study or a professional area.

4.C.9: Graduate programs provide carefully designed and sensitively monitored curricula and educational experiences appropriate to the level and orientation of the degree.

Standard 4.E, Special Programs and Courses for Credit: All off-campus and other special programs providing academic credit are integral parts of the institution and maintain the same academic standards as regular campus programs. Their functions, goals, and objectives are consonant with those of the institution and lead to academic accomplishments at least equal to those attained by traditional practices. The institution maintains direct quality and fiscal control on all aspects of all programs and provides adequate resources to maintain this quality.

Standard 4.F, Academic Planning: Academic planning is designed to achieve the aims of the institution and provides the rationale for the projected use of currently available and future human, financial, and physical resources. This systematic planning is based on continuing institutional self-evaluation and assessment of the needs of the institution's constituencies. All appropriate segments of the institution are involved in planning.

4.F.4: Human, financial, and physical resources are allocated on the basis of academic program needs and objectives, and are consistent with the academic plans. Resource planning takes into account a realistic assessment of institutional resources and stated goals.

4.F.5: The institution engages in periodic review of program and departmental quality under clearly specified and demonstrably implemented procedures. This process is based on current qualitative and quantitative data which attempt to assess strengths and weaknesses in achieving program purposes and projected outcomes.

4.F.6: Curriculum assessment and planning take into account the role of information technology and the use of computing resources.

Standard 4.G, Non-credit courses and Programs: Non-credit courses (including those offering CEUs) are consistent with the educational purposes of the institution. These courses are characterized by careful planning and high standards of instruction by qualified faculty.

4.G.1: Non-credit courses are administered under appropriate institutional policies and procedures. Campus administrators and faculty are involved in planning, administering, and evaluating non-credit courses.

Standard 4.H, Admissions and Retention: Established admission and retention standards ensure that student qualifications and expectations are compatible with institutional objectives. Admission and retention policies apply equally to students in regular and special degree programs.

4.H.8: Periodic analyses of retention data and graduation rates are undertaken to validate admissions criteria and academic standards. To avoid grade inflation, studies of grading are also made.

Standard 4.I, Academic Credit and Records: Evaluation of student learning or achievement, and the award of credit are based upon clearly stated and distinguishable criteria. Academic records are accurate, secure, comprehensive, and comprehensible.

4.I.1: Criteria used for evaluating student performance and achievement, including those for theses, dissertations, and portfolios, are appropriate to the degree level, clearly stated, and implemented.

4.I.3: Evaluation of student performance and achievement differentiates among levels of quality and among attainments. Where lower and more advanced degree programs are offered in the same field of study, clear differences in level of expectation and requirements are articulated.

Suggested Supporting Documents

1. Documents that demonstrate the appraisal of institutional outcomes: 2. Compilation of Freshman and Entering Graduate Student ability measures.

3. Samples of course examinations and other instruments used to assess student achievement competency and, when possible, available work products determined to be of different levels of quality.

4. Student retention and rate of graduation.

5. Instruments used to measure program effectiveness.

6. Procedures used for program or departmental review.

7. Research demonstrating comparability of courses or programs offered under concentrated or accelerated time frames.

STANDARD 7:

STUDENT SERVICE AND THE CO-CURRICULAR LEARNING ENVIRONMENT

Standard 7.A, Co-Curricular Educational Growth: The institution supports a co-curricular environment that fosters the intellectual and personal development of students. That supportive environment is characterized by a concern for the welfare of all students, on and off campus; a commitment to student academic and self-development; a conscious attention to ethnic, socioeconomic, and religious diversity consistent with institutional purposes; a responsiveness to the special needs of a diverse student body; a regard for the rights and responsibilities of students; and an active understanding of the interdependence of the elements of the learning environment.

7.A.1: the institution systematically identifies the characteristics and learning needs of the student population, including such constituencies as traditional-aged undergraduates, women students, re-entry and older students, student parents, international students, the physically limited and learning disabled, racial and religious minorities, the academically disadvantaged, veterans, and off-campus students such as military students. The institution then makes provision for meeting those identified needs, building and academic community that significantly involves its various populations.

Standard 7.B, Coordination and Administration: The institution has an administrative structure responsible for the overall coordination and administration of the co-curricular program. The institution provides staffing and resources commensurate with its level and size, with its goals for the co-curricular program, and with itsinstitutional purposes.

7.B.2: Arrangements are in place which ensure that students and faculty are involved in the processes of policy development, program evaluation, and planning relevant to the co-curricular learning environment.

Source: Achieving Institutional Effectiveness Through Assessment, Western Accrediting Commission for Senior Colleges and Universities, June, 1991.

This paper was developed by Randy Nelson. Additional copies may be requested by contacting the Office of Analysis and Planning at (408) 459-2446. This paper was released as an internal draft for comment in August, 1991.

Return to the Assessment and Evaluation Page