UC Santa Cruz Skip UC Seal
 2000-2004 WASC Accreditation  

Accreditation Home | Process | Presentation

 
 
Educational Effectiveness Review
Educational Effectiveness Approach
 
line
UC Santa Cruz ... ranked as one of the rising universities leading in the 21st century!

Diamond and Graham, The Rise of American Research Universities, 1997

line

Introduction

Since it opened in 1965, The University of California, Santa Cruz has received national attention. UC Santa Cruz has been acknowledged not only for the excellence of many of its programs but also for the innovative posture of the entire institution.[1]  The team that visited UC Santa Cruz in February 2004 recognized that UC Santa Cruz has achieved much in a relatively short time and with fewer material resources than some of its sister campuses in the UC system.

UC Santa Cruz is now at a critical point in its evolution. Enrollments have increased rapidly in the last decade (see Exhibit A) and are likely to continue to increase, though at a much slower pace. Through this period of growth, we have sought to maintain the high quality of our undergraduate education. We also seek to continue growth in existing graduate programs and the development of graduate programs in new areas. But we are mindful that we must do this in a very challenging economic climate. To meet California’s historical commitment to its citizens, articulated in the Master Plan for Higher Education, UC Santa Cruz must continue to do its share to accommodate the growing number of undergraduate students eligible for enrollment in the University of California, while meeting our parallel responsibilities in graduate education. These goals can be successfully accomplished only if we give serious attention to substantive issues of academic planning and resource management.

UC Santa Cruz welcomes the involvement of WASC in our planning. Believing that the review process should “serve institutional priorities in tandem with accreditation requirements” (WASC Handbook, p. 4), the administration and faculty of UC Santa Cruz welcome the process of self scrutiny facilitated by WASC. We consider the review a genuine opportunity for institutional self-reflection, learning, and development aligned with our institutional priorities and future directions. The review reinforces processes already underway. The background of recent campus strategic planning discussions is presented in Exhibit B.

The present report addresses UC Santa Cruz’s core commitment to educational effectiveness. Throughout the report we develop themes that were first identified in our Institutional Proposal and our Preparatory Review. While focusing on effectiveness in assessing student learning, we seek to be responsive to the concerns and advice articulated by the visiting team and the WASC Commission in the action letter of June 28, 2004 by attending to the framework of institutional review and planning within which we address the educational effectiveness of our programs. In our evidentiary exhibits, we elaborate and substantiate the points that we make in the essays.

Following the format prescribed by WASC, the Educational Effectiveness report is divided into three parts: this introductory section (“Educational Effectiveness Approach”), an in-depth analytic section (which consists of two essays entitled “Undergraduate Engagement” and “Graduate Growth”), and an integrative conclusion (the “Integrative Essay”). A theme that runs throughout the four essays concerns our increasing commitment to what WASC would call “a culture of evidence.” One positive outcome of this review process is the increased attention devoted to the issue of how we can most effectively gather and disseminate data about our students, our courses, and our programs to assure continued and enhanced educational vitality.
 

Our approach to the WASC review

Building on the earlier stages of the WASC review, we have continued to seek broad engagement with the WASC process among the faculty and staff. After the receipt of the visiting team’s report, the WASC Steering Committee created several working groups to develop material to explore the areas indicated by the 2002 proposal: graduate education, program review, and the subtopics in undergraduate academic engagement (first year experience, undergraduate research and internships, capstone experiences, and academic advising). Numerous people at UC Santa Cruz have been involved in the creation of the Education Effectiveness Report. The current Vice Provost and Dean of Undergraduate Education (VPDUE) has served as the Accreditation Liaison Officer (ALO) and led the development of this report. Since June 2004, Interim VPDUE William A. Ladusaw and the Vice Chair of the Academic Senate, Prof. Faye J. Crosby, have served as co-chairs of the campus WASC steering committee. Many members of the administration and of the Academic Senate faculty have served on the steering committee and have contributed to writing portions of the text and portions of the accompanying exhibits. Exhibit C contains a roster of the members of the work groups and steering committee.

During the Fall, 2004, a draft of the report was submitted to the Academic Senate and to all the principal officers of the university for discussion. Following a formal consultative process, seven standing Senate committees provided feedback (as documented in Exhibit D). Comments and further research were provided from administrative officers and staff. The report was then revised and approved by the steering committee. The report as transmitted to WASC was provided to the campus at large to stimulate further discussion in the period leading up to the February 2005 visit by the review committee.

We do not see the WASC review process as a closed or discrete one. It has effectively consolidated debates that arose in reviewing on-going planning processes and stimulated new questions for consideration. We hope that the campus will continue to discuss many of the ideas put forward in the report even after the review team has visited the campus. From the visiting committee we hope to gain valuable perspectives on our goals and some useful suggestions about how to achieve them given our challenging financial realities. We are especially open to suggestions about how we might preserve the best aspects of our pedagogical traditions – including our attention to undergraduate instruction and research and our dedication to diversity and innovation – as we continue to evolve as a first-class research university.
 

Types of data

This review has stimulated a systematic inventory of our institutional research and practices of review relevant to assessing how effectively our programs support student learning.

Inferences about educational effectiveness can be made on the basis of indirect or direct data. Systematic monitoring of retention and graduation rates constitutes indirect indicators of how well we educate our students. UC Santa Cruz annually publishes its graduation and retention rates for undergraduate students. We have routinely tracked these data for various subpopulations. Special attention is given to monitoring retention and graduation rates for the subpopulations of students defined by ethnicity as well as by indices related to admissions eligibility and selection. Entering freshman as well as junior transfer populations are monitored. Our goal is to insure that the retention rates of these subpopulations remain in line with campus averages by providing appropriate learning support and development programs. The annual report has not been able until now to track data on individual programs. After the first phase of implementation of our new academic information system is complete in January 2005, we believe that we will be able to provide academic programs with improved indicators of their success in retaining and graduating their affiliated students. Over the period since our last WASC review, our retention and graduation rates have improved steadily and the rates for minority students have stayed comparable to those for the campus as a whole. Retention and graduation rates are only an indirect measure of educational effectiveness – low rates of retention or graduation signal that the system is not effective, but high rates do not necessarily mean that the system functions well. It is conceivable, after all, that a school could produce many poorly educated graduates.

More direct indicators of educational effectiveness are those that document good outcomes and/or good procedures. Specifically, we can consider the performance of our graduating students on exit examinations (or their substitutes) and can consider the placement of our graduates as outcome measures. The better the outcome, the more certain we are that we are in fact educating our students.

Considering undergraduate programs, outcome indicators are well developed and well disseminated to relevant campus constituencies. All degree-granting undergraduate programs have a senior comprehensive requirement. Each program develops a format for its comprehensive requirement that ensures that its graduates achieve its objectives at an acceptable level. In many cases the comprehensive requirement is satisfied through a capstone seminar, studio, research project or performance. In other cases the students are required to score at an acceptable level on a departmental comprehensive exam or national examination (usually a GRE subject exam in the major area). In most cases, satisfaction of the comprehensive requirement is judged by the faculty of the program independently of the students’ grades in courses. In other cases, the achievement of the capstone expectations is incorporated into curricular requirements through a senior seminar course.

Exhibit E contains the required WASC Inventory of Educational Effectiveness Indicators, including a description of each undergraduate degree-granting program’s senior exit requirement as an example of an indicator used to determine that graduates have achieved the appropriate outcomes for the degree. The contribution of the senior comprehensive requirement to program educational effectiveness has been reviewed by the Academic Senate’s Committee on Educational Policy (CEP). That review is discussed below in the analytical essay on undergraduate academic engagement. An outcome of this review is that all programs have been asked to provide formulations of the educational objectives assessed by their comprehensive requirements.

Other components of the undergraduate program use systematic examinations to assess the outcomes of student learning independently of their course grades or as components of multi-section course sequences.

Notable among these is the approach to assessing satisfaction of the University’s Entry Level Writing Requirement (ELWR). UC administers a statewide writing examination each spring for entering freshmen that is scored by a standard rubric to determine placement in lower-division writing. All entering freshmen at UC Santa Cruz enroll in writing seminars delivered through the residential colleges. Students who have not satisfied the ELWR upon entrance sit for the exam again at the end of their first term to assess the improvement in their writing independently of their grade in the seminar. Historically, the pass rate has ranged between 85 to 90 percent. (Students who do not achieve the outcomes at this stage enroll in supplemental course work that is assessed by portfolio review.) In addition, standard examinations are given in Chemistry, Mathematics, Biology, and the languages to insure that students are placed in foundation courses appropriately or to document their satisfaction of pre-major requirements.

Instruction in courses is routinely assessed through course evaluation forms distributed to the class at the end of each term. Course evaluations are mandatory. Sponsoring departments track the percentage of return for each class. Departments may design their own evaluations or adapt standard models provided by the Academic Senate’s Committee on Teaching through the Center for Teaching Excellence. Course evaluations solicit student assessments of the effectiveness of the course design, instructor performance, as well as their own accomplishments. Data from course evaluations are routinely used in faculty personnel evaluations. They have so far played less of a role in assessing a department’s curriculum, an issue raised by both the Committee on Teaching and the Committee on Educational Policy.

Another way to gauge educational effectiveness is to note what happens to students who receive a UC Santa Cruz education. For our undergraduate alumni, several data sets exist. A comprehensive survey of graduates was published in 1999, covering graduates of 1995 to 1997. It indicated overall satisfaction with their undergraduate education among the 1816 respondents, of whom one in eight were enrolled in a post baccalaureate educational program.

Even taking UC figures as a benchmark, we interpret the relatively high percentage of UC Santa Cruz undergraduates who continue into graduate and professional schools as an indicator of effectiveness. In the most recent comparative study of UC campuses and AAU institutions on the percentage of undergraduate alumni who go on to earn academic doctorates, UC Santa Cruz ranked just behind UC Berkeley (8.5 percent vs. 8.3 percent) and above all other UC campuses. A recent survey of graduating seniors reported that 13 percent had already been accepted to graduate school for the following year and that an additional 34 percent intended to enter graduate school within the next five years. Computer Engineering and Electrical Engineering routinely track the placement of their all of alumni as measures of program effectiveness. UC Santa Cruz’s Alumni Association has recently improved its ability to track alumni, making it possible for programs to draw more systematically on data about their success.

Complementing the data on outcomes are data on the educational process itself. Periodically, external reviews are conducted of every academic department and degree program. Integral to each external review is a visit by a team of well-respected scholars from other institutions who are provided with a detailed self-study by the program or department under review. Observations about staffing, curriculum, and educational support all provide data relevant to program success. In addition, class evaluations, curriculum analyses, and student surveys of satisfaction provide additional information about how well the educational process works in each program. As will be elaborated below, the external review process is becoming increasingly systematic in providing programs with relevant data from institutional research to support their self-studies as well as directing the attention of reviewers to assessing the educational objectives of the programs and their success in monitoring the achievements of their graduates.

Surveys of student experiences and perceptions provide additional data about the educational process. UC Santa Cruz participated in National Survey of Student Engagement (NSSE) in 2000 and 2001, in order to get more direct evidence about the undergraduate student experience. In 2002 and 2003 we participated instead in the newly developed University of California Undergraduate Experience Survey (UCUES), an instrument designed to measure many of the same aspects of undergraduate student engagement specifically within UC. Where possible, we interpreted these data by comparing UC Santa Cruz’s performance to two different benchmarks:

  1. A comparable average measure for UC’s eight undergraduate campuses; and
  2. An average for institutions in our Carnegie classification (Doctoral/Research Universities – Extensive).

The first benchmark compares UC Santa Cruz’s students within a population that meets UC’s uniform standards of eligibility for admission. Using a national benchmark provides a comparison to a wider range of institutions and can help distinguish effects due to the general admission selectivity of the UC system from those that reflect the relative selectivity of UC Santa Cruz within the UC system.

Our experience with the NSSE and UCUES surveys encouraged us to develop a routine survey of graduating seniors in 2003. The content of the survey changes each year to reflect attention to different aspects of the undergraduate program. Data from the spring 2003 survey on advising is discussed below in the undergraduate essay. The results of the spring 2004 survey on satisfaction with major programs are still under analysis.

The data that UC Santa Cruz has collected on its graduate programs has been less extensive and less systematic than that for the undergraduate programs. The analysis of data for individual programs, which is the most meaningful kind, is necessarily based on much smaller populations. Institutional analysis at the graduate level has concentrated primarily on recruitment and yield in graduate program admissions and on planning for graduate student support. One outcome of this review is the realization that we need to develop a suitable inventory of indicators and measures of effectiveness of graduate education, both at the institutional level and at the level of individual programs. As will be discussed in the essay on graduate growth, we have begun to catalog the kinds of information that we already have and consider measures that will help us make efficient use of our resources as we increase the quantity and quality of graduate programs. Revisions in our external review process, discussed below, have already moved in the direction of assembling and disseminating useful data.

Exhibit F provides an overview of the various measures that have been used in the past or are under development for both the undergraduate and graduate populations.
 

External Review of Programs

UC Santa Cruz has long been committed to external reviews of its academic departments. Working in close collaboration with the Academic Senate and with the departments under review, the office of the Vice Provost for Academic Affairs conducts periodic rigorous and comprehensive external reviews of academic departments and programs. Details of the current goals and procedures for external reviews, together with two recent case studies, are provided in Exhibit G.

Because the review process is oriented toward degree programs and departmental units, it does not include regular review of the academic programs of the residential colleges or the campus general education requirements for the undergraduate program. These are under the purview of the Academic Senate’s Committee on Educational Policy (CEP).

The improved current process is more consistent than our earlier process with the emphasis on “a culture of evidence.” Standardization should facilitate the genuine involvement of key Senate committees, especially the Committee on Educational Policy, the Graduate Council, and the Committee on Planning and Budget. Each committee contributes both to the charge for the review and to the interpretation of the outcomes.

The current procedure improves on prior practice in several ways:

  • The process is more timely than before. It allows only one academic year to move from self-study to review closure, ensuring immediate attention to issues.
  • Institutional research now provides a standard set of historical, quantitative data on the characteristics of the department to be used in preparing its self study.
  • Reviews now incorporate student surveys. Undergraduate and graduate students are asked a wide variety of questions about the quality of their learning experience. During their visit, the review team is required to speak to a representative group of undergraduate majors and to a representative group of graduate students.
  • Comparisons across departments and over time are easier now than before due to the standardization of topics to be addressed (scholarship, undergraduate, graduate, and resource issues) and to the standardization of information provided to the program for its self study and gathered through the external review.

The WASC Commission’s action letter (p. 5) has brought to our attention the potential of this improved review process to stimulate programs to attend explicitly to educational effectiveness and assess learning outcomes. This has been incorporated into the proposed standard charge for the external committee. Because of the improvements that have been made to date, we now are poised to ensure that “learning outcomes are incorporated in program review” and that there are “clear institutional processes to identify data needs and quality indicators to support the assessment of student learning.” Of course, without effective publicizing of the results of the assessments, including self assessments, the newly revised procedures will have little effect; but one happy consequence of comparative data is that they invite open discussions.
 

Summary

In this section we have surveyed UC Santa Cruz’s baseline for assessing educational effectiveness at both the institutional and program level. We have omitted discussion of a good deal of institutional research and Senate review devoted to issues such as admissions selectivity (at both the graduate and undergraduate level) and enrollment management, and assessing the effectiveness of campus strategic academic planning and resource management. This narrow focus has been beneficial in showing us where we have opportunities to improve institutional research and to tighten the feedback loops between what that research shows, and how faculty are guided in the management and development of their programs.

While generally lauding UC Santa Cruz as an institution and complimenting the thoroughness of the Preparatory Report, the WASC Commission’s June 2004 action letter recommends that we make clear the evidence on which we base our conclusions and assertions. It also encourages us to forge strong connections between academic planning and budgeting as we explore the balance between undergraduate and graduate education. This encouragement has been reinforced by the Academic Senate consultation. In this introductory essay, we have attempted to specify the evidentiary basis of our reasoning about how we examine the effectiveness of our programs in stimulating and assessing student learning. In the next section, where we deeply engage the issues of educational effectiveness at the undergraduate and graduate levels, we attempt to show some of the links between educational effectiveness and resource management as well.

____________________
[1] Diamond, Nancy, and Hugh Davis Graham. The Rise of American Research Universities. Baltimore, Maryland: The Johns Hopkins University Press, 1997.

 

         

Copyright ©2004 UC Regents | Maintained by susan.jessen@adm.ucsc.edu