Transforming Program Review from Compliance to Strategic Development
Introduction
Program review has historically been a compliance-driven activity, often met with skepticism by institutional stakeholders. Questions such as “Why do we need to be reviewed?” and “Will this review lead to any actionable change?” reflect this prevailing sentiment of doubt. Some hesitated to provide complete evaluations, fearing potential repercussions for program faculty and staff. However, recent discussions indicate a paradigm shift in the role of program review. This shift involves a fundamental change in the assumptions and methodologies used in evaluation. While institutions, especially community colleges, once approached program review as a compliance task, many now see it as a critical process for fostering learning improvement. Understanding how program review has evolved to meet new educational demands and accountability standards is essential for shaping its future role.
Reframing program review
A comprehensive, effective program review provides evidence of student learning. Eggleston (2020) defines program review as “a comprehensive look at individual programs that cover staffing, curriculum, learning outcomes, action plans, etc. They often involve an external reviewer, and usually highlight the importance of assessment” (p. 4). Bers (2011) states that program review is “a type of evaluation of an instructional support, student service, or administrative program, department, or unit. Note that this definition covers virtually all departments within an institution, not just instructional programs” (p. 63-64). Regardless of the definition, a thorough evaluation using a detailed and inclusive approach is needed to support the delivery of locally responsive, cost-effective, high-quality programs and services across the institution.
Objectives of program review include fulfilling accreditation and state requirements and assuring institutional quality to students, faculty, staff, parents, alumni, and other stakeholders. Program review can also generate planning and continuous improvement and encourage strategic development. For many colleges, program review is an opportunity:
- To reflect on the successes and challenges of academic programs.
- To develop a future direction for the program.
- To consider how improvements can be made to enhance student learning and success.
- To gauge success and design improvement actions to ensure the quality of academic and student success programs and administrative services.
A comprehensive program review encourages institutional stakeholders to take a "big picture" look at their programs. It offers a chance to reflect on program achievements and growth opportunities and determine needs. Comprehensive program reviews also provide essential information for a college’s long-range planning process. Program review sustains and strengthens departments' contributions and student learning and success. The process and results offer opportunities for reflection and support strategic campus-level planning and decision-making related to instructional programming and academic support services, including program/service revisions and program closures.
Moving beyond a checkmark
The program review process must be reconsidered not as an exercise for compliance but as one that has deep value and meaning for the everyday operations of a program and the overall success of an institution. Program reviews that move beyond compliance are planned with a published schedule for all department chairs and program directors so that they know which programs will be reviewed and when. Additionally, program reviews that become a tool for student empowerment and success are systematic, transparent, and consistent. Quality program review processes that advance equitable student outcomes are those that begin with rigorous self-study.
Moving past compliance requires stakeholders to be introspective and provide candid evaluations of the current situation and future while engaging in constructive dialogue to discuss and design actions to improve growth areas, with an eye on taking advantage of future opportunities and innovation. It is critical that those collecting data work to address inequities to support program improvement.
High-quality program reviews demonstrating “more than just completing the task” are evaluative rather than descriptive. It is not enough for the report to describe the program's operation; it requires an appraisal with due recognition of challenges and achievements. Being critical and courageous in recognizing and writing through the lens of self-assessment is the most significant activity in this process of moving past compliance. Three areas—identifying stakeholders, identifying questions, and utilizing data—can help shift perspectives of program review.
A. Identifying Stakeholders
As colleges identify members of the review team, it is important to include diverse and deeply connected members of the campus community who work closely with the student populations being served by the department. Using an equitable lens, these community members are better positioned to speak to the successes and challenges of the review area compared to those who might have minimal interactions with students utilizing the services. For instance, faculty members should be encouraged to collaborate in data collection with the institutional research staff. It is also critical to encourage students to participate in the review team. Student voices are integral when examining the effectiveness of postsecondary programs and services, especially in the community college sector (Thrill, 2019).
When completing the program review process, department leaders are encouraged to review their institutions’ mission statements and determine how their programs and services are aligned. This practice helps ensure fairness and uniformity on campus. Some institutional mission statements may emphasize diversity, equity, and inclusion, but on-campus programs and services may fail to implement them. Therefore, the review team needs to think critically about how the types of programs and services offered will further these areas of the institutional mission.
B. Identifying Questions
When conducting a thorough program review that advances a college's campus-level planning and decisions, it is critical to identify and inquire about a program's goals, curriculum, student learning outcomes, quality, cost, and improvement. Identifying consequential questions allows one to explore areas overlooked in a program's daily operations. This practice also leads to discovering a high-level perspective of opportunities and challenges.
The questions asked by the program review team should include the following:
Program Goals and Alignment: What are the program's stated goals and objectives? Are the program goals clearly defined, measurable, and achievable?
Curriculum and Instruction: Is the curriculum relevant to current industry trends and professional needs? What source defines the industry trends, and can this source be referenced for future use? How well is the curriculum structured and sequenced? Are teaching methods and practices diverse and engaging, and do they incorporate effective pedagogical practices?
Student Learning Outcomes: What are the program's specific student-learning outcomes? When were the SLOs last reviewed or evaluated? How are student-learning outcomes assessed and measured?
Faculty: What are the department/program’s efforts to ensure instructional effectiveness? What are the standards for credentialing instructors? Are training-/professional-development opportunities provided to faculty?
Student Satisfaction and Engagement: What are students' perceptions of the quality of instruction, course content, and learning environment? What opportunities exist for student engagement outside the classroom (clubs, organizations, research opportunities)?
Career Preparedness: What is the program placement rate for graduates? What career-service supports are available to students?
When evaluating programs, institutions should also consider how the program addresses potential challenges related to budget constraints or enrollment fluctuations. This involves assessing whether the program has strategies to manage financial or recruitment challenges effectively. Examining how the program engages with external stakeholders, such as industry partners and professional organizations, is essential to building meaningful connections that enhance program quality and relevance. Another critical aspect is the program's ability to adapt to emerging trends and innovations, ensuring it remains current and responsive to evolving professional and industry demands.
In identifying evaluation questions, it is essential to determine their focus and purpose. Evaluation questions can revolve around either the process or outcomes of a program. Process evaluation questions assess how well the program is managed and executed. These questions explore whether the program is being implemented as intended, identify deviations or challenges, and provide insight into necessary adjustments. On the other hand, outcome evaluation questions focus on the results or impacts of the program. These questions aim to measure short-term, immediate, and long-term outcomes, providing an opportunity to assess program effectiveness, identify successes, and determine areas for improvement. Ultimately, outcome evaluation questions help evaluate whether the program delivers its intended benefits to participants and stakeholders. Institutions can ensure a comprehensive and insightful program evaluation by addressing process and outcome considerations.
For institutions that want to move program review beyond a compliance exercise, developing consequential questions examines previous activities, accounts for accomplishments, and identifies opportunities. Furthermore, exploring questions allows program review teams to benchmark and locate specific data needs, enabling evidence-based and informed responses that produce a high-quality program review process and submission.
C. Utilizing Data
The questions of program review teams can move beyond procedural obligations by expanding their use of data. Navigating institutions through the question identification process has significance in how data is requested, operationalized, analyzed, and interpreted. Incorporating data enables the broader understanding and contextualization of programs, fields of study, and the local and national industry landscape. Working with and exploring existing data identifies knowledge gaps for consideration and assessment.
Suggestions for expanded data inquiry can include:
- Environmental scans or district demographic data
- Regional labor market information
- Disaggregated data at the program, departmental, or institutional level across race, gender, age, income, and completion/graduation rates
- Any applicable or relevant industry-specific accreditation standards
- Student surveys were developed to investigate experiences around cost, quality, and need
In addition, “moving beyond the checkmark” in program review requires increased data analysis. High-quality program reviews demonstrate contextualization of the data and its meanings within an institutional context. The study provides an opportunity to understand patterns, relationships, and trends with the program, and it considers speculation of assessment. Data utilization contributes to a deepened understanding of each program in a context that enables a process and results centered on continuous improvement. Incorporating data inquiry and results in the program review process supports the development of a culture of evidence, broadly defined as “environments in which decisions are based on facts, research, and analysis…where services are planned and delivered in ways that maximize outcomes and impacts for stakeholders” (Office of Quality Assurance, Carleton University, 2014).
Conclusion
Once thought of as strictly a compliance activity of governing boards, program review has become a central process for learning improvement that aids in student success. Program review is an opportunity to discuss data and narratives that concisely depict a program's full context. The alignment of program review and its process is a purposeful tool that addresses racial, gender, socioeconomic access, retention, and completion goals. When viewed as a paradigm shift, program review offers an opportunity to engage in genuinely collaborative methods that ensure the process is well documented with results that inform campus planning initiatives, quality improvement efforts, and budget allocation decisions. By shifting their practices in relation to program review, community colleges can leverage these processes to enhance educational quality, drive innovation, and better serve communities.
References
Bourne, J. A. (2021). Advancing Equity-conscious CTE Program Review via Cross-Campus Collaborations, Dialogue. Voices and Viewpoints, Office of Community College Research and Leadership.
Carleton University (2014). Learning Outcomes Assessment Newsletter: What is a Culture of Evidence? Office of Quality Assurance.
Fox, H. L., Thrill, C., & Keist, J. (2018). Advancing Program Review: Evaluating and Envisioning the Future of Program Review at Illinois Community Colleges. Office of Community College Research and Leadership.
Eggleston, T. (2020, July). Program review and assessment for continuous improvement: Asking the right questions. (Occasional Paper No. 48). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment.
Georges, C. T., Jr., & Bourne, J. A. (2021). Advancing High-Quality Program Review for Student and Academic Support Services. Insights on Equity and Outcomes, (25), 1-6. Office of Community College Research and Leadership.
Illinois Community College Board. (2021). Program Review Manual Fiscal Year 2022-2026.
Thrill, C. (2019). Creating a space for student voice in advancing program review [Issue brief].University of Illinois at Urbana-Champaign, Office of Community College Research and Leadership.