This section includes the evaluation of outcomes at an institutional, program, and student level. However, we suggest this topic’s focus is on program evaluation, both formative and summative; institutional and program evaluation will be informed by assessment of student learning outcomes. Institutional evaluation informs organizational decision-making about a program at the institutional level. Program evaluation focuses on the decision-making related to program delivery, curriculum and content related to a degree program, certificate program, and major or minor within another program. Learning outcomes assessment relates to gathering evidence and making judgments about the attitudes, knowledge and skills students develop during the program. The assessment and evaluation processes and results at each level are linked to each other and to other underlying contextual frameworks, leadership content, teaching and learning.
The various guiding questions within this section intentionally overlap to ensure that the relevant aspects and elements are comprehensively addressed. As a consequence, some repetition within these questions is unavoidable; however, it is not as much important where those questions are addressed rather than making sure they are discussed.
Generic Questions
What are the desired outcomes of the program at the institutional, program and student levels?
Outcomes (and their assessment) are core components of any educational program that claims to be accountable for its results and to do so applying transparent processes and structures. Assessable outcomes will spell out what graduates of a program will be able to do as a result of the program. To achieve these outcomes in a coherent and consistent way, program outcomes need to be aligned with the outcomes of the institution hosting the program as well as with the respective learning outcomes of students taking the program.
Furthermore, the outcomes need to be clearly described conceptually as well as operationally. These descriptions also need to identify the connection between the individual outcomes and the expected way as to how these outcomes will contribute to growth and competency in leadership.
The following questions will help you respond to this particular guiding question:
·Does your institution identify outcomes to be achieved by its students? What are they (conceptual and operational descriptions)?
·Does your program have program related outcomes that graduates will have achieved upon graduation from the program? What are they (conceptual and operational descriptions)?
·Do the program outcomes support and / or complement your institutional outcomes?
·Does your program specify learning outcomes for its students? What are they (conceptual and operational descriptions)?
·Do student learning outcomes support and / or complement your program outcomes?
·How do student learning and program outcomes connect to and support each other? How do they contribute to growth and competency in leadership (conceptual and operational descriptions)?
What are the identified leadership competencies and proficiencies and how do they relate to the program’s philosophical and theoretical perspectives?
Outcomes of a leadership program ought to clearly identify competencies and proficiencies of both leaders and followers. The choice of these competencies and proficiencies will likely be influenced by the theoretical and philosophical frameworks underlying the leadership program. To increase transparency for and accountability towards all stakeholders (particularly students and faculty) these frameworks (see previous sections of these guidelines) as well as their links to the chose outcomes need to be explicitly named and sufficiently described.
The following questions will help you respond to this particular guiding question:
·What competencies and proficiencies of leaders and followers are identified in your program and student learning outcomes?
·Are the underlying theoretical and philosophical frameworks explicitly named?
·What are they?
·How are they linked to the outcomes?
How do the desired outcomes relate to conceptual, contextual, content and delivery related elements (see previous sections)?
Outcomes on an institutional, program, and student level need to be linked to the conceptual choices made as well as to the contextual environment of the institution, program, and the students. Furthermore, the courses need to be intentionally designed such that the chosen course content as well as the way of their delivery ‘to the students’ will most likely lead to a high success rate in achieving the desired outcomes.
The following questions will help you respond to this particular guiding question:
·How are your institutional, program, and student learning outcomes linked to the respective conceptual frameworks (see section 1)?
·How are your institutional, program, and student learning outcomes linked to the respective contextual environment (see section 2)?
·How does the course design process ensure the selection of course content that is supportive of (see section 3)?
·How does the program’s approach to teaching and learning ensure that the selected course delivery methods support the student learning and program outcomes (see section 4)?
How will you know when you have achieved those outcomes? What are your essential indicators? What is the assessment system?
Accountability and transparency as required of today’s educational programs call for outcome assessment. Given the central focus of outcomes - significant results of the program - they should be designed having their assessment in mind. Criteria and indicators of successful achievement (including the level of success) will provide meaningful feedback to the learner as well as to the program and institution. This feedback will allow to intervene on an institutional, program, and learner level on a continuous basis. Thus, outcomes and their assessment become integral to the institution, to the program, and to the learning process.
The following questions will help you respond to this particular guiding question:
·Do your outcome descriptions include assessable criteria and indicators of success?
·How are these criteria and indicators measured and / or assessed?
·How is this assessment meaningfully fed back to students, to the program, and to the institution?
·How does the program support the application of the feedback on an institutional, program, and student level?
How are the criteria for excellence incorporated into assessment?
Given that excellence is an intended feature for many educational programs it needs to be clearly spelled out what the areas and characteristics of excellence for the given program are. For the chosen areas and characteristics the benchmarking process as well as quantitative and qualitative indicators and measures need to be identified. These benchmarking and measuring processes need to be an integral part of the overall assessment system.
The following questions will help you respond to this particular guiding question:
·What the areas and characteristics of excellence for your program?
·What is the benchmarking process for the chosen areas and characteristics?
·What are quantitative and qualitative indicators and measures?
·How are these benchmarking and measuring processes integrated in your assessment system?
Institutional Level Questions
How do institutional governance processes use your assessment of learning outcomes and program evaluation to inform decisions about your program?
Being embedded in a responsive feedback and program development process as well as in a respective strategy development process will most likely increase the success rate of a leadership education program and of learners enrolled in such a program. Furthermore, implementing such an institutional governance program does indicate that the institution is walking the talk of comprehensive and consistent leadership and governance processes. Feeding back the results of learning outcome assessment and program evaluation into these processes of program and strategy development on an institutional level may play a major role in ensuring that these processes are based on indicators of learning (and learners’) successes and needs.
The following questions will help you respond to this particular guiding question:
·What are the feedback and program development processes for your program?
·What is the strategy development process for your program?
·How is your program embedded in these processes? What are the interfaces? What information flows through these interfaces?
·How are the results of learning outcome assessment and program evaluation fed back into these processes?
·What indicators of learning (and learners’) successes and needs are these processes based on?
What sources external to your institution inform your assessment and evaluation?
Leadership education programs educate for fields of practice and for communities outside of the program and institution. As a consequence, requirements coming from these fields and communities need to be taken into account when describing the intended learning and program outcomes as well as when designing the respective assessment and evaluation system. Furthermore, criteria and indicators of successful application of newly acquired knowledge and skills into practice after returning to or entering into the respective fields or communities may play a significant role for program and strategy development within the educational institution; to allow for this to happen, appropriate mechanisms and indicators of assessment need to be built into program assessment and evaluation.
The following questions will help you respond to this particular guiding question:
·What are the relevant fields of practice for your program?
·What are the relevant communities outside of the program and institution that your program caters to?
·What are the requirements coming from these fields and communities?
·How are these taken into account when describing the intended learning and program outcomes?
·How are these taken into account when designing the respective assessment and evaluation system?
·What are the criteria and indicators of successful application of newly acquired knowledge and skills into practice?
·What are the appropriate mechanisms and indicators of assessment?
·How are they built into program assessment and evaluation?
Program Level Questions
What conceptual frameworks (see sections 1 and 4) inform your assessment of learning outcomes and program evaluation?
Assessment of learning outcomes and program evaluation is built upon conceptual assumptions around values and objectives as well as knowledge and learning. Explicit discussion of those links will allow for informed decision making of the various stakeholders (particularly students and faculty) and for their meaningful input into the processes.
The following questions will help you respond to this particular guiding question:
·What are the relevant conceptual assumptions around values and objectives underlying your program’s assessment of learning outcomes and program evaluation?
·What are the relevant conceptual assumptions regarding knowledge and learning underlying your program’s assessment of learning outcomes and program evaluation?
·How does the discussion and development of these connections inform the decision making processes within your program and institution?
·How are students and faculty involved in these decision making processes around assessment of learning outcomes and program evaluation?
How do you incorporate formative assessment and evaluation as a process into your on-going assessment of outcomes?
This question tries to identify how the existing frameworks around values and objectives as well as knowledge and learning underlying the program do inform formative assessment, what role formative assessment and evaluation do play within the overall assessment processes, and how is it built into these processes.
The following questions will help you respond to this particular guiding question:
·How do the existing frameworks around values and objectives inform formative assessment within your program?
·What are the values and objectives underlying formative assessment within your program?
·How do the existing frameworks around knowledge and learning inform formative assessment within your program?
·What are the concepts of knowledge and learning underlying formative assessment within your program?
·What role does formative assessment and evaluation play within the overall assessment processes of your program?
·How are formative assessment and evaluation built into the overall assessment processes within your program?
What are the fields of practice your program serves (see section 2)? How do they influence the design and assessment of your program (outcomes)?
Leadership education programs educate for fields of practice and for communities outside of the program and institution. As a consequence, these fields and communities should significantly inform the design and assessment of the program (outcomes). Clearly naming and describing the chosen focus for the program in terms of the fields and communities that the program is intended to educate for will help the stakeholders to make informed decisions.
The following questions will help you respond to this particular guiding question:
·What are the relevant fields of practice for your program?
·What are the relevant communities your program caters to outside of the program and institution?
·How do these fields of practice and communities inform the design of program outcomes?
·How do these fields of practice and communities inform the assessment of program outcomes?
·What is the chosen focus for the program in regards to the chosen fields of practice?
·What is the chosen focus for the program in regards to the chosen communities?
·How does the chosen focus inform the decision making processes of stakeholders on program design and outcomes assessment within your program?
How do you use the summative and formative assessment of student learning outcomes to inform your evaluation of your program outcomes (organizational learning)?
Assessment of student learning may be an invaluable source of both organizational learning as well as learner centered program development. However, for a consistent and systematic feedback cycle interface processes need to be in place that facilitate mutual information exchange.
The following questions will help you respond to this particular guiding question:
·How does assessment of student learning inform organizational learning within your program and institution?
·What feedback cycles are in place?
·How does assessment of student learning inform learner centered program development within your program and institution?
·What feedback cycles are in place?
What is the evidence of organizational learning at the program level?
Assessment of organizational learning will allow for evaluation of the effectiveness of comprehensive feedback cycles connecting student learning and organizational learning on an institutional level. For example, existing evidence will be a powerful means of attracting both high calibre students and faculty.
The following questions will help you respond to this particular guiding question:
·How do you benchmark organizational learning within your program and institution?
·How is the effectiveness of organizational learning based on the evaluation of student learning assessed within your program and institution?
·What are relevant criteria and indicators within your program and institution?
·What evidence is captured within your program and institution for organizational learning in this regard?
How do the chosen outcomes inform the implementation of your program?
Learning outcomes and the various elements of program implementation (e.g., location, space, faculty and staff, delivery modes, etc.) are not separate. They should be intentionally designed to mutually support each other.
The following questions will help you respond to this particular guiding question:
·How do the elements of your program (e.g., location, space, faculty and staff, delivery modes, etc.) support the successful achievement of your program’s learning outcomes?
·How is the implementation of these elements designed within your program and institution?
·How does the design of the implementation of these elements ensure for them to be supportive of your program’s outcomes?
Student Level Questions
How do you assess competency and growth in your students (in other words, what value-added to the student results from participation in your program)?
Student learning should be at the core of educational programs. Student learning can be measured in terms of their competency at any given level as well as of their growth as a result of any given program intervention (e.g., a learning opportunity, a course, a section, etc.). Assessing those two dimensions becomes paramount for meaningful student feedback, for ongoing assessment of learning and teaching effectiveness, and for course and program development.
The following questions will help you respond to this particular guiding question:
·How does your program measure student competency in regard to your program’s and institution’s outcomes?
·How does your program measure student growth in regard to your program’s and institution’s outcomes?
·How does this assessment happen on a course level?
·How does this assessment happen on the level of an individual assignment?
·How are the assessments on these various levels connected and integrated?
·How is this assessment on these various levels fed back to the students?
·How does your program and institution support students in the application of the feedback received?
·How is the students’ application of this feedback assessed?
·How is this feedback applied within program and organizational development?
References