Recommendations for mutual recognition of institutional evaluations

Contents

Introduction

One can imagine a continuum between pure programme accreditation on the left side of the continuum, pure institutional evaluations (accreditations, audits or assessments) on the right side, and mixes between programme and institutional evaluations in the middle. Programme accreditation used to be most frequent type of evaluation in Europe. However, there is a tendency in European “external quality assurance” to switch from programme accreditation to combinations of some forms of “institutional evaluations” and programme evaluations. These forms may differ in quite meaningful ways: the Austrian colleagues of ÖAR execute real “institutional accreditations” of the private universities in Austria, whereas in other countries the institutional evaluations are restricted to the “internal quality assurance systems” on the institutional level. In some countries (e.g. 2010 system in the Netherlands) the institutional evaluation (audit) and programme accreditation are intertwined; a positive institutional audit results in a lighter touch for the programme accreditations of that institution. In other countries institutional and programme evaluations are separate systems of evaluation. It is also possible that one type of evaluation is mandatory and the other voluntary. Of course, these different types of evaluations might cause trouble for “mutual recognition”, especially if it is not clear how teaching or the quality of the programmes are involved in these evaluations. We should therefore strive for a common description of the institutional evaluations in order to make mutual recognitions possible.

Recommendations

Recommendation 1:

“in mutual recognition agreements, institutional “evaluations” are regarded as fit for these agreements if they are carried out according to the commonly agreed rules and guidelines of ECA (code of good practice, selection criteria for experts, commonly agreed evaluation standards)”

Recommendation 2:

“institutional evaluations optimally fulfil the purpose of mutual recognition agreements if:

• they focus at least on the internal quality assurance systems of higher education institutions;
• they provide sufficient and reliable information about at least the quality of educational provision;
• they are based upon transparent and internationally accepted standards (ESG).

Institutional evaluations

For the process of the institutional evaluations it is necessary to get the outlines of the procedure known and to obtain information on topics, standards and outcome of the evaluation. Six topics seem essential in order to understand the quality of the evaluation process and the meaning of the outcome:

1. the composition and independence of the panel;
2. the standards used;
3. the basis for the evaluation, as prepared by Higher Education institution;
4. the site visit;
5. the outcome and consequences;
6. publication of the outcome and report.

The composition and independence of the panel

ECA has adopted “common principles for the selection of experts” in Dublin, June 2, 2005. For institutional evaluations the following principles for selection procedures and panel composition are relevant.

Procedures

• any decision regarding the expert panel should be based on the policies, procedures and criteria of the accreditation organisation or on relevant legislation
• panel members must be independent and in a position to make unbiased judgements. Any possible conflict of interest must be disclosed
• the selection criteria for experts panels must be established and published by the accreditation organisation
• applicants undergoing evaluations are given the opportunity to comment on the selection of panel members
• the decisions on the evaluation should be made by the relevant authority and not by the group of experts themselves
• panel members must be committed to treat all material and findings as strictly confidential
• panel members are briefed adequately by the accreditation organisation on the context within which they are operating (national legislative environment, criteria, procedures and guidelines)

Composition

The number of panel members may vary depending on the range of competence of individual members. Gender balance should be taken into consideration when appointing a panel of experts. The expert panel should have the following mix of expertise appropriate to the objectives of the accreditation procedure:

• experience in quality assurance in higher education
• appropriate academic or professional qualifications
• expertise in institutional governance and management
• leadership experience in research/academic/professional management
• relevant international experience that provides a basis for making international comparisons
• knowledge on teaching and learning methods
• expertise in development, design, provision and evaluation of higher education programmes
• knowledge of the country-specific system of higher education, institutions and applicable legislation

Depending on the national context it is recommendable to include in the institutional evaluation panel:

• student representative(s)
• representative(s) from the labour market
• a significant proportion of panel members from outside the country

Recommendation 3:

The ECA-members should put the guidelines on the procedures and the composition of the panels for the institutional evaluation into practice.

The evaluation standards

The institutional evaluation does at least involve the “internal quality assurance system” on teaching and learning. There are of course different methods and criteria to assess this system. Some organisations use a limited set of standards and criteria, whereas others are using a larger number. The standards and criteria used have to be published.

Recommendation 4:

The ECA-members should publish the standards and criteria used in the “institutional evaluation” process in the English language.

The basis for evaluation, as put together by the institution

In the evaluation report should be mentioned what has been the basis for the evaluation by the panel. In some countries self-evaluation reports are made by the institutions (“critical self-reflection”), in other countries management reviews are used for the evaluation, and in Norway for instance annual reports and the “going concern” are used by the panel.

The site visit

The programme of the site visit and the participants in the site visit should be mentioned in the report.

Recommendation 5:

There should always be a site visit when carrying out an institutional evaluation.

The outcome and the consequences

The outcome should be mentioned in the report as well as the different categories in which the outcome may be classified. Some systems are dichotomous: only “yes” or “no” are possible. Other systems will allow the accreditation organisation to give a “conditional yes” as an outcome of the evaluation. The consequences of the evaluation may differ from system to system. It would be very helpful if the possible outcomes of the evaluation are published by the accreditation organisation and if the consequences are explained.

Recommendation 6:

The ECA-members publish and explain the possible outcomes of the evaluation process, and explain the consequences of the different outcomes.

Publication of the outcome and report

Publication is an essential condition for mutual recognition; without a published outcome there is nothing to recognise. Although it is possible to recognise a positive outcome without an underlying public report, there is a problem of transparency. The ideal situation is that both the outcome and the report are published. However, in practice the report is often not published, e.g. if the outcome is negative or the application withdrawn after a negative outcome. In the latter case, and also in some systems in case of a negative outcome, not even the negative outcome is published. There are also systems in which only an extract of the report and not the full report is published.

Recommendation 7:

ECA members should take steps to increase the publication of both outcomes and reports of institutional evaluations.

ECA Mutual recognition of institutional evaluations recommendations