CIDEC logo CS & IT EDUCATION DEVELOPMENT CENTER

Tempus CME-02078

Tempus logo

Back Workshop on Accreditation of IT High Education

Contents

Foreword. (Jaan Penjam)

People are not satisfied with the current situation in high education quality at our universities. Several reforms have been going on arising new problems e.g. allowing freedom for every student to compose his own curricula has resulted in nobody knowing what anybody has learned.

Also in Europe as a whole people are concerned with the quality of higher education (assessment and assurance), we are not the only ones who are asking for materials about how it is done.

The first step we could do to start with quality assurance is to gather and spread information what curricula are currently used at our universities. We need to get an overview how things are composed in Tallinn and Tartu (course system, how concepts are interpreted, what is a curriculum, a course).

We are planning to do this within the framework of this QUASIE project.

Current situation in partner universities: most professors do not know about quality assurance. It is the field of special commissions.

How will it be with us? Our aim is to at least know, be involved, informed. Curricula and teaching process needs assessment, if quality is bad the specialities should be closed.

Plan for the first year of the project is to work out the conception plan for quality assurance, during the second year implement it in practice.

One important thing that could be actually done: a database on computing courses in Estonia.

State of Art in Quality Assurance in Estonia.

Translation of terms:
Why?

What is quality? It is a complex characteristic of a system, does not appear itself, therefore a lot of work is needed, also in education. The amount of knowledge necessary to obtain during high education is very large.

Problems with quality - variety in interpretation, not measurable, subjectivity, complexity.

In production and service unification has been under development for over 10 years. International standardisation organisation ISO 9000 series (about 1990), also ISO 10000. Has been translated into Estonian (in 1995), the new versions that appeared in 1994 are currently in publication.

Definition of quality according to ISO 9000 - the maximal satisfaction of the direct and expected needs of the client (and the continuous enhancement of one's activities).

In education - who is the client? What are the needs? Student, family, employer, professional organisations, society - everyone has its own requirements.

In evaluating the quality of education it is considered as a framework of multiple quality facets, if all these appear to be satisfactory we can hope to have a good quality.

In evaluating curricula the following quality facets are considered in Estonia: educational policy, curriculum itself, management of studies, study process, students, teaching staff, study environment (both infrastructure and resources), feedback, internal quality assurance.

Another approach, distant description - to set recognised educational goals and to try to create the best possible environment to enable students (and teaching staff) to achieve these goals. The problem here is how to determine the completeness and accordance of the goals.

Estonia is a partner in world-wide INQAAHE - International Network for Quality Assurance Agencies in Higher Education (email aau@nzvcc.ac.nz).

High Education Assessment in Estonia.

In summer 1995 the government founded the High Education Quality Assessment Council, until now there is no organisational establishment. Half members of HEQAC represent universities, half other society.

Accreditation procedure: first the higher educational establishment (university) volunteers (or is forced to volunteer) for accreditation. Our university law states that in four years all curricula should pass an accreditation. Already it is clear that fulfilling such plan is unrealistic.

Stimuli for university - only universities that have successfully passed the accreditation can in the future apply for state recognition and also financial resources.

Organisational procedure: assessment can be done in two aspects - either institutional or curriculum-oriented.

There exist several systems and habits of assessment. In most countries the main goal is evaluation focusing on enhancement in quality, to get recommendations for the university on the basis of evaluation, in some countries also reports are required. Our ministry of education is in favour of the mere accountability system - they want to bind financing with accreditation (a need to cut resources). However, so far such experiments in other countries has not proved to be successful, maybe only in the case when only additional finances have been offered on the basis of an excellent evaluation.

As a whole it is useful to gain knowledge where and in which educational institutions things are better. The institutional accreditation evaluates the structure of the university and the studying environment, it does not go into details at particular curricula level (due to the number of different curricula the evaluation commission has to be rather large - at least an expert per curricula, it would difficult to get to the essence). In Estonia the main type is curricula-oriented accreditation. Only if in one educational institution a lot of curricula will disqualify an institutional accreditation is supposed to be initiated.

First step for the university after applying for accreditation is to present a self-evaluation (on the basis of a questionnaire). In practice most self-evaluation reports are quite ideal - one should be self-critical, but …

After that peer visits take place by experts - usually professors from other universities, not professional assessors. In some countries there exist a separate inspection system, but it is not related to accreditation procedure. Problem in Estonia - about 80% of all curricula are taught only in one place, no local experts can be found outside the institution. Inviting foreign experts is complicated e.g. the self-evaluations are written in Estonian language, need translation. On the other hand our language law does not allow to demand the self-evaluations only in a foreign language e.g. English.

In Estonia 2-day visits per institution are used, full days, hard work. If possible, we have invited experts for four days to visit two institutions with similar curricula. HEQAC does not demand large reports, substantial remarks are more valued (strengths, weaknesses, proposals).

The expert evaluation report is in average about 10 pages. The Council discusses it and the answer of the university and on the basis makes the decision. In several countries the accreditation as the process does not exist, the recommendations given to universities by the experts is considered sufficient.

In Estonia the accreditation shows state approval - the institution is allowed to use the official form of graduation certificate. The exist three degrees of accreditation:

The Ministry of Education is supposed to have an official register of curricula, it does not exist yet. There is statistical information about curricula - contains over 300 different curricula plus master and PhD curricula. Only in TTU there are over 160 different curricula (too much although one reason for that are continuous reforms).

During already taken place assessment of economic curricula, HEQAC united similar curricula (50% or more coincided) into cognate areas to be assessed by the same commission. Using such cognate areas makes it possible to reduce groups of curricula to about 35.

The necessity of small countries like Estonia to invite foreign experts is rather expensive one expert costs about 15 000 EEK. One cannot say that the expert opinions are useless. In spite of not being local, good experts very quickly pick out important issues. The advantage is that they assess from the international point of view.

It would be reasonable to ask also the opinion of people from industry. They could evaluate the quality of graduates of different schools. It would be good to co-operate with professional organisations, this system does not work yet.

In some countries similar curricula are compared and ranking lists are composed. HEQAC has given this option up. For example, assessing all economic curricula by the same expert group would take 3,5 weeks of hard work, impossible to find experts for such a long consecutive period.

In informatics statistics could be used to compose a ranking list. To obtain data cheaply questionnaires could be passed among industry representatives e.g. at the computer spring fair.

Conceptions Related to Curricula and Study Process.

Components of education

If the subject in an educational institution is aimed develop know-how and skills mainly by teaching, it ends with a credit test. Norms of behaviour cannot be really taught, they can only come into being as a subproduct in teaching team work.

Formation of knowledge. George Carlin's taxonomy:

Characteristics of knowledge.

Understanding (süvamõistmine) - ability to use the obtained knowledge in creative way in solving problematic and investigative tasks.

Know-how (oskusteadmus) - ability to solve problems mainly on the basis of experimental knowledge obtained during a similar situation earlier.

The diploma studies should give the know-how and how to use it creatively, problem solving mainly on the basis of experience.

The academic studies should reach the level of scientific conceptions, ability to solve any problems and investigative tasks.

The process of creating a curriculum should be clearly and accurately determined. It is a document that specifies educational goal, but leaves some flexibility.

Two possible strategies. Process strategy - it is believed that desired result will be achieved by determining the structure of subjects based on the initial knowledge level of students. The result is expected to appear automatically.

Result strategy - First the educational model of a graduate is specified and then what one needs to achieve it. Advantage of this strategy is that indirect aspects are taken into account naturally.

A curriculum is just one educational project.

In Tallinn Technical University the process strategy is used. The Estonian university law is rather inflexible. The length of every education stage should no be fixed in time.

Since the amount of credit units per semester (20 credits) is fixed, we can only influence the contents of such frame.

In TU the concepts of the main and additional subject are used. In TTU there is a slightly different approach - module structure - subjects are united into designated studies.

Bachelor level i.e. DEGREE STUDIES (suunaõpe) 160 credits, in TTU consist of:

FIELD STUDIES (valdkonnaõpe) 80 credits, including:

There are 10 study fields in TTU, which contain over 20 different study courses. The information technology study field incorporates 4 study courses: telecommunication, computer and systems engineering, software engineering, and electronics and biomedical engineering.

CORE STUDIES (põhiõpe) 20-30 credits are speciality-specific.

SPECIAL STUDIES (eriõpe) 20-30 credits, include more application-oriented subjects. Capacity of core studies and special studies is 55 credits (so total with field studies 135 credits).

During the former fifth-year study programme everyone had to take also an additional module (in TU it is "kõrvalharu"). It does not show in the current 4-year study programme, but there part of the courses have to be chosen from a selection of optional courses.

In addition there are FREE STUDIES (vabaõpe) 5 credits, INDUSTRIAL TRAINING (praktika) 5-10 credits, and FINAL THESIS 15-20 credits forming the total capacity of 160 credits.

At graduation the amount of cu per block is considered, not so much attention paid to separate subjects.

MASTER STUDIES include 8-15 credits of general and basic studies, 15-25 credits of core and special studies and master thesis approximately 40 credits. It is possible to apply for a professional degree, in such case the thesis can consist of a project or a survey analysis.

DOCTORAL STUDIES - 40-50 credits as a whole, including 5-15 credits general and basic studies and 20-30 credits core and special studies (formerly total was 70 credits) and about 100 credits for research and doctoral thesis.

There exist non-credit subjects like foreign languages, not included in the study program, but graduation conditions required to pass an exam in one foreign language.

Questions.

The field study program council determines what subjects form the basic and core studies at graduate level. Basic studies usually include additional mathematics, physics, some humanitarian subject, mainly classical stuff.

Study courses in TTU in the field of information technology and related people:

It is in the state interest to divide students among specialities, solution is quota.

There is a time bomb, due to popularity information technology gets the most capable students, other fields are left with worse.

Terminology. Õppekava - internationally curriculum, also study program (in TTU the latter is used). The curriculum could possibly reflect the part dealing with auditorial studies (135 cu). No agreement on terminology so far.

For information: UNESCO 92-96 published a 2-volume multilingual terminology collection, where commissions from many countries (including USA and Canada) presented 50-200 educational concepts with explanations. In the 2nd volume 50-60 concepts collected and explained in English by vice-rectors of Estonian universities.

Every study course (õppesuund) has a study program which is proposed by the field study program council (about 10 members) and it has to be approved by the involved faculty council and also a couple of other faculties. Also there should be and outside assessment, but there is no suitable body to ask from yet. After that the field study program council endorses it. Unsolved the aspect of coordination with potential employers.

Education Quality Framework

To start a discussion on how well we teach, we need to agree with quality criteria we shall consider. IT and CS quality is not much different from others maybe only in details, but that is where we focus.

Ideal could be a scale for a commission to evaluate single items and then summarise the results. The problem is there are no two similar curricula in Estonia. One way could be also exchanging subjects between universities: teaching to an unknown auditorium usually makes you reassess what you lecture.

On the basis of available material, a paper has been prepared (in Estonian) on aspects in assessing curricula.

Twelve Aspects of Estimating Curriculum and Process of Its Implementation (remarks).

  1. Aims and Curricula.
    Is it possible to present the aim in a way that it is interpreted in the same way both by lecturers and students? Social needs that are declared to us, should be prevalent. We have to teach not what the client wants, but what he needs. Should Tallinn and Tartu teach the same?

  2. Curriculum Design and Review.
    What are the needs of a student? B2: Often it is impossible to organise teaching due to lack of computers of textbooks. B3: how elderly students are considered (their initial knowledge is different) ? B4: The checking mechanism?

  3. The Teaching and Learning Environment.
    Academic atmosphere. Teacher-ethics. Bad environment tears down the quality level of a curricula. We should estimate if is good within available resources - how effectively the resources have been used.

  4. Staff Resources.
    Receiving finances might not automatically produce necessary people.

  5. Learning Resources.

  6. Course Organisation.
    Distributing information about course organisation. F3: specialist? In the west it is common to invite outside experts, representatives from industry, research professors. Lecturers often themselves do not know who of the colleagues is supposed to teach what.

  7. Teaching and Learning Practice.
    Learning is more important than teaching. Even over a half of curricula could be obtained by self-learning. In TTU in the new study programs the auditorial studies should not be more than 50% and still the accreditation commissions emphasise the abnormal proportions. Still there is the problem of availability of teaching materials that could replace the lecture. There are different approaches.

  8. Student Support.
    Necessary to distribute information how to handle the curriculum. Problem especially in master and doctoral studies - there is only one person who wants the subject. The curriculum should determine the course of action in case of a single student, e.g. 3 students or a larger group. H7: not securing welfare but friendliness - general friendly attitude, no molesting or persecution - part of implementation quality.

  9. Assessment and Monitoring

  10. Students' Work

  11. Output, Outcomes and Quality Control.

  12. Student Placement
Originally for social workers. IT and CS need it, there is not enough contacts with industry. Problem of organisation.

Quality assurance in partner countries.

Experiences with Quality Assessment in Denmark on the basis of reports of the Danish Centre for QAEHE

QAEHE is the Danish Centre for Quality Assurance and Evaluation of Higher Education.

The Centre initiates evaluation processes, develops appropriate assessment methods, inspires and guides universities in aspects concerning evaluation and quality, and compiles national and international experience on evaluation.

In Denmark all curricula leading degrees will be evaluated within the period of 5-7 years.

Evaluation model ; Initiator; Self assessment; User survey; Report from External Examiners; Site visit; Conference; Evaluation report; Centre for Evaluation; Steering Committee

In Denmark the evaluation process has the following phases:

  1. Planning (Steering Committee, Project Secretariat)
  2. Self-assessment
  3. User Surveys (graduate, student and employer surveys; report of external examiners)
  4. Site Visit
  5. Reporting(Evaluation Conference, Final Report)
The following qualitative and quantitative aspects are evaluated.
Qualitative aspects:
  1. overall objectives of curricula
  2. institutional and organisational context
  3. physical facilities
  4. structure of study programme
  5. teaching methods
  6. assessment of students
  7. corps of external examiners
  8. students' qualifications
  9. co-operation with other institutions (in Denmark and abroad)
  10. arrangements for internal quality assurance

Quantitative aspects:

  1. costs and sources of income
  2. number of staff
  3. number of student body
  4. internationalisation

Quality Assessment in Germany

The basic scheme is the same as in Denmark.

Curricula are evaluated by several lecturers from the same institutions. No high-level evaluators (e.g. institution's administration), because there are protection laws of employees, not allowed to pass information about them. Also not allowed to make the results public e.g. put on WWW. Due to protection laws it is not even possible to fire somebody if he is performing badly.

If students judge the material to be completely out-of-date, they do not come to lectures, but cannot ignore totally, because the examination is required.

We just have to trust in reasonable behaviour of lecturers. The hiring of a lecture is rather a thorough procedure: publication lists are examined, sample lecture given.

We have questionnaires to perform user surveys. Lectures are estimated from the didactic point of view, how easily understandable, the level of being up-to-date, speed of proceeding, how written material corresponds to the lecture - maybe outdated if any at all, how exercises and lectures are combined.

It is asked if the student feels motivated, if the lecture contains questions, lecturers behaviour towards individual remarks, questions, whether lecturer is available at other times - consultation hours.

There is an everlasting conflict between ideal research and lecturers, it is difficult to find balance.

There exist external examiners and site visits. In 1990-92 there was and evaluation in all East Germany universities performed by examination groups formed from people from West German universities, however including people formerly from East Germany who were familiar with the situation.

These groups gave recommendations, some people were fired. On one hand it was hard, on the other hand reasonably objective, quality has improved.

Now also Western universities have evaluations mainly to reduce parallel teaching of some subjects. Mixed groups are evaluating.

Assessment in mainly done on voluntary basis, only in some rare cases have been mandatory. Site visits too often would be too time-consuming.

What aspects are evaluated: teaching methods - using whiteboard and foils. Recently we have introduced tele-teaching, not video-conferencing, just include videos in asynchronous way, some animations with Power Point foils.

We have tried video-conferencing with e.g. Mannheim (about 20 minutes) and also public-domain internet software applications (whiteboard ...). Necessary equipment: workstations, at least 64 Mbytes of memory, video and audio devices.

Study system. We have a diploma that in quality equals approximately to MSc, however not accepted in most cases as equivalent. We have had discussions about introducing MSc and PhD, mainly USA oriented, although there is no American model as a whole.

Freedom of studies in Germany. First and second year courses fixed, some additional are optional. On the third and fourth year some courses are mandatory and there exist a bunch of fields (16 different in Dresden) that one must choose 2-3 from. After that half a year for diploma. So total is 9 semesters, however average studying time is 11-12 semesters. Exams are written for 1st and 2nd year and for mandatory courses later. Oral exams are 1-22 hours covering material of 10-12 hours per week.



Back Top Created 09/04/1997. Comments: monika@cs.ioc.ee