|
Stakeholder Evaluation: giving everyone a voice in the feedback process Introduction This paper describes the development of a model of ELT evaluation that fully incorporates the key stakeholders involved in the programme. The aim of the model is to involve the stakeholders in the evaluation process at the stages of planning, data collection and reporting, in an attempt to enable all stakeholders to have an input into the development of the programme. The model is formative in nature, intended to inform the development of the ongoing programme. Although designed for use with corporate language programmes the rationale behind the model would seem to be equally applicable to development projects as they usually involve a diverse group of stakeholders. Lynch (1990) provides a succinct definition of evaluation, "…the systematic attempt to examine what happens in, and as a result of, language programs, typically serves as the basis for judgements and decisions about programs" (p. 23). Interestingly, Rea-Dickins (1994) includes the concept of stakeholders in her definition of evaluation, maintaining that the evaluations are stakeholder orientated unlike most other educational research. The term stakeholder has now become widely used in many fields particularly in business and management studies. Aspinwall et al. (1992) provide a comprehensive analysis of the concept of stakeholders in the context of educational evaluations and state that "A stakeholder is any group or individual who is affected by, or can affect the future of the organization, programme or activity…. Stakeholders can be both internal (e.g. staff) and external (e.g. parents) to the school" (p. 84-85). The stakeholders in any given programme will thus be determined by the nature and context of the programme.
The rationale for this study One reason for developing a stakeholder model of language programme evaluation was the gap in the field, particularly in relation to ELT, between current writing on evaluation that acknowledges the importance of stakeholders (Alderson, 1992; Rea-Dickins, 1994; Weir and Roberts, 1994; Gurr, 1996; Lynch, 1996), and application in terms of programme evaluations involving stakeholders. This anomaly is noted by Johnston and Peterson (1994) who state that "an important part of the new approaches to evaluation is the increasingly felt need to incorporate the viewpoint of stakeholders…. Such ideas have been raised in ELT but little applied as yet" (p. 66). However, the major reason for embarking on the area as a research project was that my experience in providing and overseeing corporate language programmes, and further discussions with colleagues, had made me aware of a conflict of interests operating in many such programmes. I have found that a lack of consultation can lead to significant mismatches between the expectations of those involved in the programme and the realities of the programme itself. I felt that this was largely due to a lack of channels of communication between the various parties involved in the programmes, both at the initial planning and the evaluation and reporting stages.
The context of the study The company in which this stakeholder model of evaluation was developed and piloted was Ericsson Telecommunications, Thailand. The company aims to establish a level of language proficiency for different positions, then test the language level of all staff and provide training for all those that do not achieve the required standard for their post. During the period of the evaluation my organization was running three courses at, elementary, pre-intermediate and intermediate levels. As it is often very difficult in Thailand to obtain frank feedback by informal means, and the school felt that it was imperative to improve its own procedures of evaluation, we decided to develop a model that involved the key stakeholders.
General frameworks for programme evaluation Works on educational programme evaluation in the 1990s provide useful frameworks of criteria that those involved in organizing an evaluation should consider. (Rea-Dickins, 1994; Rea-Dickins and Germaine, 1992; Aspinwall et al., 1992; Alderson, 1992) all provide similar checklists or procedures to consider when conducting an evaluation; below is a synthesis of the main points that are covered:
The importance of context Although such checklists are no doubt invaluable for evaluators, it can be argued that they are in many ways generic and do not sufficiently consider the context in which the evaluation is taking place or the people connected to the programme. By focusing on the context, the players or stakeholders in the programme assume a more central role in the framework. Rea-Dickins and Germaine (1992: 19) develop a model called ‘a context for evaluation’, which places a language course within its context and illustrates the complexity involved in doing this. Their context includes the school and those within it who influence decisions made about classroom practice, such as the Head; it also looks at factors outside the school, including parents, local education authorities and society at large. Gaining an understanding of such factors and how they influence classrooms is the only way in which evaluators can gain the necessary depth of understanding of a programme. Divorcing an evaluation from its context may simplify the evaluation design but is likely to result in an incomplete understanding of the programme. The most comprehensive consideration of context in relation to evaluations is developed by Lynch in his ‘context-adaptive model’ outlined in (1990) and slightly refined in (1996). He asserts (1990: 24) that "The distinguishing feature of this model is its flexibility in responding to the range of contextual constraints that program evaluation can encounter." He presents a seven-step model for evaluators to consider, which involves detailed analysis at each of the following stages (ibid.):
It is the first three steps of Lynch’s ‘context-adaptive model’ that set it apart from other more generic models of programme evaluation and provide the focus on contextual factors as the starting point of the evaluation design. Lynch in his conclusion (1990: 39) argues that starting the evaluation with an analysis of the context in which the programme exists ensures a high degree of flexibility making the model highly adaptable and concludes (ibid.: 36) that "… the context-adaptive model for the evaluation of ESL programs is intended as an elaboration of a generalized model that can be refined through its application in a variety of language teaching contexts." However, his model is lacking in that it does not include an analysis of the evaluation design and its effectiveness. It is always important for evaluators to consider the appropriateness of the methods they use and Alderson (1992) suggests that the main criteria for evaluating an evaluation is the usefulness of its findings and states:
Identifying stakeholders As identified earlier, there has been a general trend in the 1990s to include the concept of stakeholders in EFL evaluation literature. The concept is, however, more developed in other fields such as management. Savage et al. (1991), for example, argue that managers must take their organization’s stakeholders seriously: "[o]rganizational stakeholders are a significant force affecting organizations. Today, organizational strategists must consider how to manage the stakeholder" (p. 61). They go on to propose methods of managing stakeholders in ways that will ensure maximum benefits to the organization. They analyze stakeholders according to their potential to either co-operate with, or threaten the organization and then outline ways in which organizations can involve stakeholders in ways that either reduce their potential threat or increase their potential co-operation. The model of stakeholder identification presented by Savage and colleagues includes all the organization’s stakeholders and not just those involved in a particular area of business or programme and is too broad a model for the context I was working within. They do, however, make the important point that the identification process should be ongoing. Pouloudi and Whitley (1997) also provide a comprehensive model for stakeholder identification in inter-organizational systems; they contend that many analyses of stakeholders fail to explain how the main stakeholders were identified in the study. They use the terms "hubs" for those initiating and sustaining the system and "spokes" for those participating in it, and maintain that the context is an important consideration and as the context changes, so might the stakeholders. They believe that it is the interrelations between stakeholders and their differing interests that is significant in stakeholder analysis. Their model is flexible, firstly, identifying generic groups of stakeholders followed by an analysis of their perception of the environment, which may lead to changes to the original stakeholder map. They conclude that researchers must decide where the stakeholder identification should stop as the more stakeholders involved in an analysis, the greater the chance of conflict in their accounts of the situation. Other considerations when deciding which stakeholders to include in a programme evaluation are their ability and willingness to contribute to the evaluation. Greene (1987), included a diverse group of stakeholders in her evaluation design, evaluating community services, and some had to be dropped from the evaluation as (p.386), "… they did not know enough about the program to make their participation meaningful…" Similarly, Coleman (1995), identified a broad group of stakeholders using a model that specifies the domains of impact of a language programme at three levels: individual, institutional and national. This far-reaching model may be of use in large-scale development programmes but is not applicable to smaller programmes. It is, therefore, important that stakeholders are close enough to the programme to (a) be sufficiently informed to evaluate it, and (b) have enough of a stake in the programme to motivate them to be involved in the evaluation. A clear overview that applies the concept of stakeholders to educational evaluation is provided by Aspinwall et al. (1992), who claim that in both planning and evaluation it is important to consider the stakeholders and their expectations. They argue that their stakeholder analysis "allows us to think critically about the ‘who’ dimension and asks us to go a step further to examine what the stakeholders’ expectations might be with regard to the institution, development or programme" (p. 91). The following model is used for mapping stakeholders:
Figure 1: Stakeholder map of a secondary school (Aspinwall et al. 1991: 86) Their commentary states that: "[t]he institution is placed at the centre and the perceived importance of each group is indicated by the length of the arrows, the more important stakeholders placed nearer the centre" (p.85). This model is useful, although it is designed for institutional evaluation and needs adapting for programme evaluation placing the programme in the centre. Rea-Dickins and Germaine’s model (1992) of the context of evaluation is useful for such refocusing. The other limitation of this model is that it does not identify the interrelations between the stakeholders or the channels of communication between them. Johnson and Peterson (1994) with their ‘program matrix’ model below, firstly, distinguish between ‘processes’, the stages of the programme and the ‘elements’, the people and materials involved in the programme. In this model, evaluation is only part of the matrix. They do, however, see the identification of stakeholders as essential in any framework for describing or analyzing language programmes. They identify four major groups of stakeholders: learners, teachers, administrators and controlling authorities. Their model below shows the relationships between the groups and the outside influences on each; this places the stakeholders in a broader context, which is useful. The model does, however, fail to identify the stakeholders’ relationship to the programme, which seems to be a great omission.
Figure 2: Influences on principle stakeholders (Johnston and Peterson, 1994: 73) In establishing my method of stakeholder identification, I have synthesized the key features of the two models above in order to provide one suitable for the context of my investigation. In mapping the stakeholders in the programme, I wanted to show both the proximity of each group to the programme, and the interrelationships between the different groups. The map below also divides the stakeholders into two key groups, those within the company and those within the school. The diagram is not exhaustive, as I feel that stakeholders with a low stake in the programme are unlikely to be able to contribute much to the evaluation. I used my own knowledge of the programme as an insider to identify the key stakeholders that should be included and also gained information and approval from the Training Manager on the stakeholders within the company. Finally, the model of mapping aims to be flexible so that it can be used in various contexts.
Figure 3: Stakeholder map of the Ericsson/ELCA language programme Previous evaluation models involving stakeholders As was seen earlier, stakeholders are becoming increasingly important in educational evaluation theory. Much work on stakeholders comes from the evaluation of adult literacy programmes in America and looks at how stakeholder evaluations can democratize the evaluation process. Jurmo (1994) identifies the stakeholders in such programmes and argues that the evaluation should:
He argues for a ‘collaborative approach’ in which "[a]ll stakeholders — especially learners have responsibility for deciding content and format of program activities" (p.14). He analyses various forms of evaluation and argues that if stakeholders are involved in the evaluation this is likely to build support for the programme. He is in favour of qualitative approaches to evaluation and describes the concept of ‘responsive evaluation’ which identifies the goals of stakeholders. This involves observation of the programme and the materials, comparing the difference between the goals and the implementation. Finally, he discusses ‘empowerment evaluation’ which provides participants with the necessary skills to conduct self-evaluation. Although interesting, such a democratic and egalitarian approach may not work in more commercial or politically sensitive settings, and such involvement of all participants would only be possible if they had a strong commitment to the programme. Dean (1997) similarly argues that stakeholders should be involved in establishing the evaluation criteria, although a valuable approach it seems costly and time consuming and may only be appropriate for large or long-term programme evaluations. Another model that claims to incorporate stakeholders in the evaluation of teachers is presented by Gurr (1996: 6-8) his model of teacher evaluation includes several stages that explicitly involve stakeholders:
Although his checklist is of use, his elaboration of the points is unclear and incomplete. He identifies the main stakeholders as the teachers and the learners without justifying this; surely the school’s management would also be key stakeholders in a student evaluation of teaching. There do not seem to be many examples of studies in the literature that fully incorporate stakeholders, and those that do claim to involve stakeholders seem to do so with varying degrees of success. An example of a study that involved stakeholders and focused on the management of change to the assessment procedures of a medical course is discussed by Usherwood et al. (1995). They identified the stakeholders as those concerned with the assessment process. They surveyed the students and the honorary clinical teachers (HTCs) and interviewed the departmental tutors about the assessment process, analyzing the results to find common themes. The findings were then discussed by the tutors and HTCs to reach a consensus on assessment processes. One important conclusion that they make is that it is important to involve those who are going to be involved in future changes. Although it is questionable whether the level of student participation in the model was sufficient. Another study that involved stakeholders and assessed the merits of such an approach was carried out by Greene (1987); like Usherwood et al. the evaluation is not of a language programme, however, it does present some interesting findings. The evaluation is of local human service agencies and considers the link between participation and evaluation use. A varied group of stakeholders were extensively involved in the evaluation design and determining the areas that the evaluation should focus on. All stakeholders were also involved in the preparation of a written programme description. The evaluation was viewed as a process, and she believes that this was the key to its effectiveness, as all parties benefited and learnt from their involvement in the evaluation process. She concludes that "stakeholder participation in evaluation design is worth the effort" (p.393). Although it may be felt that such extensive involvement may prove too costly and time consuming for the evaluation of many programmes. Another difficulty arising from drawing on too broad a group of stakeholders was faced by Coleman (1995), who attempted to assess the impact of a training programme on a varied group of stakeholders in ten categories ranging from the course participants to the hotels in which course participants stayed. His results show the difficulty of obtaining meaningful data from the more removed stakeholders. For some groups of stakeholders he was unable to obtain any data and for others the response rate was low. Evaluators must, therefore, balance the value of information from the various stakeholders against the effort required in obtaining it. An early example of an evaluation of a language programme that involved stakeholders successfully was conducted by Barrington (1986), whose ‘naturalistic model’ involves all the key stakeholders, although the term stakeholders is not used. Data on the programme were collected from the students, the teachers, the course administrators and ‘interested parties’ and the final three groups also played a role in feeding back into the evaluation loop with recommendations before the final evaluation report was completed. This early attempt at involving stakeholders systematically in the evaluation process seems to have been successful. He concludes that "[t]o meet the needs of a variety of audiences and interest groups and to foster workable outcomes, an emergent, collaborative design engendered a responsive evaluation study which had a positive impact upon the program under review" (p.49). A final study that attempts to apply the concept of total quality management (TQM) to the evaluation of mainstream secondary education is presented by Davies and Ellison (1995). They focus on an analysis of the levels of customer satisfaction amongst the schools’ ‘clients’. They identified the clients to be the parents, students and the teachers, and used questionnaires to establish satisfaction levels. The results over a period of time aim to show trends in areas of satisfaction. The information provided was used for decision-making within the schools and the results allowed them to prioritize areas most in need of attention. They argue that there needs to be sensitivity in the reporting of findings. They conclude that "[t]his approach has provided valuable insights for the case study schools to improve their management practice but the approach is very contextually-bound" (p.49). This study provides a useful model of evaluating stakeholders’ satisfaction levels in education.
Conducting the stakeholder evaluation The table below provides a checklist of the stages that I used in planning the stakeholder evaluation. Table 1: Stakeholder Evaluation Checklist
The design of the evaluation aimed to incorporate stakeholders at various points in the delivery of the programme, focusing on the stakeholders’ position in relation to the programme and their willingness to contribute to the evaluation. Methods of data collection selected reflected the numbers of people in the given stakeholder group and the relative ease of access to the group. Questionnaires, interviews and focus groups were chosen as research tools as they were considered appropriate for measuring the construct of satisfaction. Data were collected at mid-course and post-course stages and findings were fed back to the relevant stakeholders at appropriate times to ensure that they could be utilized in informing the continuous process of programme design. The table below outlines the methods of data collection utilized and the stakeholders with which each was used, as well as the point in the programme at which they took place. Table 2: Methods of Data Collection
The focus groups provided very good opportunities to gauge opinions and also acted as a testing ground where those involved could discuss ideas with their peers. One problem, particularly with the students, was they were not really able to come up with original ideas or suggestions for improving the programme. However, I found that when various alternatives were suggested, the ensuing discussion soon developed in such a way that the more or less favoured alternatives were clearly identifiable. This might suggest that we can use focus groups as a testing ground for new ideas relating to programmes and they may, therefore, act as a forum where ideas can be discussed in order to get a feeling of opinion relating to a topic. The results from the focus groups were much clearer and more useable than those obtained from the questionnaires.
Recommendations The recommendations that I would make for further evaluations of this kind are:
Conclusions There is much scope for developing ways of evaluating, designing and managing language programmes that involve key stakeholders. I am sure that a lot could be learnt from applying the model to a variety of language programme contexts. Greater stakeholder involvement can lead to a more democratic approach to learning, and to course design that can take the burden off any one party to provide the ‘service’ or ‘knowledge’, with the result that this redistributes more equitably the balance of power in the decision-making process. This gives all those involved a greater stake in the programme and forcing them to take a more responsible and proactive approach to it. There will always be a key role for those who are the specialists in the field to take, however, they may no longer be able to impose their knowledge on other parties unquestionably. The more sophisticated and discerning stakeholders are, the more say they will want in shaping the service that they are receiving / providing. I feel that formative stakeholder evaluations have the potential to substantially redistribute the power within certain kinds of language programme, as well as enhancing their design and delivery. The model has proven to be informative in the context of corporate language programmes and is likely to be equally illuminating if utilized in the context of development programmes, which similarly serve the needs of diverse groups of stakeholders.
References
|
|
 |
PDF version of this paper |
