The Scottish Government

« Previous | Contents | Next »

Listen

3. Forming a research and evaluation group

3.1 Rationale

The schools have responded in different ways to the evaluation dimension of their School of Ambition status (see Table 5). The different ways of organising the school evaluation group reflect: (1) the existing culture within the school; (2) dominant perceptions of what constitutes appropriate 'research' in this context; (3) the available resource that it is judged appropriate to deploy to this task.

At the 'One Year On' event for the first tranche of schools in September 2006, the leader-managers of the Schools of Ambition were encouraged to consider forming a school enquiry group (see Appendix 2). Schools were advised to consider sharing responsibilities for evaluation across a small team of individuals who would each hold responsibility for leading specific strands of enquiry at particular points (individual action research projects). The synthesis of these strands would inform future developments at a whole school level. The Research Support Team was keen to ensure that the work undertaken was sharply focused, purposeful and manageable in the context of the work of schools. The case for forming a research group can be expressed as follows:

• Sharing responsibility - dividing tasks to ensure manageability, range and focus
• Strengthening relationships - critical friendship, peer support, cross-school links
• Developing capacity - skill set of individuals and resource for organisational learning
• Contributing to sustainability - year-on-year development (staff turnover)

Table 5: Schools' approaches to research management and organisation.

Responses to this encouragement varied across the schools. Not all of them were receptive to the idea of forming a research group and some indicated that they had not been fully aware of the research and evaluation aspects of the SoA work. Where a group was developed however, and in keeping with the autonomy afforded through the SoA approach, the composition of the evaluation team was a matter for local deliberation within each school.

3.2 Organisational forms

Four forms of organisation have developed across the first tranche of 21 schools (see Figures 1-4 below). In the early stages of the research, all the schools created a traditional command structure, which reflected existing hierarchical relations within school. The need for project management was met through delegation to senior colleagues in school, most commonly depute headteachers with a School of Ambition remit. In some cases the headteacher assumed responsibility for conducting evaluation activities within their role; as integral to their strategic leadership of the programme. Where this was the case, accounts reveal that this was motivated by concerns regarding workload issues in school and limited capacity to create additional resource to support evaluation responsibilities ⁵. At the end of the first year of the research strand, this remains the case in just one of the first 21 schools.

Five schools continue to operate with one or two lead evaluators reporting to the headteacher. This model creates a concentration of activity among a small group of senior staff. It has the benefit of offering clear channels of communication among a small group who meet regularly; but has the limitation of concentrating skills and 'ownership' of change within a narrow stratum of staff. Much has been written of the 'cultural distance' of leader-managers from classteachers and the recent flattening of career structures within teaching is aimed at increasing a sense of colleagueship. The notion of 'distributed' or devolved leadership (Spillane, 2006; Gronn, 2003) is difficult to discern in the approach being taken in these schools, which retains something of the tenor of models of 'heroic' or transformational leadership. In interviews with members of the leadership group, strong leadership was frequently associated with 'bringing hearts and minds with you' (depute head); attending to the affective dimensions of headship (culture management). The 'leader-follower' relationship in these accounts can sometimes appear rather more prescriptive than empowering; inviting 'buy in' rather than encouraging creativity.

'It's making sure you are giving them time to follow behind you before you hit them with something else new and slump them back down again' Depute head.

In three Schools of Ambition, the approach taken has been to involve individuals with non-teaching roles as principal agents in their evaluation (as in Figure 2). In two schools the evaluation is delegated in its entirety to one senior Project Manager. Two other schools have elected to involve project assistants with non-teaching responsibilities in evaluation activities. This approach clearly demarcates roles and responsibilities and has the potential to strengthen relationships with other professionals in school (a key objective in the promotion of full service/extended/community schools). However, reliance on non-teaching staff may also signal a view of evaluation and review activities as divorced from the core responsibilities of the classteacher. Whether motivated by pragmatic purposes and a desire to protect staff or not, the restriction of involvement impacts on capacity building and the potential for professional learning.

Figures 1-4 Organisational structure of research roles in the Schools of Ambition

Case Three

In School C, the headteacher has taken the main responsibility for organisation of the evaluation activity. One strand of the school's transformational plan is a project which employs a youth worker whose time is funded separately by a children's charity. The person's remit is to work with the students to achieve the project aims and also with pupils who are taking the lead in another strand of the TP. For evaluation purposes, the headteacher decided that in view of worker's involvement with the pupils and time availability, s/he was the person best placed to collect data from pupils who had taken part in the programme, rather than the teaching staff in the school.

With regard to evaluation of other strands of the school plan, at the outset the mentor actively encouraged the headteacher to identify staff who might take on this role. However, the headteacher felt that the staff were already committed to various development groups and working beyond their required duties, and could not be asked to take on additional commitments.

'Basically it's a workload issue, in terms of staff in the school…I didn't have anybody else to call on to get involved. Most of staff are down to the minimum time. Rural authorities don't get as much money as urban authorities to staff schools. I'm staffed at a significantly lower rate than a similar sized school in the central belt… Apart from PTs virtually all staff are down to the minimum non-contact time, not conducive to taking on additional work'. Headteacher.

With the support of the mentor and advice from the critical friend the headteacher is taking responsibility for devising research instruments to evaluate the impact of several strands in the coming year. The mentor has also given advice on the analysis and reporting of findings from pupils', staff and parents' feedback on the current social enterprise project and their aspirations for the development of future activities.

Although there was one member of staff undertaking research through the Chartered Teacher programme, it was not related to School of Ambition.

'I wasn't aware of anyone else as yet interested.' Headteacher.

3.3 Engagement with research and evaluation

Within the schools where responsibility is concentrated, there is some recognition of the value of a systematic approach when managing significant change and the importance of connecting curriculum development and innovation with processes of reflection and review. Given the scale of the proposed changes, and often the pace of development, it is difficult to monitor processes effectively across the spectrum of activities without clear planning. One depute head spoke of the need to move from 'intuition' and informal questioning to more rigorous forms of sustained critical reflection.

Case Four

School D's transformational plans involve six interrelated strands. Each strand includes a team of staff and senior pupils who hold regular meetings. The school is fortunate in having two non-teaching staff who play a key role in the project team.

The DHT who leads the team indicated a strong commitment to evaluation activities and the school had already collected data from pupils via questionnaires at the beginning of year 2. However, the evaluation has largely been the responsibility of non-teaching staff who have collected additional data, for example, video-recordings of lessons.

The DHT indicated that the challenging nature of carrying out the school's plans has meant that staff had to focus on 'the doing' rather than the evaluation, despite advice and encouragement offered by the mentor who produced evaluation plans for each strand. A visit to the school by the mentor in April provided the opportunity to meet face to face with the DHT, non-teaching project team, three teaching staff and a 6 ^th year pupil. As a part of the workshop the mentor provided an overview of the research process, and pros and cons of the different methodologies to use for evaluation.

This meeting provided the impetus for the DHT to recognise the need for a member of the teaching staff to take on a central role in the evaluation activity. Additionally as senior students have played a significant role in the delivery of the TPs, the DHT plans to include them in collecting data.

'Year one was laying the foundations. You do stand back and ask, 'are they secure, are they embedded?' Those questions, we were asking. All S1 pupils completed a questionnaire. 1C became a class group that undertook a pilot for the year to see if there was potential. We're now building the bricks of what we are doing. In that sense we've been analysing what we've done, how we've done it, almost weekly when we see all the staff and student coordinators. Whether we then look at the data they gather, and whether we are being rigorous enough to ensure they are doing it, is a different story. That's something we need to sharpen up. During this year we've seen the importance of that. You intuitively watch and know it's working or is not working, [we need to] formalise that and be willing to be quite rigorous about that. I think that is how it has evolved'. Depute headteacher.

Case Five

The Leadership Group in School E identified four areas that they wanted to investigate in a systematic manner. These areas had provoked some curiosity; they posed 'problems'/questions where there was a need for further information. The Leadership Group was also concerned to encourage the professional development of teachers at a relatively early stage of their teaching career (4-5 years experience). A group of teachers was convened to establish whether there was an interest in forming a research group to investigate the areas prioritised by senior management. A small group of teachers met with a member of the Research Support Team to explore ideas, strategies and methods. Each teacher took responsibility for preparing a draft plan for one line of enquiry. Through an iterative process, with support from the external research mentor, research plans were formulated for each project. The teachers accepted a modest honorarium for the additional work involved in leading these strands of enquiry and agreed a timeline for completion of the work staggered over a year.

The group struggled to find time to meet and share ideas and plans in the early stages. The mentor forwarded relevant research briefings and encouraged group members to construct a timeline with target dates for the completion of activities mapped against pressure points in the school calendar. The group identified areas where they needed support, for example working with focus groups and constructing interview schedules, and these needs will inform the programme of support for the coming months.

'The biggest thing that we have been finding is that people need to talk to each other and spend time with each other and that is really difficult as a teacher because your job is to teach pupils in a classroom and you are on your own in that. There is very little time to actually speak to other people and spend time with other people sharing ideas and talking'. Classteacher.

The research group plans to hold six-weekly working lunches with the Leadership Group to share work in progress. They are supported by a Depute headteacher who provides coordination and facilitates communication/requests for support/ access to resources.

The headteacher plans to expand the group to include a wider range of staff. It is planned to invite post-probation teachers to participate in the group for specific purposes e.g. supporting the lead teacher-researchers where appropriate. It is also planned to recruit more experienced senior colleagues to join the group to encourage cross-departmental and cross-role collaboration. One of the strands of enquiry focuses on primary-secondary transition and this brings a further dimension to the work, strengthening cross-sector links.

Case Six

The headteacher of School F was initially concerned about finding the resource in a small school to embed evaluation activities within all the developments associated with the Transformational Plan. The school has six interrelated strands within the TP and fewer than forty teachers. With the support of the local authority, the school was quick to begin implementing change. Whilst conscious of the dangers of 'initiative fatigue', the headteacher wanted to establish the impact of the changes introduced. Were they producing changes in teaching and learning? How could good practice be identified and shared? How could positive developments be sustained beyond the pilot projects?

In January 2007, the six member research team was established. A lead teacher took responsibility for evaluating the impact of each of the six strands of development contained in the Transformational Plan. In addition to synthesising routine school data (e.g. attainment, attendance, referrals etc.), the teacher-researchers would investigate the processes of change for the purposes of lesson drawing and wider dissemination. A principal teacher was appointed school 'Research Coordinator' to support the group and liaise with the external research mentor. The research team met with the mentor in January and May and generated evaluation plans for each strand.

Case Seven

In School G, key themes in the transformational plan included promoting leadership across staff and pupils and empowering pupils. To facilitate this, the school has introduced three strands to drive improvement:

• Excellence through the Arts
• Staff leadership
• Pupil leadership.

Each strand has a member of the school management team to oversee and manage activities. These Strand Leaders remit also entails responsibility for conducting research and evaluation relevant to the appropriate transformational goals. They work closely with relevant staff and report to the Depute Headteacher and Headteacher. The research findings then inform school planning. Some senior pupils have already been involved in assisting with data analysis and it is anticipated that other teachers and pupils will become increasingly involved in the research process.

One Strand Leader has conducted a major staff survey that has informed the school's progress towards Investors in People status and influenced school planning concerning the roll out of SoA activities and wider involvement by departments.

Case Eight

School H provides for pupils with additional support needs arising from social, emotional and behavioural difficulties. The focus of the Transformational Plan is on bringing together school provision and alternative strategies so that a range of flexible packages can be offered in partnership with mainstream education and partner agencies. Initially, SoA activities focused on developing skills and leadership across all staff to take forward the transformational plan. The emphasis then moved to implementing the flexible curriculum programmes and approaches.

Research and evaluation in the school concerning SoA is overseen by the SMT and has recently become much more decentralised with the various programme strand co-ordinators and staff (including teaching, non-teaching and specialist staff) playing a key role in gathering evidence of progress. It is also hoped that students will also become more involved in the research process. The range of staff and others working together on the transformational activities means that a particularly diverse range of support is required from the research mentors.

The Research Support Team has encouraged attention to the capacity building opportunities afforded within Schools of Ambition. Adopting an extended view of professionalism ⁶, the Schools of Ambition programme offers the opportunity for participating teachers to develop skills of systematic enquiry with the support of an external research mentor. Six schools (28%) have moved towards this model. Mentors have provided support through the delivery of research skills workshops to groups of teachers, individual support meetings with teacher-researchers, and through the provision of relevant research reports/briefings and other literature.

What emerges from these accounts of the efforts in several of the schools to develop a systematic approach to evaluation and research is that it is sometimes difficult to move from a very functional schematic orientation where evaluation becomes an attempt to measure effectiveness, to a more collaborative and formative approach where evaluation is seen as a way of raising questions as much as a means of providing 'results'. The pressures on schools, their staff and pupils to improve performance, particularly through exam results, is considerable. The pressures to respond positively to new policy initiatives from both local and national levels continue. However, what these cases demonstrate is some success in several of the schools in 'taking control' of the research and evaluation agenda, such that it does provide an enhancement of the development process.

As we have seen, the move towards assembling a research group has been a relatively slow process in most cases. Following initial visits to the schools by members of the mentoring team from October 2006, six schools had established a group by February 2007. Such groups are becoming increasingly significant in ensuring that a rigorous and integrated approach to research and evaluation is sustainable. The recruitment of individuals to the evaluation team is an important development, but it is also important to build conditions conducive to the development of the group. In some cases group work has failed to prosper and responsibilities have fallen back on one or two individuals. Once formed, groups need support to maintain momentum. The many demands on teachers' time may threaten the activities of the group and reduce possibilities for peer support. The pressure to attend to 'urgent', immediate tasks and to focus on the imperative of 'delivery' can work against reflection, learning and longer-term planning. It is important to build reflection points into the evaluation process to synthesise developments and inform next steps.

'The initial discussions I had with the Head teacher identified the need to put together a small team. I expected the small team to have been a bit more involved, but because of various other commitments that maybe has not come to fruition the way I thought it was going to come to fruition. There have been one or two other individuals involved but a lot of the time they are being called to go and do different things, at different times, and we are finding that aspect of it quite difficult. So, I have ended up doing quite a bit of it, the research and the planning myself which really was not part of the original set-up.' Principal Teacher.

Segmenting the Transformational Plan and delegating responsibility to strand leaders is an effective way of initiating research across a number of themes. It is important, however, to link individual activity with whole school development. This requires a strong focus and shared understanding of purpose and clear communication and linkage across projects. The role of the school Research Coordinator is important here.

'Not having an overall picture. It's like a jigsaw. Different people are doing different things in different places. The senior management has one idea of what is going on and certain expectations and then [other colleagues] may have other ideas about what is going on. It's very difficult trying to get everybody thinking in the same way or knowing exactly what is going on and sharing the same goal. It's coordination and getting everyone together when you need them to get together. That's really difficult logistically, managing to do that. I need to draw on other people's support to make that happen.' Classteacher.

From the introduction of the research strand, school leaders were concerned about the workload implications of undertaking 'additional' evaluation activities for staff. As one headteacher in October 2006 explained, 'your lieutenants who are actually making the changes are the same people all the time'. Ten months into the programme, the composition of the research group in most schools does not appear to have significantly widened participation beyond senior figures or enthusiastic innovators in particular fields. The journey from 'pockets of innovation' to school transformation is not insubstantial. The culture of 'voluntarism' in teaching appears to underpin a great deal of activity.

'It's the enthusiastic and keen staff that drive things forward - and that's not to say that there is not a whole lot of outstanding work going on in other Departments with people who do not give so freely of their time, but they invest their time in the bread and butter issues - and it's the enthusiastic staff that give the extra time.' Headteacher.

The work by the Research Support Team over Year 1 has certainly established the fact that school cultures vary in the degree to which they are open to systematic practitioner-led enquiry 'from within'. When initiating self-study within a workplace setting it is important to engage in dialogue to develop a shared understanding of the parameters and purposes of any investigation. One classteacher commented, 'I would have loved to have had more freedom over the questions I could have asked and who I was allowed to put the questions to'.

Communication is also important in sustaining participants' commitment throughout the evaluation process. Action research is premised on improving professional practice and researcher-evaluators are keen to share the insights that are emerging through their work. The motivation for participating classteachers was to 'make a difference', to improve practice in a specific area and share developments with colleagues in school. Teachers commented that they 'expected to learn something', 'to make some sort of contribution to the school', 'to improve things by doing this'. The provision of opportunities for dissemination needs to be planned in order to promote wider learning.

Case Nine

The headteacher in School I initially questioned the relevance of the research strand within Schools of Ambition, suggesting that the school had a robust evaluation strategy that had withstood scrutiny from the local authority, SEED Senior Advisor and HMIE. The mentor stressed the formative role of action research and offered to support the school in identifying appropriate instruments for their evaluation.

In February a team of five teacher-researchers was assembled and attended a workshop on research design and analysis delivered by the mentor. The workshop addressed questionnaire construction and the use of qualitative research instruments.

The group were reluctant to set target deadlines for the completion of evaluation activities and stressed that they were fully occupied in delivering the objectives of the transformation plan. The proposed research activity was summarised into one A4 outline, which was shared using the VRE. The plan indicated three strands of enquiry and a combination of quantitative and qualitative approaches including pupil and teacher questionnaires, pupil focus groups and analysis of pupils' work.

In May 2007, the Research Coordinator, who is working towards SQH, was the only active member among the original research group. The mentor developed a questionnaire as the main evaluation tool and provided advice on data analysis. This involved reviewing data files and suggesting ways of manipulating the data e.g. cross-tabulation using Excel.

At this stage, the evaluation through teacher-led enquiry in School I is limited and the mentor has taken a role in the design of instruments. Capacity building is restricted with the decline in active membership of the research group, although the research activity is contributing to the professional development of the lead teacher.

'Having the time to explain to people about what it is you are doing is really important, because there is no point in me just doing the research if it does not actually have any impact and nobody actually knows what it has found out…It's the communication aspects of it which I think is a huge challenge for any researcher….that you have found something out but it only really helps if other people know about it and act on it.' Classteacher.

'I think that the majority of the staff don't even know that we are doing anything yet. It's not had an official launch.' Classteacher.

In the course of the leadership interviews conducted from October 2006, some schools described the process of constructing the original Transformational Plan as a negative experience. Headteachers commented on the difficulties in navigating the different timescales, financial regulations and reporting templates of local authorities and SEED.

'It is very, very tricky because you are trying to satisfy two separate and distinct demands from just one particular thing, so it's all work. It's bureaucratically all work. There is a lot of paperwork involved in it. It's not terribly helpful... It's complex. It's highly complex. It's a bit like trying to fit square pegs into round holes.' Headteacher.

The experience of developing different versions of the Transformational Plan has left some teachers feeling wary and uncertain about what will be expected in terms of the reporting of evaluation activities. Although the school-owned evaluation strategy should be mapped carefully against each objective set out in the TP, there remain residual concerns that a new agenda may emerge to which the schools will have to respond. The extract below also reveals a continuing dislocation of development work ('doing') from processes of systematic reflection and review.

'We are furiously collecting, digesting, researching but I still do not have a clear picture of what I will have to be giving…We do not have a clear idea of what is expected. Nobody from SEED has said, 'here is what we will require'. I do not think they know. In a sense that is positive. So you need to have six sets of statistics because there is always this worry that if the evaluation does not meet these undefined criteria that somehow the project will be deemed to have failed, rather than the evaluation.' Head teacher.

3.4 School culture and evaluation

In many schools there is a view of evaluation activities as an 'add-on', 'extra work', a 'project' that is tangential to the core objectives of the Transformational Plan. This is evidenced in the relatively low level of awareness and value attached to the range of evaluation activities that are being undertaken; and a tendency for activity at this stage to be fragmented rather than joined-up. In some schools the evaluation process is viewed as marginal, rather than central to the development process ⁷.

'I wonder if I was to leave, would it be sustained? I'm not sure whether they value it enough yet or see it as integrated.' Depute head.

'There needs to be a whole school evaluation mindset…this needs time to develop…its easier for people to focus on their own parts but we need them to join it all up.' Depute head.

A model of full integration of evaluation is outlined diagrammatically in Figure 5, drawing from Hopkins (2007).

Figure 6 shows the different ways in which the distribution of roles and responsibilities for evaluation activities are currently configured in the Schools of Ambition. A distributed division of labour (example C) spreads the workload and has the potential to build capacity by enhancing skills among a wider audience and is closest to Hopkins' integrated model, above.

'My SMT are pulling their hair out because they are over the top with work, the day-to-day work of managing a school with a complex set of social circumstances here and at the same time deliver on the School of Ambition agenda.' Headteacher.

'What still inhibits change and transformation are people that are concerned about moving forward, concerned about changing how they do things. People either have always done it this way, or want to always do it this way.' Headteacher.

Figure 5 Integration of development and evaluation

Adapted from Hopkins (2007:110)

Figure 6 Distribution of roles and responsibilities for evaluation

In reviewing the progress made in developing their research and evaluation by the first tranche of Schools of Ambition over the first year of this work, it may be helpful to suggest that each school may be able to locate itself along a continuum running from a predominantly 'instrumental' approach to a predominantly 'developmental' approach. The two characteristics of the extremes of this continuum are set out in Table 6 below. The developmental end of the continuum may be seen as more challenging than the instrumental, but also as one that is more likely to be sustained and to have deeper effects on the school. During this first year, most of the schools have been working closer to the instrumental end of the continuum than the developmental. However, there have been movements during the year, towards the developmental (although we have also indicated one or two situations where the movement has been in the opposite direction). The Research Support Team sees its role as helping schools to move towards the developmental end, as work progresses over the years ahead because this end of the continuum does represent an approach that is most likely to produce sustained change, both in terms of 'organisational learning and development' and in terms of the professional development of teachers and other staff.

Table 6 A continuum of approaches to evaluation

The instrumental approach tends to emphasise 'getting the job done' rather than a broader commitment to developing skills to enhance learning. The utilitarian orientation of the former is dominated by attention to routine data collection, monitoring and audit that can be concentrated within the senior management team. This approach approximates Tyler's objectives model of evaluation ⁸. In contrast with the narrow attention to outcomes measurement in the instrumental approach, the developmental approach attends to issues of process. It follows that developmental approaches are more likely to devolve responsibilities to those who are positioned to offer finely grained case studies of programmes in use. Developmental approaches also accommodate members joining and leaving the evaluation group.

By identifying a range of positions along a continuum, the Research Support Team are not suggesting that schools fall neatly into discrete categories. The complex array of objectives expressed in the Transformational Plans requires the selection of appropriate tools that have the capacity to both reliably measure impact and illuminate processes of change.

'It is not the mechanistic completion of progress and success checks that is important, but rather it is the enhancing of the teachers' professional judgement that is the crucial aspect of embedding an ethos of enquiry and reflection within a school' Hopkins (2007:110).

« Previous | Contents | Next »