The Scottish Government

« Previous | Contents | Next »

Listen

Appendix 7: Questionnaire with overall returns

What is this questionnaire for?

This questionnaire is part of a wider evaluation work that the Research Support Team are undertaking to look at staff's views on the support they have received and still require as well as feedback on the overall impact of the Schools of Ambition ( SoA).

Who needs to fill it in?

We are asking staffwho are involved in any way with evaluating Schools of Ambition work within their school to complete this questionnaire. This can include: those with a managerial/co-ordinating role, those delivering or supporting the SoA programme (including administrative staff) but who have some involvement in evaluation and monitoring. We will explore the role of pupils and others in the evaluation process at a later stage.

Your contribution is very important to the success of our study and the Research Support Team and SEED value your views.

No individual will be identified in any reporting of the findings unless with their express permission.

Please return your completed questionnaire as soon as possible but no later than 22 ^nd June 2007.

Questionnaire report

Purpose, design and distribution of the questionnaire

At the end of this school year, a questionnaire was distributed to all of those in the schools who had been most closely involved in the development of the research strand, including school leaders and teacher researchers. This questionnaire (above) was intended for those individuals across the twenty-one tranche one schools who have a key role on managing and conducting evaluation activities for Schools of Ambition in their schools.

The survey focused on participants' experiences of engaging in systematic evaluation activities and how confident they felt in undertaking this work. It was also important to identify support needs and preferences as we approached the second year of the programme and a significant increase in the number of participating schools. The questionnaire also sought to provide data on whether (and how) the original transformational plans were being modified and to reflect at the end of the year on the key drivers and barriers to school self-evaluation. In the light of insights gathered from the recent round of teacher-evaluator interviews, it was also important to consider how participation in Schools of Ambition evaluation activities was contributing to the professional development of participating teachers (promoted and non-promoted).

A draft questionnaire was constructed by colleagues in the SCRE Centre in late May 2007 and circulated to all members of the mentor team at the universities of Glasgow and Aberdeen for comment and review. A copy was also shared with Deirdre Kelly, SoA Project Coordinator within SEED. Following a process of refinement, which included piloting the instrument with a group of teachers in one school, an e- PDF version of the questionnaire was distributed to 62 leader-managers and teachers on 18 ^th June 2007. To increase the return rate, colleagues were offered a variety of methods of response. These included completing the questionnaire online and responses by mail/fax. The initial email distribution was followed by telephone and email contact by members of the mentor team. Data entry closed on 10 ^th July for analysis of the first round of questionnaires. 30 completed questionnaires were returned (a response rate of 48%), which included a least one return from 20 of the first 21 tranche one schools.

For the purposes of the survey, the following definition of evaluation/ (action) research was offered:

'systematic enquiry conducted by any teaching and non-teaching staff, pupils and other stakeholders to help to understand the outcomes and processes of Schools of Ambition activities'.

Survey respondents

Responses were received from teachers employed in 18 of the 19 local authorities in tranche one (Table 1). 46 per cent of responses (n=14) were provided by members of the Leadership Group in the Schools of Ambition (Headteachers or Depute headteachers). 20 per cent of respondents (n=6) were classteachers leading action research projects in their schools (Figure 1). A further 20 per cent were in Principal Teacher posts and one respondent identified him/herself as a Chartered Teacher.

Table 1 Local authorities of respondents

Figure 1 Professional title of respondents

Involvement in evaluating School of Ambition work

The individuals who elected to participate in evaluation activities for Schools of Ambition undertook a range of responsibilities. Table 2 below shows the distribution of responsibilities according to self-defined professional titles.

Not surprisingly more senior members of staff associated their role with coordination and design. Of the 5 headteachers who responded, 2 did not identify their roles as determining evaluation priorities and 2 were involved with every activity listed.

The 6 participating classteachers reported a more directed role, largely associated with collecting and analysing data/evidence. These colleagues were more likely to be leading action research projects in specific areas of provision.

Table 2 What role do you have in evaluating your school's SoA work?

Out of the 30 respondents (37%), 11 had no previous experience of conducting evaluation or research activities before their involvement with School of Ambition.

The range of previous experiences included academic study on award bearing courses, including pre-service (university degree) and post-qualification awards, such as Masters degrees (including Chartered Teacher) and the Scottish Qualification for Headship ( SQH) (table 3).

Respondents also reported general involvement with school data gathering activities for the purposes of school development/improvement planning e.g. pupil/parent/staff surveys, audit/review of curriculum provision and outcomes.

Table 3

Despite over a third of the teacher-evaluators having no previous experience of research or systematic evaluation, relatively high levels of confidence were reported across the 30 responses, especially with regard to determining priorities, coordinating activities and reporting findings (Figure 2). Teachers were least confident in their ability to employ qualitative research designs. 37% (n=11) of respondents reported that they lacked confidence in designing or conducting qualitative evaluation methods and 40% (n=12) were not confident in their ability to analyse qualitative data. The teachers felt slightly more comfortable working with numerical data. Twenty-three teachers (77%) were using formal questionnaires in their evaluations. A particular training need was identified concerning Excel data analysis, especially the use of pivot tables to manipulate statistical data.

Support from Research Mentors

All 21 of the first tranche schools received a number of visits over the course of the school year from their designated research mentor. It is the mentor's role to act as 'supporter', providing advice and guidance to the teacher-evaluators on research design, the range of available and appropriate methods of enquiry and techniques of data analysis. The mentor can also provide guidance on access to research digests for teachers and on writing research and enquiry summaries. Face-to-face meetings are supplemented by telephone/email contact and use of the VRE.

The questionnaire asked teachers about the types of support they had received from their mentor and also (separately) whether this support was useful. Respondents felt the support they had received from their mentor was useful in helping schools to identify evaluation priorities (91%). They also felt they had received useful advice on issues of research manageability/scope/timelines (88%) and on quantitative (81%) and qualitative (83 %) data collection and analysis (Figure 3 and Table 4).

Figure 2
Q3.1 How confident do you feel in conducting the following evaluation activities in your school?

Figure 3 Was the support you received from your research mentor useful?

Respondents valued guidance on refining the focus of evaluations and help with planning designs that were feasible in the context of the busy work of schools.

We were over ambitious in our original research proposal. Our mentor outlined the possible pitfalls of such an approach and offered suggestions on how to avoid the pitfalls. Principal Teacher.

Helped me to focus my research findings and arrange them in a way that was more relevant to the School of Ambition aims. Principal Teacher.

Focused our planning so we could design clear and manageable research projects. Chartered Teacher.

Table 4

When asked to identify priorities for research support in the next year, the following main areas were identified:

• conducting and analysing data from focus groups (working with qualitative data)
• questionnaire design and analysis of quantitative data (Excel data analysis)
• constructing manageable timelines
• reporting findings, presentation of information

Areas for development

In developing the mentor strategy in the future, the following three areas for development are identified: (1) 16 teachers (53%) reported that they had received no support for critical analysis of literature; (2) 13 teachers (43%) indicated that they had not received advice on using ICT for communication, for example accessing electronic resources; and (3) 10 teachers (33%) reported that they had not received support from their mentor in identifying relevant supporting research materials. All of these issues relate to access to information and resources and signal a need for the Research Support Team to attend more closely to the information needs of the teacher-evaluators within the Schools of Ambition. It is worth noting that, in discussing this finding, members of the Research Support Team believed that they had provided opportunities for consideration of matters such as these and that these questionnaire responses may to some extent reflect the very different stages of research development across the schools.

Data gathering instruments used in the evaluation strands

Although choice of research instrument will reflect the particular purposes and priorities of the evaluation in each school, similar approaches are being adapted to meet specific local needs (Figure 4). The most commonly used research instrument is the survey (paper-based and online). Twenty-three teachers (77%) reported that surveys were among the main methods of data collection used in their schools. Interviews and focus groups were also popular techniques. Schools were making use of the range of information already available to them through school records and were adapting research instruments previously employed for new purposes, for example HMIe questionnaires for pupils and parents. 14 teachers (47%) had been involved in working with external agencies to extend the range of information available to the school. External partners included the Centre for Confidence and Well Being, the Hay Group, Columba 1400 and the Working in Health Access Network ( WHAN). 50% of respondents (n=15) identified a role for parental involvement.

A range of other types of activity were also cited by 6 respondents. These include observation of classes, the use of video to record performances, digital photography and the use of pupil-centred self-assessment. It is encouraging to see a wider of range of approaches to data gathering developing among some of the Schools of Ambition. There is much potential for the development of the use of digital technology for evaluation purposes, although the analysis of visual data for research purposes should not be seen as unproblematic. (The same issues of analysis, transcription, ethics, and appropriate use in workplace settings apply to technological records as apply to other techniques). The mentors have a role to play in encouraging and supporting the responsible and effective use of new technology in school evaluation processes.

The need to develop techniques that will shed light on local and immediate research problems may explain the relatively low priority attached to sources of information that might be seen to have a more general focus; or which were seen as less directly relevant to the particular evaluation needs of the school e.g. reviews of research literature.

Figure 4
The main evaluation/ research/ evidence gathering activities used in Schools of Ambition

Contribution of evaluation activities: Professional development and organisational learning

Teachers were asked about the impact of research engagement (Table 5). A number of respondents felt that a judgement on impact was difficult to ascertain at this time and were reluctant to commit to a positive or negative response. Teachers commented 'we are currently analysing the data', 'we are in the very early stages of the work'. Most evaluation work is currently focused within school (involving pupils and staff) and assessing the impact of Schools of Ambition activities on the wider community was an area where schools were least confident of claiming a positive impact.

Table 9
Have research and evaluation activities of Schools of Ambition work in your school contributed to:

One third of respondents (n=10) reported that the original transformational plan for the school was being modified (to some extent) as a result of findings emerging from school-led evaluation of the first year of the Schools of Ambition programme. Changes included a shift in emphasis, refinements to staff development or changes due to budgetary issues e.g. a bigger focus on pupil empowerment, changes to ICT training as a result of interviews showing disappointing progress after initial training, or aspects of the original plan proving too expensive.

Nine teachers reported that no changes had been made to the original plan and a further 10 were unsure. At this point in the transformation journey, colleagues outside the Leadership Group were unsure whether the original TP had been modified. Where no changes were reported, this did not mean that reflection and modification to the management of the TP had not taken place.

The plan has not changed in any way but the evaluation has led us to give feedback to staff on areas of concern related to the accelerated curriculum changes. Depute headteacher.

It is encouraging that a high proportion of respondents reported that their involvement in the Schools of Ambition programme was developing their capacity to engage in systematic self-evaluation. When asked if their involvement in research would increase their ability to integrate research approaches into everyday practice, 9 teachers (30%) agreed and 16 (53%) felt this was possible with support. There is a role for mentors in continuing to provide support to develop the confidence, as well as the capacity, of teachers to undertake work of this nature. There is also a role for line managers in providing institutional support for teachers to continue to develop and share their research and evaluation skills. Only 4 teachers (13%) suggested that integration of research was 'not very likely', and only one teacher (3%) suggested that future research engagement was 'not at all likely'.

It will increase the range of methods used to evaluate projects and increase the robustness of the responses. By applying these methods to School of Ambition projects it will be then possible to apply them to school policies in the future as staff will be aware of the need for such approaches and be familiar with their use. Evaluation should become less threatening. Depute headteacher.

I am confident in gathering and analysing quantitative data. With the support I have received, I am confident I will also be able to gather and analyse qualitative data. I would then look to use this analysis to inform any decisions I took with regard to developing approaches to learning and teaching. Principal Teacher.

This process has made me think much more about evidence and the need for evidence, especially when it comes to inspections. Principal Teacher.

Working with our link partner from SCRE has made it possible for me to more easily read into the findings of Surveys and so the fear factor of using it in everyday activities is diminishing. Depute headteacher.

It has made me more aware of the need to evaluate my practice and set targets/change my practice accordingly. Principal Teacher.

In identifying factors that support teachers' ability to conduct evaluation work, respondents noted the importance of collegiality and strong channels of communication between promoted and non-promoted staff. The issue of time to meet, plan and act was also frequently cited.

Collegiality. All the staff want to work together for the good of the pupils and the school. Principal teacher.

Supportive senior management, time being given to conduct the research, time to meet with other members of staff. Classteacher.

Regular meetings of leaders of the various SoA strands. Good communication between leaders. Depute headteacher.

Collegiality between staff and support from SMT. Classteacher.

Teachers also emphasised the importance of existing support structures, such as a commitment to systematic self-evaluation and an ethos of enquiry.

Staff are encouraged to be "reflective practitioners". This includes reviewing pupils' progress and their success in achieving the learning outcomes of courses/units. Greater use of assessment is for learning is embedding this reflective process in teachers' everyday work. Principal teacher.

School self-evaluation procedures encourage data gathering. Depute headteacher.

Monitoring and evaluation is a key feature of the work carried out at departmental and whole school level. Depute headteacher.

Factors that inhibited the ability of teachers and other staff to conduct their own evaluation work included time and workload pressures, particularly where curriculum restructuring increased commitments to course development. Of the 24 responses offered to this question, 22 mentioned constraints on time. Lack of confidence and knowledge/expertise were also cited by 5 respondents as inhibitors. Other issues included the suitability of formal evaluations for a school setting and the 'isolation' of classteachers leading to fragmented understanding of whole school development.

Too formalised an approach, making the evaluation fit an academic format which is considered inappropriate for the constant evaluation that occurs in a specialised setting. Detailing all evaluations would be too great a demand for staff. Principal teacher.

Not knowing what is going on at senior level. e.g., classroom teachers often don't have the whole picture of where the school is going, what plans are being put into practice etc. Classteacher.

Sometimes staff working too much in isolation need to join with other staff not necessarily from own school. Depute headteacher.

Despite the barriers identified above, participation in research and evaluation activities appear to be having an impact at both the level of whole school development and improvement planning and also at the level of individual classrooms.

I have already found that I am more likely to ask pupils their views and evaluate new strategies. With experience, my questionnaires are better designed to elicit useful information. Chartered Teacher.

Involvement in the research project is influencing my practice already in small ways. Need to share this with other colleagues and build on the good practice that myself and other colleagues have experienced. Need support of management team, time and resources in order to do this though. Feel more confident because certain aspects of practice are evidence-based. Classteacher.

Over 80% of respondents felt that participation in evaluation activities might help them to integrate research approaches into their day-to-day practice. A smaller proportion saw this activity as contributing to 'professional development'. 70% of respondents (n=21) reported that research activity was having an impact (to some extent) on their professional development.

I am becoming more confident in my understanding of data and how it can inform decisions and development planning both personal and whole school. Depute headteacher.

I will have an objective, analytical and reflective standpoint from which to look at school policy and the activities in a school which result from it. Classteacher.

Has made me aware of need for full evaluation and not just evaluation for evaluation sake. Made me more aware of different forms of evaluation and how best to look at these responses things I was not aware of before. Depute headteacher.

It has allowed me to see the importance of evidence-based evaluations. Principal teacher.

There was also an understanding that the ability to make appropriate use of research evidence was an element of teachers' professional knowledge, not a substitute for professional judgement.

The need for more flexible approach. A greater understanding that highly structured research can demotivate staff having confidence in their own decisions because it does not fit the system being promoted. Principal teacher.

50% of respondents (n=15) expressed some interest in pursuing options for accreditation for work undertaken as part of the Schools of Ambition evaluation. Six teachers are actively considering accreditation (towards Masters awards) and a further 9 would like more information (including 5 of the 6 classteachers). Ten teachers currently involved in evaluation activities were not interested in seeking accreditation; of these, 9 were in promoted posts (2 Headteachers, 5 Depute headteachers, 2 Principal teachers). Four respondents did not consider accreditation an appropriate option as they did not occupy 'teaching' posts (two headteachers and two non-teaching Project Managers).

Impact of Schools of Ambition activities: moving towards 'transformation'

A total of 87% of teachers (n=26) reported that evaluation activities were having a positive impact (to some extent) on the achievement of transformational objectives.

Activities that respondents identified as 'good practice' worth sharing with other schools included:

• The development of social enterprises
• Development of vocational learning programmes
• Development of leadership skills through in-house Ambassadors' programme and Columba 1400
• 'Girls of Ambition' project, nominated for a Scottish Educational Award
• Pupil participation in curricular development
• Pupil participation in the use of interactive whiteboards
• Use of creativity in teaching and learning to raise pupil self-confidence (including Dance)
• Age and stage curriculum projects
• Development of enterprise links
• Health promotion
• Partnership working (with cultural coordinator)
• 1000x1000 exhibition of pupil work from a range of subjects
• Enhanced primary-secondary liaison
• Development of assertive discipline measures
• Staff participation in the planning and delivery of in-house CPD programme
• The development of pupil writing skills across the curriculum

At the end of the questionnaire, teachers were offered the opportunity to add any additional comments. One teacher noted the issue of sustainability when funding ends, another commented on workload pressures. One headteacher indicated some continuing unease with the model of action research promoted as the basis for the School of Ambition evaluation.

We had plans from the start for a big evaluation and evidence providing exercise, perhaps along the IIP model or the format of our Standards & Quality reporting. I believe that the parachuting in of the action research style, especially for a school where there are no teachers undertaking such work as part of for example Chartered Teacher, SQH etc, has been counter productive. It may not be like this for tranche 2 & 3, if they are warned from the very outset that this is the model, or in schools where there is a project leadership model as in [City name] but if we had known, I do wonder if we would have gone ahead with involvement. Headteacher.

Several respondents took this opportunity to reflect more generally on the experience of being a School of Ambition. Their comments indicate that they have found the programme challenging, but worthwhile. They noted how the programme appears to promote a 'way of thinking', a 'change in mindset', a new 'language'. In these accounts the Schools of Ambition programme promotes 'approved risk taking' that supports innovation and the breaking down of 'traditional barriers' to improvement.

'For me, almost every aspect of this programme has been a joy! It is transforming our school and we are learning so much that we can share. SoA is a way of thinking, it is a language we share with other Schools of Ambition ... I even see it in interviewees who come from Schools of Ambition. It has the potential to transform all our schools through the lessons we learn.' Headteacher.

'Our transformational plan is extensive. The outcomes change with the experience gained by staff and pupils. The plan was the beginning of our evolution which will always be ongoing. There is no final objective except to continue to improve the service we offer the pupils. This makes research challenging as the answers we are seeking constantly morph with our experiential approach. This acts as a motivator and provides an element of approved risk taking which has allowed us to be so successful.' Headteacher.

'As Year 2 of SoA ends it has been a really positive, progressive, dynamic exercise for those active participants and hopefully for the school, wider community and international partners.' Depute headteacher.

'I would recommend any school to take part in this opportunity. It has been a huge learning curve, but a journey well worth undertaking. It is an exciting opportunity for any educational establishment.' Headteacher.

'The School for Ambition initiative is an important way to change the mindset and targets for the whole school community. It is giving scope for some of the traditional barriers to be broken down and new ventures to be attempted. The opportunities and development already experienced should facilitate the emergence of ACfE in the coming years.' Principal teacher

« Previous | Contents | Next »