Scottish Compact Baseline Review

CHAPTER FOUR: DEVELOPING A MONITORING AND EVALUATION FRAMEWORK

4.1 The purpose of this Chapter is to look at some of the issues that any Compact monitoring and evaluation exercise has to be aware of. It may be that little can be done about many of these, for example understanding of the Compact amongst those who are theoretically covered by it. However an awareness of them may at least condition expectations of monitoring and evaluation. The starting point is to define these 2 terms.

DEFINING MONITORING AND EVALUATION

4.2 The starting point in developing a monitoring and evaluation framework is to understand what the 2 terms mean. The processes are closely related and are both about assessing the extent to which resources spent on policy implementation result in the goals of this policy being attained. What distinguishes the 2 processes is:-

Monitoring is an ongoing and a systematic activity in which information is collected to enable progress towards attaining policy goals to be assessed. As such, it is undertaken in parallel with policy implementation. An effective monitoring system is one that has some formal mechanisms whereby information is collected that relates to specific aspects of the policy. This information is fed back to the policy makers or implementers, thereby providing them with the opportunity to take action if it appears that policy goals are not being attained. This type of monitoring is also known as formative evaluation, in so far as the information gathered can be used to influence policy implementation. An effective monitoring system therefore includes a feedback loop. The Annual Review of the Compact has the potential to be used as a form of formative evaluation. What currently stops this happening is the lack of any feedback so that the exercise is undertaken as almost an end in itself which is not the purpose of monitoring. What monitoring is concerned with is often, although not always, measuring the inputs to a policy, that is the resources (financial and non-financial) used in an intervention, and the outputs, that is the activities that are undertaken. For example one output from the Compact might be the number of good practice guides produced or the number distributed;
Evaluation, in contrast, is often a backward looking activity. It is undertaken after policy implementation has ceased and policy makers want to see if goals were attained and if there were any other impacts arising from the intervention. This activity is known as summative evaluation and aims to influence future interventions so that policy makers can learn from good (and bad) practices. As monitoring tends to focus upon inputs and outputs, the focus of evaluation is upon outcomes, that is the impacts or the results of the intervention. For example: what affected did the distribution of the good practice guides have? Has there been an improvement in relations between the sector and government? Has there been a growth in mutual confidence and trust? Given that the Scottish Compact is an ongoing activity summative evaluation is not, in isolation, appropriate. Accordingly evaluation of the Compact is likely to cover both the process (that is monitoring) and the outcomes, (summative evaluation). The aims of compact evaluation would therefore be both to influence ongoing activities and to identify what impact these activities have had to date.

Having considered the characteristics of the 2 processes the next issue to be addressed is what exactly is to be monitored and evaluated.

WHAT IS THE COMPACT?

4.3 There would seem to be at least 4 overlapping answers to the question as to what the Compact is:-

The first is that the Compact is a state of mind, a concept, whereby central government and its agencies chooses to work more closely with the voluntary and community sectors developing a relationship based upon mutual trust and confidence;
There is then the Compact document, published in 1998. Clearly this has the disadvantage of being relatively old, pre-dating the Parliament. However it has the big advantage of being a symmetrical document (Paragraph 3.28) in so far as it makes clear that the Compact is a partnership involving obligations, not just on Government and its agencies, but also on the voluntary sector. This is made explicit in that the commitments of both parties are outlined;
There are then the Good Practice Guides. Unlike the Compact document these are asymmetrical, dealing only with the obligations and commitment of government. (Paragraph 3.30). SCVO's guidance is similarly asymmetrical, being addressed solely to the sector. As such there must be questions about the extent to which both sets of guidance reflect the spirit of the Compact. They are essentially a means for ensuring that administrative procedures are transparent. Whilst this may be part of a relationship building initiative, it is only a part and perhaps a relatively small one at that; and
Finally there is documentation covering such things as guidance on grant sources and standardised grant offer, acceptance and payment application forms. This seems to be a consequence of the Good Practice Guidance, with the intention of improving information dissemination and clarity.

4.4 At a strategic level, perhaps when viewed from SCVO or the Voluntary Issues Unit, all these things make up the Compact. However it seems likely that the further away from the centre one is, either in Government or within the voluntary and community sectors, then the more partial one's view of the Compact will be. At the extremes this might take the form of a vague awareness of the ethos of the Compact picked up through conversations or passing references in documentation or a government department making use of the Direct Funding Guide, without being aware, other than in very general terms, of why it has been produced.

4.5 Any monitoring and evaluation framework therefore needs to have a clear understanding of what is to be monitored and evaluated. Given the diversity of the Compact, ranging from the development of mutual support and understanding to conformity to often very detailed guidelines, the implication is that any framework needs to reflect this diversity, albeit that this could be problematical.

4.6 Hems makes the point forcibly that the Compact is about relationship building and therefore the development of confidence and trust between the sector and government. Unfortunately the Scottish Compact has not been disaggregated in any great detail to allow the specific areas around which mutual confidence and trust could be built to be identified. In contrast, the Northern Ireland Compact has been clearly linked into the Executive's wider policy agenda (the "Programme for Government") through the 4 identified aims and the 3 cross-cutting themes (see Paragraphs 3.21-3.23). This should make the process of mainstreaming, which must be the ultimate goal, far easier as Departments (and the sector) can see where the Compact fits into the "bigger picture". This level of detail tends to be less well developed in the Scottish Compact so that it might be quiet easy to see the Compact as almost a parallel initiative, being separate from the Executive's main priorities. This vagueness then tends to be carried through into the Good Practice guides which, although often very prescriptive, have no timescales attached to them and are not prioritised.

4.7 These problems in adequately defining the Compact may mean that the development of mutual trust and understanding is something that is best dealt with through evaluation. However, if monitoring is to focus upon such things as the implementation of the Good Practice Guides then there is a danger that only one element in the development of an improved government-sector relationship (changes by government) is being dealt with. One implication of this is that the existing Compact documentation is deficient as a basis for the development of any monitoring and evaluation framework.

THE SCOPE OF THE COMPACT

4.8 There are then questions to be asked about the scope of the Compact in terms of the departments and organisations that are included within its embrace. The Compact document is explicit about it being about "all Government Departments, agencies and Non-Departmental Public Bodies" (The Scottish Office, 1998, p. 3). It therefore covers bodies such as Communities Scotland, Scottish Enterprise and the Scottish Prison Service. It does not, however, cover the sector's dealings with local government. However this distinction may not always be apparent on the ground. For example Social Inclusion Partnerships would seem to fall within the remit of the Compact. It may be that many voluntary bodies would tend to identify these more closely with local, rather than central, government.

4.9 Yet despite not covering the totality of the government sector the Compact does embrace the whole of the voluntary and community sectors. Even if bodies are not receiving funding from the Executive or its agencies, all are likely to be affected, at some time, by government policy and actions. Even those groups that the Compact accepts will not "have an interest in seeking partnership with Government" (ibid, p. 1) are likely to be affected by the Executive or its agencies at some stage. The Compact is not therefore just about the relationship between government and the various umbrella bodies that exist within the sector. It is all embracing in its coverage of the sector, having an impact even on small local groups whose main interface with government may come, apparently, at the local authority level. One consequence of this is that the sector, or individual elements within it, may be involved in, and affected by, compacts that operate at a variety of levels. Whilst this is not widespread at the moment in Scotland, in England it is possible for a voluntary body to be involved in the National Compact, a local compact and even a sub-regional one, as is increasingly being developed by bodies such as the health service.

4.10 There is therefore again a degree of asymmetry in the scope of the Compact. It is all embracing for the sector, but only partially so for government. Whilst this distinction may be clear to some it may not be obvious to all in the sector nor indeed to all in government. This could influence the responses to any questionnaire or other monitoring methodology.

4.11 The issue of asymmetry also arises within government when the Good Practice guides are considered. They cover all the sectors of government that are within the Compact's scope, regardless of the degree of involvement the various parts have with the sector. However, undoubtedly the intensity of the relationships between the 2 parties varies. For example the Mapping Exercise undertaken by the Executive (see Paragraph 5.19), although not a definitive overview of the links between the Executive and the sector, did show that some voluntary bodies (for example SCVO) had a multiplicity of links with various parts of the Executive, whilst some Departments (for example Health) had far more contact than others (for example Justice). Given this, there may be merit in concentrating monitoring, initially, on those areas of the sector-government interface where the most intensive interaction occurs. Monitoring could then be rolled out to the other parts of the interface at a later date. This follows the advice given in England where it is suggested that priority in Compact implementation be given to those Departments that have greatest contact with the sector (HM Treasury, 2002).

THE SECTOR-GOVERNMENT RELATIONSHIP

4.12 The aim of the Compact is to strengthen and develop the relationship between the sector and government. Unfortunately there is very limited information to describe what the current state of this relationship is. For example, there is no baseline information dealing with the situation at the time the Compact was devised (in 1998). It may be that as 4 years have passed the situation in late 2002 has been affected by the implementation of the Compact. It seems very unlikely that cause and effect could be disentangled in such a scenario.

4.13 What information there is on the relationship is fragmented, partial and/or anecdotal. For example, as Chapter 5 makes clear, there is reasonably good information on the funding relationship. However in other areas, such as consultation, systems are only just being set up to capture data on interactions.

4.14 There is also very limited information on what the key factors are that restrict the growth of trust and confidence. Based on the Annual Review, the following seem to be the main areas of conflict, or concern, identified by government and the sector:-

Decisions not to provide funding to organisations;
The stability of funding;
Inability to make funding decisions within the 3 month notification period;
Problems in getting voluntary organisations to claim grants or the late or incomplete provision of financial returns by them;
Difficulty in meeting the 3 month consultation deadline;
Concerns about the representativeness of the sector;
Issues and problems being brought to the Executive's attention at a late stage; and
The extent to which the roles of similar organisations within the sector differ.

4.15 That government had concerns about the sector over such things as grant administration and representativeness would seem to indicate 2 things:-

Parts of the sector are in breach of the commitments outlined in the Compact document; and
There is a need, in the Good Practice Guides, to define the commitments and responsibilities of the sector as well as government. Although this is done in SCVO's guidance it may be that this lacks the force and status needed to ensure greater conformity by the sector.

This lack of any definitive information on sector-government relations therefore means that there is very limited guidance on the issues that monitoring and evaluation should initially concentrate upon. Clearly the Annual Review seemed to indicate that funding was a key area of concern. How representative this concern is is impossible to know. Concentrating upon finance also runs the risk of over emphasising one facet of the relationship: one that in the long term may prove to be less important than, for example, consultation or policy proofing.

CAUSE AND EFFECT IN COMPACT EVALUATION

4.16 Finally, when evaluating the Compact it needs to be borne in mind that many of the areas where the Compact aims to influence the government-voluntary sector relationship are also being influenced by other parts of the government's policy agenda. For example, consultation is an important area in the Compact and the information to be collected through the Consultation Registration and Evaluation System (currently being set up by the Executive, see Paragraphs 5.8 to 5.14) covers a number of aspects highlighted in the Good Practice Guides (for example allowing adequate time for consultations and providing feedback to consultees). Yet the Compact is not the reason for the System being set up. It is being established to enable the Executive to better monitor and report on its consultation activities. There are also a number of other initiatives that are being implemented by the Executive, and other arms of government, independent of the Compact which will have an impact upon its objectives. For example there is internal Good Practice Guidance on Consultation, which is currently being revised and is due to be relaunched by the Executive by April 2003. As a consequence there are likely to be major problems isolating cause and effect, as there are other initiatives that have aims in common with the Compact. However these problems of attribution are common to most evaluations.

4.17 This problem of attribution also raises a more fundamental question. Does it matter if it is not possible to identify the precise impact of the Compact if a number of complementary initiatives are helping to attain its broad objectives? If the aim of Government is relationship building and the development of greater trust and confidence between it and the voluntary sector then it may be that what is important is simply the growth in trust and confidence rather than attempting to ascribe this to specific actions. Even if this is not the case the setting up of initiatives whose aims overlap with the Compact may mean that attribution, in terms of identifying impacts rather than process outputs, may be very difficult.

WHAT TO MONITOR AND EVALUATE

4.18 Having outlined some of the issues surrounding Compact monitoring and evaluation there is now a need to move forward and begin to make decisions on what should be done.

4.19 The Good Practice Guides are, perhaps, the most well known element of Compact documentation. They contain a total of 96 commitments that government has to meet. It might be possible to monitor progress on each of these, albeit that this would give an unrepresentative picture of the Compact, covering only 1 partner and would also be mixing the mundane with issues that are more fundamental. Ideally what should happen is that the sector would meet with government and identify areas where it is felt the Compact relationship building is being undermined or is poorly developed. These areas could then be incorporated into an action plan and progress in meeting them monitored. This would be a way of prioritising the many dimensions of the Compact and of beginning to move forward on implementation in a systematic way.

4.20 As this has not been done in Scotland, and as the only source of information on the areas of concern comes from the Annual Review which, although a useful document, was not without its flaws (see Paragraph 5.17), developing a monitoring and evaluation framework needs to go back to the basic principles underpinning the Compact and its implementation. Drawing on the earlier Chapters it is possible to identify a number of areas that seem to capture the essence of the Compact and its implementation. These are:-

Awareness of the Compact. The facets of this might include copies of the Compact having been received and staff being aware of the Compact and its objectives. Clearly it is possible for staff to be aware of the Compact, as indeed many no doubt are, but to have a limited understanding of it. Awareness may be seen as the start of the relationship building process, which is consummated when this develops into understanding;
Dissemination is the next facet that needs to be considered. If the Compact is to be mainstreamed then dissemination is critical so that knowledge and understanding permeates into organisations and departments, especially to the staff who have frontline dealings, either with the sector or government. One way of ensuring that this happens is for there to be a champion within organisations who takes on this dissemination and contact role;
Usage is clearly critical. There is little point in people being aware of the Compact if it is not used. The Compact in this instance refers to both the formal document and the Guides (both the Executive's and that produced by SCVO). What would be of interest is knowing if they have influenced the way the sector and government responded to one another and if they are referred to in the course of discussions and negotiations;
Usefulness then needs to be considered. Here the most productive focus is probably upon the Compact document itself and the Guides. Despite the latter's asymmetrical nature they contain a large number of tangible commitments. It would be of value in exploring how useful the partners felt these were. However there is probably little merit in addressing every commitment. Developing a general overview is likely to be more productive;
Regardless of the value of the Guides they are, probably, the most tangible and widely publicised aspect of the Compact. Accordingly each needs to be considered in terms of 2 aspects: whether they have been used; and the impact they have had. Accordingly Consultation, Policy Proofing, Partnership and Funding should each be considered in turn according to these 2 criteria;
The Compact is about developing better relationships between the sector and government. Accordingly Mutual Understanding needs to be considered, to see if the Compact implementation process has resulted in the partners developing a better understanding and awareness of the other;
Confidence is one facet of any relationship. This therefore needs to be assessed. According to Hems (2002) confidence relates to "the day to day functioning of the relationship". It is often governed by some form of contract, for example grant approval, and control processes that place obligations on both parties. It might, therefore, cover the confidence that the sector has in funding decisions being made on time or of the Executive being confident that the sector would put in place appropriate funding control systems; and
Finally Trust needs to be considered. Again, referring to Hems' work this is defined as being "linked to the overall goals of the relationship". As such the growth in trust could be seen as the ultimate goal of the Compact.

Relating these areas back to the earlier questions as to what the Compact is, it is clear that the Compact is being defined as all 4 of the elements identified in Paragraph 4.3, albeit that the final one, support documentation, is not being addressed explicitly.

CONCLUSION

4.21 This Chapter has outlined some of the issues that are likely to impact upon the design, and the interpretation, of the outcomes of any monitoring and evaluation exercise. Some of these may seem to be rather pedantic, reflecting issues of definition. However they are likely to condition responses to monitoring and evaluation. Whilst it may be difficult, in the short term, to take account of these they do need to be borne in mind. The Chapter then went on to synthesise a variety of information in order to identify the key areas that any monitoring and evaluation framework needs to try to assess. Essentially each of the areas highlighted in Paragraph 4.20 is an area for which some monitoring and evaluation indicator needs to be developed. The next Chapter, therefore, looks in more detail at the existing data and information sources that might have potential for providing such indicators.

« Previous | Contents | Next »

The Scottish Government

Research

On this page: