Annette Cockburn and Chris Giles
The Homestead is an organisation which aims to provide care and rehabilitation to Street Children. We run an intake shelter, a second stage home and an alternative education programme. The education programme is not included in this evaluation.
It is perhaps axiomatic to point out that the process of evaluating a service is both frustrating and exciting, and far more time-consuming than any of us expected. The enormous challenge was our determination to submit our practice to empirical measurement, to emerge with facts and figures. It seems to us that evaluation exercises were often extremely nebulous, vague and unspecific for example, “20 boys attend the life skills workshop which has been running well.”
This was the first formal evaluation to be undertaken by the Homestead since its inception in 1982. Obviously we are evaluating all the time, asking a number of questions about our practice: Is this policy working? Is this child more settled? How is our food programme running? ... and many other such issues from the agendas of weekly staff meetings in all three centres. Over and above this ongoing level of discussion, this formal evaluation provided us with the opportunity to:
1. Step back and look at the organisation as a whole.
2. Do it as a joint staff
3. Quantify some information
4. Set Standards
5. Provide a data base for future evaluations.
6. Become more aware of the “gaps” and “strengths” in our practice.
7. Identify key tasks and see how these fit with existing job descriptions. We have emerged with no clear direction on some “macro” issues e.g. should Patrick’s House move, or does the second stage model really work?
Design and structure of the evaluation
This evaluation was designed within the framework of action research. It was intended that the process of the evaluation be an affirmative, enjoyable and empowering experience for all those involved. The evaluation was based ultimately on factual observation and measurement, which the staff themselves determined and carried out. These were clearly linked to the key tasks of each member of staff. These key tasks in turn flowed from the negotiated objectives of the organisation's programmes, goals and mission statement. In a sense what we embarked upon was far more than evaluation. We had no clear idea of who we were, or where we were really going, what we were engaged in was paradigm-building, at least initially, but until we had done that, we could not evaluate.
The structure took the form of five workshops. Staff were informed that participation was mandatory. It was decided that management committee members and boys would also take part in the evaluation. We identified the following areas for investigation:
1. Child Care
2. Social Work
4. Management Committee
5. Boys' responses to aspects of the organisation
Some cautionary words
Firstly, the time this whole exercise took was quite considerable, but obviously this will be much reduced in future years since much of the groundwork has now been done: enabling functions have been negotiated, accepted and written indicators have been established, standards negotiated, etc.
Secondly, there has been little time for staff to study the finished product, nor as yet have many of the recommendations been implemented, but within these constraints the evaluation process was in many ways, exciting, engaging revealing and reassuring. We always knew we had wings and now we know we have roots as well.
The flow of the evaluation
We have broken the process of evaluation into ten stages in order to make it easier to understand. Each stage is more specific that the preceding one, so that in the end we are dealing with the directly observable, with behaviour. These stages fall into two groups “those which enable the evaluation those which execute the evaluation. Each stage will be explained briefly.
Enabling The Evaluation
A mission statement sums up in one sentence what your organisation wants to achieve. It states the most basic values of your organisation, and all actions performed in the name of your organisation must be consistent with your mission statement. Your evaluation must also give expression to your mission statement: it must help you on your way. Example: We believe that at the Homestead we should provide child-centred and unconditionally accepting care to street children, within the framework of participatory management.
Your list of goals identifies the main aspects of what you want to achieve. It breaks your mission statement into the main aspects of service you want to provide. You have moved another step towards defining exactly what it is that you want to do. Example: Our goal is to provide a good, well run service, developing socially acceptable people who can earn their own living, get back into their family or communities, and care about themselves and others. More specifically, our goals of child care are to provide for the basic needs of the child and to provide an environment in which he can learn life skills, and therefore look forward to a brighter future.
Each goal can be broken down further into objectives. An objective is a statement which tells you how the goal will be reached. Again you are a step nearer knowing what you should do in your daily practice, and the link with your starting point, your mission statement, is still preserved.
It is at this point that you must check that all the objectives of the organisation are accommodated somewhere in the job descriptions of some member of the organisation. You can also check that nobody has tasks in their job decription which are not necessary, or which even make the fulfilment of the mission statement more difficult. Example: The child care objectives are to maintain good standards in cleanliness, nutrition, health, clothing, education, daily routine, religious teaching, counselling, discipline, and rapport with the children.
This step, and the one following, are intended to make the evaluation itself easy, clear, and satisfying. You can’t measure everything or record everything that anyone does in your organisation. Also, some aspects of your service are not obviously quantifiable - how do you measure “close relationship” or “intense counselling?”
For these reasons you must choose a few observable and quantifiable objects or behaviours which all involved agree can stand for, or represent the complex intangible world of child care. These indicators are a sample, a manageable number of observations from all the millions you could make. They may also be concrete representatives of abstract processes.
It is very important that these indicators are agreed on by all the staff involved, because otherwise the measurements made will not be accepted as valid and the whole evaluation will be pointless. Example: The indicators for cleanliness were that all rubbish was in bags, that all drains were cleaned, the number of boys who washed each day, how many had lice, and the number of boys who had a toothbrush of their own.
Once the indicators have been agreed on, the next step is to agree the standards to be expected. The principle here is that the people doing the work which is being evaluated, should negotiate with their superiors what will count as a “pass mark”. The pass mark must be set at a level which allows excellence of work to be noted, but it must not be set at a level which will mean that many people “fail”. This “pass mark” must be decided and accepted before any measurements are made. Example: The standard for boys washing was 60% “this means that after discussion it was agreed that this proportion of children should have a shower each day. It was expected that 50% of children at the intake centre would have lice.
Summary of enabling stages
Up to this point the work has been to create a framework which expresses what the organisation and all its employees are trying to achieve, how this can be stated in writing, and how it may be measured. It also set negotiated standards against which any employee can find out where they are working well, and where they need to make changes. It is not necessary to repeat these stages with successive evaluations, so that the second time you evaluate it is not such hard work. Each evaluation must however begin by making sure that the enabling framework (the mission statement, goals, objectives, indicators and standards) are still valid and appropriate. It is always possible at the time of your annual review to make any needed changes.
Executing The Evaluation
There are four stages in this part of the evaluation process. Once again, each will be described in turn.
Once the indicators have been agreed on, the task of doing the actual measurements must be shared by all those whose work is being evaluated. What the measurement actually involves will depend on the relevant indicator. It may mean making some observations, it may mean counting some objects, or it may mean asking structured questions of colleagues. Example: In order to establish how many boys have a shower or the percentage who have lice, it is necessary for staff to decide who will make the necessary obversations and on which days.
Each staff member is responsible for taking, writing up, and handing in the measurements allocated to them. It is then the responsibility of senior staff to integrate all the measurements and come up with preliminary findings and recommendations for discussion at a full staff meeting. It is at this stage that the importance of having set standards is felt. There are no ugly surprises, and the majority of findings will be positive. Example: It was found that 25% of children at the intake centre had lice infestations. It was also found that fewer children (12%) had lice at the second stage centre.
The findings need to be carefully considered. Achievable recommendations need to be formulated. It is at this stage that the best overview of the organisation can be built up, and opportunities for improvements clarified. It may even be useful here to get in somebody who is skilled at interpreting information or who is familiar with other child care institutions and how they work. The recommendations should be made for each area of service and for each level of worker. Example: It was recommended that special shampoo be bought and used regularly. The finding that more settled boys were less likely to have lice is some indication that they are acquiring self care skills. Finally there was comment from the principal that the initial staff standard, that a 50% infestation rate is acceptable, is too high, and that in future a higher standard should be set.
The results of the evaluation exercise should be written up and a report produced. This work is well worth the effort as it ensures that the final product has internal consistency, and that it is available not only for the next year’s evaluation, but also for the information of other similar organisations. It has one final function: it serves as a testament to the competence of the child care workers involved. The report may be obtained on request from the Principal.
It is distressingly common to find child care institutions which operate year after year without systematic evaluations of the work they do. The reason this is distressing is that without evaluations there is no guarantee that a consistent and appropriate service is being offered. There is no check that all staff are working to the same end, and there is no check that the institution is having the effect on the children that staff would like it to have.
People who would drive a car with their eyes closed would be considered menaces. It is no less irresponsible to “drive” a child care institution with your eyes closed. And opening them can be such fun.
This feature: Cockburn, A & Giles, C. Evaluating and Child Care Programme. In Gannon, B., & Biderman-Pam, M. (1989). (eds.) Competent Care Competent Kids, Cape Town: NACCW. pp.246-252