There are many different ways of doing action research evaluations. This article outlines how action research evaluations will be used by projects in the Stronger Families Fund. It is intended to serve as an introduction for Stronger Families Fund projects, but also to be of interest to others working in the field of early intervention and prevention.

Action research is both an approach to research and a move towards change or continuous improvement in organisations. It can also be used as an evaluation method.

In the Stronger Families Fund, the focus is on using action research for project evaluation, to help projects with continuous improvement. At the same time, it is also about gathering and sharing the information about what works and doesn't work for early intervention and prevention projects so others can learn from it.

There are many different ways of doing action research evaluations. This article outlines how action research evaluations will be used by projects in the Stronger Families Fund. It is intended to serve as an introduction for Stronger Families Fund (SFF) projects, but also to be of interest to others working in the field of early intervention and prevention.

Background

Evaluating projects is a useful way to find out whether they are working or not. It is an important component of most community programs and projects. It may be a new activity for some groups starting out on their first project, or it may be a familiar concept for those who have more experience. Wherever project teams are coming from, it is important to note that the components of an action research evaluation might not necessarily be completely new for a program or project team.

In many cases what happens in an action research evaluation builds on what already happens in projects and uses people's existing skills, for example local knowledge, noticing, describing, listening, planning, involving participants or clients in activities or services, responding to local needs and circumstances, explaining, making decisions, networking with community services and other organisations, documenting, reporting and improving as you go along.

In these cases, doing an action research evaluation gives a focus to the things people would be doing anyway (Crane and Richardson 2000). It also gives permission for the project team to take time out from their work to reflect and write down their observations. This can help busy project teams to get back in touch with the bigger picture of what they are trying to do and why. It can help keep projects alive and vibrant, especially when teams come together to share what is important to them, their observations, insights, values and passions.

Key features of an action research evaluation

Action research can best be explained by looking at the two words in its title: it is about action or change, and it is about research, a word that means 'to find out'. Put simply, it is about research that informs action or change in a project or program. The two go together. When used as an evaluation method, it encourages people to look at and think about how projects are going as they going along, instead of just evaluating them at the end.

Purpose of an action research evaluation

Action research evaluations are most often used because of their focus on action or change, or continuous improvement.

Building action research into a project is useful because it can help the project, the organisation and in some cases, whole communities to be more responsive to feedback and insights gained, and to change as they go along. Doing action research gives project teams the opportunity to look at the meaning and context of their work, document their strategies, then to test and refine them over time. It allows and requires project teams to build records of their development and to justify their practices. Projects have much more scope to be flexible over the funding period, because of the scope to implement changes quickly based on a reasonable research base.

Additionally, when the research findings are reported outside the project, they can help build knowledge about what works and what doesn't work in particular places or contexts, which is useful for other projects to learn from. Research reports can also inform Government policy and other research.

Who is involved in an action research evaluation?

An action research evaluation usually involves everybody who has a stake (it is participatory and collaborative). Previous studies have found that when a participatory action research evaluation is part of a project, it can help projects to be more sensitive to the variety of needs of diverse communities. The ongoing evaluation can inform activities and programs so they are more likely to work better in particular locations and with particular groups of people. It is an approach that helps build partnerships between stakeholders, and bring in others, such as local businesses that can make a contribution.

Involving everybody who has a stake, in a spirit of partnership, can help people have a sense of ownership of a project. Participation helps to create change because, 'among other reasons, change is usually easier to achieve when those affected by the change are involved' (Dick 1999).

In addition, people affected by projects often have the best insights into their situation. Anne Garrow (2001) suggests project teams need to affirm and acknowledge participants as experts in their own lives, support community control and participation, and stand alongside participants rather than taking a one-up position.

Involving everybody - asking them for their ideas and suggestions about what is happening, what should happen and what it means - not only generates more valid and reliable information, but it can create connections between people and groups, and can empower participants and communities and even workers to act to change their situations. Projects that use this approach have been shown to work better and have greater long-term effects, sometimes carrying on in some way after the official project finishes (Stayner et al. 2000). In other words, doing an action research evaluation, and projects acting on results as they go along, helps promote project sustainability.

Consequently, it is seen that SFF project teams will greatly benefit if they can find ways to involve and value the contributions of participants and their families, workers, local agency managements, community members, local services, government and non-governmental organisations.

How to involve people in an SFF project: Setting up a reference group

If the project does not already have one, setting up a reference group or groups is a great opportunity to involve participants and/or clients in both the project and the evaluation to see how the project is going.

A reference group may have a different role in different communities. At least half, or in some cases, all of those in the group should be those who the project is trying to serve - the people 'who-it's-all-for' (Wadsworth 1997a) to make sure they have their say into both the content and the process of the evaluation. Other participants can be people from local health and welfare agencies, community groups, community leaders and community members. It can start small and grow as the project progresses.

This group may need to be flexible and informal and fit in with local ways of doing things. It may need to take account of people's busy lives, and in some cases, project teams may need to offer transport assistance or pay people, especially unwaged people, for their time. Gaining trust and credibility, especially with some participants and clients, or other agencies the project team may have competed against for funding, may take time.

In some communities and projects, groups may include opposing stakeholders, and there are opportunities for constructive conflict to lead to positive solutions. In others, there may need to be more than one group to take account of conflicting views or to look at different activities. Being able to discuss inevitable differences and conflicts can make the evaluation results richer and more well-rounded. In fact, if everyone in the group agrees on everything, it is a good idea to seek out opposing views to ensure your evaluation embraces diversity.

People invited by project teams to join reference groups might ask: 'What's in it for me?' Phil Crane and Leanne Richardson (2000) offer some possible answers based on another government project that uses participatory action research as an evaluation method, the Reconnect early intervention program on youth homelessness. They suggest project teams can offer participants and clients the opportunity to contribute to project development, a say in what the issues are and an opportunity to talk about their own experiences. They can offer members of the local service network a chance to have input into the exploration of local solutions to local problems. Being involved can establish and enhance relationships and allow for more opportunity for collaborative strategies within the community. Community involvement allows communities a way to develop responses to issues and allows members to understand more about the problems some people face. It ensures the approach suits local circumstances and ensures community 'ownership' and involvement in developing strategies.

Apart from members of the reference group, there are other people who may be sources of help in the evaluation process. These might include individual people who have a story to tell, and co-researchers or peer researchers - people project teams employ to find out the views of specific communities they want to involve and work with, for example, local indigenous people or people from culturally and linguistically diverse backgrounds.

What is the action research cycle?

An action research evaluation is often talked about as a dynamic process: cycles of planning, acting, observing and reflecting, then planning again for a new action (Kemmis and McTaggart 1988; Wadsworth 1997a). The accompanying figure gives a picture of how these cycles might happen.

These cycles work best if they are tied to natural project cycles, but as a guide, they might go around every six months or so. Sometimes the stages may overlap or happen in a slightly different order, but an indication of how a cycle might evolve in a project can be gained from the following overview.

Plan

• The project team and reference group members come together to talk about evaluating a project, and discuss why the evaluation is important.
• The group gathers baseline data (information about the situation the project wants to improve).
• The group defines an idea or assumption they want to test, or an issue or problem or question they want to find out more about that will help the project to improve, starting with small, specific questions and building up to bigger ones as confidence increases. A simple way to get started could be to ask 'what could we do better?'
• The group decides how they want to carry out the evaluation and the level of involvement of reference group members.

Act

The group starts the first planned activity or program of activities.

Observe

The group looks at, listens to and thinks about what is going on, asks questions of all those involved about how it is going, and gathers this information, focusing on the questions raised in the planning stages but also being open to other feedback.

Reflect

The group thinks about what the information means for what they are doing (or not doing), and whether what they are doing can be improved. This includes thinking about whether and how the results help in understanding issues and problems (a critical analysis of the situation). The information and knowledge gained is recorded.

Then the cycle begins again, with more planning, this time having a fresh look at the project and taking account of what has been learned to see if any changes might be made. At this stage, the group also plans the next cycle of evaluation, either to deepen understanding of issues from the last cycle, or to research a new idea, assumption, issue, problem or question. This is followed by more action, observation and reflection, and so on.

As the process becomes more familiar and comfortable, bigger questions can be asked and more people are likely to become involved. As the project progresses, as well as at the end of the project, the group can revisit the baseline data and assess how the situation has improved.

A useful variation on the above (plan, act, observe, reflect) model has been developed by Tjikalyi Colin and Anne Garrow in the Indigenous community of Ernabella in South Australia. Best described by the title of their book Thinking, Listening, Looking, Understanding and Acting as You Go Along (1996), their evaluation map (reproduced here) provides another way of thinking about the practice of an action research evaluation.

Research methods: How can evidence be gathered?

For groups that have done program evaluations before, an action research evaluation can be thought of predominantly as a process evaluation (that is, determining the extent to which a program is operating as intended and target populations are being served). However, an action research evaluation is also likely to incorporate an assessment of the impact of the programs on participants (an outcome evaluation).

What is different about an action research evaluation, however, is that it happens in cycles, it involves participants, the methods are varied to suit the program and the results are used to improve the program as it goes along. Typically, an action research evaluation is also mainly qualitative, which means the focus is on gathering stories rather than statistics (which is not to say that statistics, or quantitative data, cannot be gathered).

Quality information (data) can be produced by using a range of different ways of gathering this information (research methods). This is known as a 'multiple methods' or 'triangulation' approach. Information gathered from a range of participants (for example children, their parents, grandparents and community members) and using a range of methods gives groups a better chance to develop a good picture of what is happening. Information gathered can range from individual stories to statistical data on any or all aspects of the project.

In broad terms, some of the ways of gathering this information could include:

• descriptions of programs and services: this can include documentation of the process of developing the project, records and reflections from project workers, other staff , volunteers and management about what project teams are doing, and why and how they are doing it, using for example meeting minutes and a journal or log book to keep track of insights, observations, anecdotes and questions, and reflections on the research process itself;
• photographic and video documentation;
• paintings, drawings, music and songs;
• counting the numbers of people who participated in activities/programs, and information about those people (for example who they are, where they come from, why they came, how many sessions they attended);
• participant information, referral sheets, work log books and other agency paperwork;
• participant feedback sheets or documentation of verbal feedback;
• group brainstorming, talking circles or discussion groups with participants and/or other stakeholders (for example, other services in the area, local community groups);
• suggestion boxes, comments books, email and/or websites where people can leave comments and ask questions;
• larger forums or conferences involving a range of stakeholders;
• in-depth individual, family or group interviews, using open-ended questions which allow diverse experiences and perspectives to emerge;
• statistics and surveys;
• naturalistic observation;
• case studies; and
• comparing participants' knowledge/skills/ views/behaviour before participation in a program and after participation to see if it made a difference (the traditional evaluation method of pre-test post-test analysis, but adapted to suit the project); a wide variety of methods could be used for this.

The ways of finding out information and gathering evidence can be diverse. Any and every method can be used depending on what needs to be found out and the people involved. For more information about research methods, see Yoland Wadsworth's (1997b) excellent introduction for community groups, Do It Yourself Social Research.

An example of gathering evidence: Baseline data

A project group's first gathering of evidence in their evaluation work will be gathering baseline data. This activity is undertaken because is important to have a clear idea of why the project is significant. This may already have been written into the grant application or project workplan. In most cases, project teams and reference groups need to start with gathering more descriptions, problems, questions, statistics, previous research and community-identified priorities that help paint a picture of the situation that the project grant wants to improve. This information is called baseline data.

The information will be different in every community and is most useful if it focuses on things that might change as a result of the project. Local government statistics, available from local councils, may be useful if the project falls into a local government area. They may not be so useful if the project crosses these areas, or takes place in a specific community where local issues are 'swallowed up' by statistics for a larger area. In these cases very specific local information is helpful, if it is available. As the project progresses, it may be possible to look back at this information and assess how the situation has changed or improved.

Research ethics: Protecting the privacy of the people who tell their stories

Informed consent

All of the people who might have a story to tell in an interview or discussion group need to have the research project explained to them and be invited to join in. They should have their rights explained and be told that if they do not want to join in they will not be disadvantaged, and that they can change their mind at any time and their story will not be used. Potential participants should sign an informed consent form saying that they understand the project and their rights. This form should be kept separately from their story in a locked filing cabinet.

Privacy and confidentiality

If people give details about themselves, these details should be confidential (kept in a safe place where no one except the project worker can have access to them). No information about individuals should be used that could identify them: summaries of information only should be used, or stories and case notes should have some details changed so that no one will know who the story is about.

In some communities, doing ethical research might also include:

• involving participants in discussions about what is going to be researched, endorsing it and controlling the analysis and distribution of findings, usually using a reference group or community meetings (Colin and Garrow 1996);
• making sure research takes place in community languages and with appropriate interpretation or translation of the findings; and
• finding a way to feed back research results to the research participants. This can be done either during the research process to understand what has been said (which gives a good opportunity to deepen understanding by stimulating further dialogue), or by offering a summary of the final results.

What are the challenges or potential problems?

The cyclical approach of an action research evaluation gives groups the chance to progressively look at and think about whether the project is doing what it set out to do and if there is anything that could be done better, and to report on this regularly enough to justify changes if they are needed. If the organisation is already geared to continuous improvement, incorporating the cycles of an action research evaluation and writing up what happens may only be a small change to the way things are done. However if it is a new approach, it may take longer to get used to. Because an action research evaluation is participatory and happens in repeated cycles while the project is happening, it often takes longer than other kinds of evaluations.

In some communities, the hardest and longest part may be getting people to join in, especially when people are busy, are geographically separated, from different cultural backgrounds or may not have ever been asked their opinions before and do not trust who is asking. In other communities, getting people involved may be easy but the methods chosen, for example interviews, may mean there is a lot of information to take in and analyse.

Whatever the challenges in particular situations, the Stronger Families Learning Exchange can provide support to projects that will help them address these issues and gain maximum value from their action research evaluations.

References

Colin, Tjikalyi and Garrow, Anne (1996), Thinking, Listening, Looking, Understanding and Acting as You Go Along: Steps to Evaluating Indigenous Health Promotion Projects, Council of Remote Area Nurses (CRANA), Alice Springs.

Crane, Phil and Richardson, Leanne (2000), Reconnect Action Research Kit, Department of Family and Community Services, Canberra.

Dick, Bob (1999), What is action research?, Online http://www.scu.edu.au/schools/gcm/ar/whatisar.html

Garrow, Anne (2001), Participatory learning and action with drug using populations: A review of the PLA component of the Injecting Drug Use Prevention Education Project, Myanmar, Report commissioned by World Concern.

Kemmis, Stephen and McTaggart, Robin (1988), The Action Research Planner (3rd edn), Deakin University, Geelong.

Stayner, Richard, Foskey, Ros and Ramasubramarian, Laxmi (2000), The Continuing Effects of Action Research Projects, Report to the Department of Family and Community Services, Canberra.

Wadsworth, Yoland (1997a), Everyday Evaluation on the Run (2nd edn), Allen and Unwin/Action Research Issues Association, Sydney.

Wadsworth, Yoland (1997b), Do It Yourself Social Research (2nd edn), Allen and Unwin/Action Research Issues Association, Sydney.

Tania Lienert is a Senior Research Officer in the Stronger Families Learning Exchange at the Australian Institute of Family Studies.

Home

Stronger Families Learning Exchange, c/- Australian Institute of Family Studies, 300 Queen Street, Melbourne Vic 3000. Tel: (03) 9214 7888. Fax: (03) 9214 7839. Email: strongerfamilies@aifs.gov.au.

AIFS