- Home |
- About Us |
- Contact Us |
- Site Map |
Compiled by Nick Richardson, National Child Protection Clearinghouse.
Published by the Australian Institute of Family Studies
ISSN 1448-9112 (Online)
You can access this resource sheet in HTML or PDF (222K) format. You will need an Acrobat Reader which is free from the Adobe Systems Web site .
A child abuse prevention program is a structured activity, practice and/or strategy that is designed to reduce or manage the causes of and therefore reduce the occurrence of child abuse. Programs are generally services operated and funded by government and non-government welfare organisations. For example, parent education programs are intended to provide information awareness and skills to the participants on aspects of how to best raise children. Personal safety programs are designed to educate children to identify, and therefore protect themselves in, situations leading to possible child maltreatment.
Program evaluations aim to determine the effectiveness of a program (in other words, has the program achieved what was intended?). It involves the collection and analysis of information or data. Evaluation includes an analysis of how services are used, the profiling of service users, the ability to inform ongoing improvement and refinement of program content, as well as the provision of a measure of overall program success for funding bodies and stakeholders (Schalock & Thornton, 1988).
There are many types of program evaluation. Evaluations can be classified as a process, impact, or outcome evaluation; however it should be noted that different types of evaluation are intrinsically linked.
Process evaluations are at aimed at enhancing the understanding of a program. Process evaluations measure what is done by the program, and for whom these services are provided. Ideally, process evaluations assist in the identification of 'active ingredients' of treatment (Pietrzak, Ramler, Renner, Ford, & Gilbert, 1990). The aims are to determine the extent to which a program is operating as intended, and to identify areas for change that can enhance service delivery.
Usually process evaluation involves the description of a program's operation, including the staffing, persons served, the services provided and the costs involved (Schalock & Thornton, 1988). These are measured against the community's needs, service-provider goals, best-practice standards and cost effectiveness (Pietrzak et al.,1990). Key questions may include:
It is considered that process evaluation provide important information for further impact and outcome evaluation.
The most common form of program evaluation assesses the 'impacts of the program' or the extent to which a program or intervention affects the characteristics of participants or causes of child abuse (US Department of Health and Human Services, 1995). For example, an outcome of a parent education program could be attitudes towards and the use of physical punishment. Any change that is found is assumed to result from participation in the program.
Outcome evaluation assesses program efficacy or effectiveness in producing long-term changes among a defined population. Commonly, outcome studies focus on the occurrence or prevalence of child maltreatment. Outcome evaluation is also known as follow-up.
It is recommended that outcomes be monitored on a regular basis to identify trends in the success of one or more programs
Non-experimental studies use information on the characteristics, experiences and presenting problems of the people who have used the program. They usually compare information collected before and after participation in the program as a measure of the change that the program may have produced (Chalk & King, 1998). This is called a pre/post-test design. Information can be anecdotal reports, or client feedback and satisfaction measures.
Such studies may produce useful information for service providers (or researchers developing research) such as a profile of the characteristics of those who used the program, or enable the identification of program implementation or process issues.
However, non-experimental studies are considered to be insufficient to accurately determine causation or program impact. This is because it is difficult to tell if a 'confounding' variable (i.e., something else other than the treatment) is influencing the result. For example: if a group of participants who did not significantly improve also all lived in the same street, while everyone who improved lived on the other side of town, their worse outcome might be the result of socio-economic or environmental factors, rather than the failure of the treatment program.
Quasi-experimental research has been considered as a minimum form for assessing program impact.
Such research typically involves at least two groups of participants:
At the end of the treatment, a comparison is made between the treatment and the non-treatment groups. Any differences between the groups are assumed to provide evidence that participation in the program resulted in particular outcomes.
Studies with a comparison group are generally perceived as more reliable than a simple pre/post-test design (Chalk & King, 1998; Fink & McCloskey, 1990). However, because individuals are not assigned randomly to the two groups in quasi-experimental studies, there can be differences between the comparison and experiment groups before treatment that were not apparent to the researchers. For instance, the 'no treatment' group may have had more difficulties than the treatment group. This reduces the certainty that researchers can conclude that the outcomes were due to these initial differences between the groups or to the program.
The highest level of evidence of program effectiveness comes from experimental designs, because such designs have the strongest degree of control over confounding variables (Chalk & King, 1998; Fink & McCloskey, 1990; Nixon, 1997).
Unlike quasi-experimental studies, experimental studies involve the random allocation of participants to either an experimental group or a control group. This provides the best chance of determining the effect of participation in a program. Such designs are often referred to as scientifically or methodologically rigorous, or as 'true experiment' (Chalk & King, 1998; Fink & McCloskey, 1990).
While there have been a large number of program evaluations of child abuse prevention programs, very few rigorous evaluations have been done in Australia or internationally (Melton & Flood, 1994; Tomison 1998).
The majority of current evaluations are non-experimental and thus provide no firm basis for determining the effect of a program (Chalk & King, 1998). Similarly, little is known about which programs are most effective for different types of child maltreatment (i.e., physical abuse, psychological abuse, sexual abuse, neglect) because most studies do not adequately document the type(s) of maltreatment that the program aimed to address or that program participants experienced or perpetrated. It is important to note that any type of maltreatment is generally not experienced in isolation (Higgins & McCabe, 2000).
In their review of child abuse prevention programs in the UK, MacDonald and Roberts (1995) commented that the vast majority of interventions had not been evaluated prior to introduction, and those that had were usually employed non-experimental methodology.
Fink and McCloskey (1990) reviewed 13 US program evaluations recommended by experts and published from 1978 to 1988. They reported that most of the studies were methodologically sound in using control groups. However, they concluded that it was not possible to determine whether specific aspects of family functioning had improved as a result of participation because the studies:
An exception to the rule in terms of child abuse prevention programs, is home-visiting services, which have been found to be very effective in detecting and identifying maltreating families and/or alleviating maltreatment concerns. The most scientifically rigorous outcome evaluation of a comprehensive home visiting program, and arguably one of the most rigorous evaluations of a child abuse prevention program, is the Prenatal/Early Infancy Project developed by David Olds and colleagues (Olds et al., 1997). For more detail on this program, see Tomison & Wise (1999).
Little systematic research has preceded the implementation of prevention programs in Australia.
Often overseas programs are adapted for use without any investigation of the needs of the community for which the program was intended (Vimpani, Frederico, Barclay, & Davis, 1996).
Given the identified limitations of international program evaluations and knowledge of program effectiveness, the Australian reliance on international evaluation studies and the efficacy of international programs may be misplaced.
An early audit of the National Child Protection Clearinghouse's database of prevention programs and research showed that little attention has been paid to the evaluation of family support/ parenting programs in Australia (James, 1994).
In a subsequent audit of the NCPC's Prevention Programs database however it was apparent that service providers had become more aware of the need to implement program evaluations with some degree of methodological rigour (Tomison & Poole, 2000). Although the majority were still non-experimental designs, a sizeable proportion of programs were reported to incorporate a pre-test/post-test evaluation design, is a positive development.
Some agencies fear that an evaluation, particularly an external evaluation, may produce negative results or indicate that their program is not successful.
Some service providers perceive program evaluation as diverting badly needed resources away from service provision, which could potentially harm participants and place an additional work on frequently overloaded staff. The potential complexity and technical nature of evaluations, combined with a lack of in-house expertise and knowledge as to who can provide assistance may often lead service providers to conduct only a limited evaluation, or to fail to conduct any evaluation (US Department of Health and Human Services, 1995).
Random assignment of participants to experimental and control groups raises ethical issues in the child abuse prevention area as this creates the potential for participants who may be in serious need of intervention to be allocated to the 'no treatment' control group. Thus, the use of waiting list participants as a matched comparison group in a quasi-experimental study is generally seen as preferable by service providers (Farmer, Burns, Guiles, Behar, & Gerber, 1997).
Given the limitations of current child abuse prevention program evaluations, evidence-based practice may offer a means of establishing a reasonable body of research upon which to base a prevention strategy (MacDonald & Roberts, 1995).
Under an evidenced-based approach, the object is to identify all studies, published or not, including those studies which produced negative effects or no change to participants. Information is also generated from routine practice, and the widest range of sources available (Clark, 1997). Weaknesses in methodological rigour are therefore compensated for, in part, by the richness and quantity of the data gathered.
Action research can be described as a collaborative venture between an evaluator and service providers where the intention is to increase the integration between research and practice or program development. Action research involves a continuous cycle of reflection and monitoring to better understand things that affect practice, and to use this understanding to change aspects of practice (Crane & Richardson, 2000).
Although program evaluation may seem to be a complex and difficult task, a number of recommendations have been forward for individuals and organisations to assist them in preparing of an evaluation study (Chalk & King 1998; McCurdy 1995; Schalock & Thornton 1988; US Department of Health and Human Services 1995). These include:
For more detailed information about the points outlined above, see:
Tomison, A. M. (2000). Evaluating child abuse prevention programs. Issues in Child Abuse Prevention, 12, Melbourne: Australian Institute of Family Studies. Available at: www.aifs.gov.au/nch/issues12.html
Chalk, R., & King, P. A. (Eds.). (1998). Violence in families: Assessing prevention and treatment programs. Washington DC: National Academy Press.
Clark, R. (1997, March). A research agenda - what does it mean? Paper presented to Research Agenda Workshop, Youth & Family Services Division, Department of Human Services Victoria, Melbourne.
Crane, P., & Richardson, L. (2000). Reconnect action research kit. Canberra: Commonwealth Department of Family and Community Services. Available at: www.facs.gov.au/internet/facsinternet.nsf/aboutfacs/programs/youthreconnect_ action_research_kit.htm
Farmer, E. M. Z., Burns, B. J., Guiles, H. B., Behar, L., & Gerber, D. I. (1997). 'Conducting randomized clinical trials in children's mental health: Experiences and lessons from one venture'. In C. T. Nixon & D. A. Northrup (Vol. Eds.). Children's mental health: Vol, 3. Evaluating mental health services: How do programs for children 'work' in the real world? (pp. 203-230). Thousand Oaks: Sage.
Fink, A., & McCloskey, L. (1990). 'Moving child abuse and neglect prevention programs forward: Improving program evaluations'. Child Abuse & Neglect, 14, 187-206.
Higgins, D. J. & McCabe, M. P. (2000). 'Multi-type maltreatment and the long-term adjustment of adults'. Child Abuse Review, 9, 6-18.
Huberman, A. M., & Miles, M. B. (1994). 'Data management and analysis methods'. In N. K. Denzin & Y. S. Lincoln (Eds.). Handbook of qualitative research. Sage, Thousand Oaks: Sage.
James, M. (1994). 'Analysis of the programs database'. Child Abuse Prevention, 2, 1-3.
MacDonald, G., & Roberts, H. (1995). What works in the early years? Effective interventions for children and their families in health, social welfare, education and child protection. Essex: Barnardos.
Melton, G. B., & Flood, M. F. (1994). 'Research policy and child maltreatment: Developing the scientific foundation for effective protection of children', Child Abuse & Neglect, 18, 1-28.
Nixon, C. T. (1997). 'How does evaluation of mental health services for children work in the real word?' In C. T. Nixon & D. A. Northrup (Vol. Eds.). Children's mental health series: Vol, 3. Evaluating mental health services: How do programs for children 'work' in the real world? (pp. 1-15). Thousand Oaks: Sage.
Olds, D. L., Eckenrode, J., Henderson, C. R., Kitzman, H., Powers, J., Cole, R., Sidora, K., Morris, P., Pettitt, L. M., & Luckey, D. (1997). 'Long-term effects of home visitation on maternal life course and child abuse and neglect'. Journal of the American Medical Association, 278, 637-643.
Pietrzak, J., Ramler, M., Renner, T., Ford, L., & Gilbert, N. (1990). Sage sourcebooks for the human services series: Vol. 9. Practical program evaluation: Examples from child abuse prevention. Newbury Park, CA: Sage.
Rapoport, R. (1985). 'Research and action'. In R. Rapoport (Ed.). Research and action: Innovation for children, youth, and families. London: Cambridge Press.
Schalock, R. L., & Thornton, C. V. D. (1988). Program evaluation: A field guide for administrators. New York: Plenum Press.
Tomison, A. M. (1998). Valuing parent education: A cornerstone of child abuse prevention. Issues in Child Abuse Prevention, 10, Melbourne: Australian Institute of Family Studies. Available at: www.aifs.gov.au/nch/issues10.html.
Tomison, A.M., & Poole, L. (2000). Preventing child abuse and neglect: Findings from an audit of prevention programs. Melbourne: Australian Institute of Family Studies. Available at: www.aifs.gov.au/nch/pubs/auditreport.html.
Tomison, A. M., & Wise, S. (1999). Community-based approaches in preventing child maltreatment. Issues in Child Abuse Prevention, 11, Melbourne: Australian Institute of Family Studies. Available at: www.aifs.gov.au/nch/issues11.html.
US Department of Health and Human Services (1995). The program manager's guide to evaluation: An evaluation handbook series from the Administration on Children, Youth and Families. Washington DC: US Department of Health and Human Services.
Vimpani, G., Frederico, M., Barclay, L., & Davis, C. (1996). An Audit of Home Visitor Programs and the Development of an Evaluation Framework. Canberra: AGPS.
Whipple, E. E. & Wilson, S. R. (1996). 'Evaluation of a parent education and support program for families at risk of physical child abuse'. Families in Society, 77, 227-39.
Compiled by Adam Tomison & Nick Richardson, National Child Protection Clearinghouse
Australian Institute of Family Studies - Commonwealth of Australia 2004
300 Queen Street, Melbourne Victoria 3000
Phone: (03) 9214 7888 Internet: www.aifs.gov.au
ISSN 1448-9112 (Online)
Fact sheets can be downloaded from: www.aifs.gov.au/nch/facts/menu.html
Return to Resource Sheets Menu