Evaluation methodologies

This bibliography provides a selection of recent references from the Australian Family & Society Abstracts  database and the Australian Institute of Family Studies library catalogue.

The publications listed are held in the Australian Institute of Family Studies Library and can be borrowed via the inter library loan system or acquired directly from the issuing body. Direct links to documents on the Web are provided when available.

References are arranged alphabetically by title.

 

0027 case study: using a modified evaluation methodology for assessing the performance of contract service providers.
Davies, Greg; Milne, Chris
In: Evaluation in emerging areas: Australasian Evaluation Society International Conference 2006. Lyneham, ACT: Australasian Evaluation Society, 2006, 7p, Online only (PDF 72KB)

The Australian Government's Reconnect youth homelessness intervention program utilises nearly 100 service providers and has been refined over time to meet changing local needs. In this paper, the authors describe the development and methodology of the service evaluation approach they devised to evaluate such a varying and complex program. The authors outline the service requirements, performance ratings, and individual assessments, and discuss the strengths and weaknesses of this approach and alternative assessment methods.

 

A conceptual framework for making evaluation support meaningful, useful and valuable.
Coryn, Chris L S
Evaluation Journal of Australasia v.6 no.1 2006: 45-51

How does support differ from reporting and dissemination? How, and to what extent do existing evaluation theories attend to support? How, and for what purposes should support be provided? Does support infer that evaluators have ethical, moral, professional, contractual or other obligations to provide support? Is support an evaluation service, or should it be considered an evaluator competency or skill? Or, as with the traditional research paradigm, should evaluators merely let their reports speak for them? While the Key Evaluation Checklist (KEC) and other evaluation theories and approaches have provided a conceptual basis for evaluation support, further clarification is necessary in order to make support an integral part of evaluation practice. The evaluation support construct as posited here necessitates means that are direct and indirect, technical and general, and includes alternative scenarios for the purposes of advocating for, assisting and helping evaluands, clients, stakeholders, and audiences and users of evaluation. (Journal abstract)

 

Approaches to evaluation of affordable housing initiatives in Australia: National Research Venture 3, housing affordability for lower income Australians.
Milligan, Vivenne; Phibbs, Peter; Gurran, Nicole; Fagan, Kate
Melbourne, Vic: Australian Housing and Urban Research Institute, 2007, 94p (Research paper no.7), and Online (PDF 532KB)

A project was undertaken to develop an evaluation framework and to propose evaluation techniques suitable for application to existing or new affordable housing programs and initiatives in Australia. This report: reviews the range and mix of policy ideas and programs being discussed to improve housing affordability for selected groups in Australia; develops an approach to evaluating the cost effectiveness, equity and appropriateness of these interventions on a consistent basis; and illustrates the approach by applying it to selected schemes or projects.

 

Assessing outcomes of health and medical research: do we measure what counts or count what we can measure?
Wells, Robert; Whitworth, Judith A
Australia and New Zealand Health Policy v.4 Jun 2007: 6p, Online

Governments worldwide are increasingly demanding outcome measures to evaluate research investment. Health and medical research outputs can be considered as gains in knowledge, wealth and health. Measurement of the impacts of research on health is difficult, particularly within the time frames of granting bodies. Thus, evaluations often measure what can be measured, rather than what should be measured. Traditional academic metrics are insufficient to demonstrate societal benefit from public investment in health research. New approaches that consider all the benefits of research are needed. (Journal abstract)

 

Challenges and issues in applying empowerment evaluation principles in practice: case study of the evaluation of a national school breakfast program.
Miller, Wayne; Lennie, June; Yeatman, Heather
In: Evaluation in emerging areas: Australasian Evaluation Society International Conference 2006. Lyneham, ACT: Australasian Evaluation Society, 2006, 13p, Online only (PDF 80KB)

Empowerment evaluation is a collaborative and participatory model of evaluation, recommended for evaluating community programs. This paper describes the use and benefits of empowerment evaluation principles, as shown in the evaluation of the Good Start Breakfast Club, a national school breakfast program for primary schools in Australia. After describing the program and the evaluation process, the authors use the 10 empowerment evaluation principles to assess the findings from the program's evaluation workshops in New South Wales.

 

Complexity in longitudinal evaluation of integration programs: participatory approach in the case study of Qld Resident Support Program.
Fisher, Karen; Robinson, Sally
In: Evaluation in emerging areas: Australasian Evaluation Society International Conference 2006. Lyneham, ACT: Australasian Evaluation Society, 2006, 8p, Online only (PDF 52KB)

Complex integration programs across government agencies, service types and professions present methodological difficulties for evaluation. This paper uses the case study of the evaluation of the Resident Support Program Queensland, conducted in 2003-05. The Program coordinates and provides support services for residents living in the private residential services sector (boarding houses). The paper discusses how the evaluation design responded to the complexity in the program and the evaluation task. Participatory methodology, prioritising the interests of people with disability, was used to overcome these complexities. (Author abstract, edited)

 

Doing evaluation research.
White, Rob
In: Walter, M. ed. Social research methods: an Australian perspective. South Melbourne, Vic: Oxford University Press, 2006, p159-185, figures, tables

Ways of doing evaluation are examined in this 'evaluation toolbox'. The chapter discusses the purposes of evaluation, methods and techniques of evaluation, and how to prepare an evaluation report. It includes case studies and exercises. The chapter emphasises that social research evaluation, because it directly involves people, requires honest reflection about aims, recognition of moral and ethical responsibilities, and a professional approach to the gathering of information.

 

Evaluating human service delivery through client outcomes.
O'Brien, Terrence; Whiting, Scott; Stephenson, Anne; Hinch, Simon; Pearson, Michelle; Hockey, Neil; O'Malley-Jones, Madeleyn
In: Australasian Evaluation Society International Conference 2005. Lyneham, ACT: Australasian Evaluation Society, 2005, 9p, Online only (PDF 168KB)

This paper describes how evaluating client outcomes can be used in evaluating services. It discusses measuring outcomes, evaluation rationale of prevention and early intervention programs within the Queensland Department of Communities, and evaluation methodologies, and describes two of the pilot programs, Project Circuit Breaker, a crisis family support and child protection service in northern Brisbane, and the Mununjali Jymbi Centre, a family and community program for Aboriginal people in the Beaudesert region.

 

Evaluation capacity-building: a tale of value-adding.
Beere, Diana
Evaluation Journal of Australasia v.5 no.2 2005: 41-47

Evaluation capacity building entails not only developing the expertise needed to undertake robust and useful evaluations; it also involves creating and sustaining a market for that expertise by promoting an organisational culture in which evaluation is a routine part of the way things are done. A challenge for evaluators is to contribute to evaluation capacity building while also fulfilling their key responsibilities to undertake evaluations. A key strategy is to focus on both discerning value and adding value for clients and commissioners of evaluations. This paper discusses two related internal evaluation projects conducted for the Queensland Police Service that have added value for the client and so, have helped to promote and sustain an evaluation culture within the organisation. It describes key elements of these evaluations that contributed to evaluation capacity building. The paper highlights the key role that evaluators themselves, especially internal evaluators, can take in evaluation capacity building. (Journal abstract, edited)

 

Evaluation methodology basics the nuts and bolts of sound evaluation.
Davidson, E. Jane
Thousand Oaks, Calif.: Sage Publications, c2005. xvi, 263 p. ; 24 cm.

This book provides a step-by-step guide for conducting an evaluation. It focuses on the main kind of 'big picture' questions that need to be answered. It examines how the nature of such questions is linked to evaluation methodology choices. How to combine a mix of qualitative and quantitative data with 'relevant values' (such as needs) to draw explicitly evaluative conclusions is also explained.

 

Evaluation: what is it really for? A framework for examining issues of evaluation purpose and use.
Barlow, Anne
Evaluation Journal of Australasia v.5 no.1 Sept 2005: 11-17, tables

A project that evaluated midwifery services in a small New Zealand public maternity unit is used in this article as a case study to illustrate challenges relating to evaluation purpose and the dissemination and use of findings. The article suggests that the purpose and findings of any evaluation study require a degree of acceptance in order to be useful. It proposes an acronym framework based on the keyword 'acceptance', which can include the following themes: accessibility, audience, criteria, credibility, credentials, communication, ethics, purpose, politics, participants, position, timeliness, tone, attitude, appearance, numbers, critique, excellence.

 

Experimental and quasi-experimental methods of microeconomic program and policy evaluation.
Borland, Jeff; Tseng, Yi-Ping; Wilkins, Roger
Melbourne, Vic: Melbourne Institute of Applied Economic and Social Research, University of Melbourne, 2005, 40p, tables, figures (Melbourne Institute working paper no.8/05), and Online (PDF 278K)

New empirical methods for evaluating microeconomic policies are reviewed in this paper, which describes the motivation for the use of experimental and quasi-experimental methods, the types of policy effects that they can identify, and how they are implemented. The paper defines the impact evaluation approach, and what is meant by experimental and quasi-experimental methods; describes the main methods of estimating program and policy impacts; reviews several Australian applications of experimental and quasiexperimental methods; and discusses ideas on the way forward for program and policy evaluation in Australia.

 

Handbook of evaluation policies, programs and practices.
Shaw, Ian. (ed); Greene, Jennifer C. (ed); Mark, Melvin M. (ed)
London, Sage, 2006. xxi, 608 p. ill.; 26 cm.

Contents: The Evaluation of Policies, Programs and Practices: An Introduction / Melvin M Mark, Jennifer C Greene and Ian Shaw -- The Purposes of Evaluation in a Democratic Society / Eleanor Chelimsky -- Roles for Theory in Contemporary Evaluation Practice Developing Practical Knowledge / Stewart I Donaldson and Mark W Lipsey -- Evaluation for Practice Improvement and Organizational Learning / Patricia J Rogers and Bob Williams -- Evaluation and the Study of Lived Experience / Thomas A Schwandt and Holli Burgon -- Evaluation, Democracy and Social Change / Jennifer C Greene -- Evaluation after Disenchantment? Five Issues Shaping the Role of Evaluation in Society / Peter Dahler-Larsen -- Government as Structural Context for Evaluation / Phil Davies, Kathryn Newcomer and Haluk Soydan -- The Social Relations of Evaluation / Tineke A Abma -- Intellectual Contexts / John Stevenson and David Thomas -- The Relationship between Evaluation and Politics / Ove Karlsson Vestman and Ross F Conner -- Ethics in Evaluation / Helen Simons -- A Comparative Analysis of Evaluation Utilization and Its Cognate Fields of Inquiry Current Issues and Trends / J Bradley Cousins and Lyn M Shulha -- Contextual Challenges for Evaluation Practice / Elliot Stern -- Methods for Policymaking and Knowledge Development Evaluations / Melvin M Mark and Gary T Henry -- Embedding Improvements, Lived Experience and Social Justice in Evaluation Practice / Elizabeth Whitmore U [et al.] -- Managing Evaluations / Robert Walker and Michael Wiseman -- Communicating Evaluation / Marvin C Alkin, Christina A Christie and Mike Rose -- On Discerning Quality in Evaluation / Robert E Stake and Thomas A Schwandt -- The Practice of Evaluation : Challenges and New Directions / Lois-ellin Datta -- Evaluation in Education / David Nevo -- Evaluation of Health Services Reflections on Practice / Andrew Long -- Social Work and the Human Services / Ian Shaw with Carol T Mowbray and Hazel Qureshi -- Evaluation in Criminal Justice / Nick Tilley and Alan Clarke -- Evaluation of Development Interventions and Humanitarian Action / Osvaldo Feinstein and Tony Beck -- Evidence-Based Evaluation in Different Professional Domains Similarities, Differences and Challenges / Alan Clarke.

 

Improving accountability for participatory processes through effective evaluation.
Carson, L; Twyford, Vivien
In: International Conference on Engaging Communities, 14-17 August 2005: Conference abstracts and papers. Brisbane, Qld: Department of Communities, 2005, 7p, Online (PDF 49K)

This paper presents the results of a study that developed and tested tools to evaluate the efficacy of public participation frameworks in Australia. The paper includes a review of the literature on public participation.

 

Increasing the rigour and trustworthiness of participatory evaluations: learnings from the field.
Lennie, June
Evaluation Journal of Australasia v.6 no.1 2006: 27-35

Participatory evaluation and participatory action research (PAR) are increasingly used in community based programs. These methodologies focus more on knowledge generated through lived experience than through social science. The scientific ideal of objectivity is usually rejected in favour of a holistic approach that acknowledges the diverse perspectives, values and interpretations of participants and evaluation of professionals. However, evaluation rigour need not be lost in this approach. Reflections about the use of feminist and participatory forms of evaluation and PAR have identified significant sources of rigour: participation and communication methods that develop relations of mutual trust and open communication; using multiple theories and methodologies and multiple sources of data and data collection; ongoing meta evaluation and critical reflection; critically assessing the intended and unintended impacts of evaluations, using relevant theoretical models; using rigorous data analysis and reporting processes; and participant reviews of evaluation case studies, impact assessments and reports. (Journal abstract, edited)

 

Participatory evaluation in practice: challenges and early lessons with an Aboriginal community run centre for Aboriginal boys.
Spooner, Catherine; Flaxman, Saul; Murray, Colleen; Seymour, MaryAnn; Pulver, Lisa Jackson; Moore, Elizabeth; Howard, John
In: Australian Social Policy Conference 2007. Sydney, NSW: Social Policy Research Centre, University of New South Wales, 2007, 14p, Online only (Power point presentation in PDF format 312KB)

This presentation outlines the development of a program evaluation and monitoring system for the Tirkandi Inaburra Cultural and Development Centre, an Aboriginal community-run residential program in rural New South Wales for at-risk teenage boys. A participatory research approach was selected, to address such challenges as distance, budget, cultural appropriateness, informed consent, and interference with probity.

 

People-centred evaluation.
Dart, Jessica; McGarry, Phil
In: Evaluation in emerging areas: Australasian Evaluation Society International Conference 2006. Lyneham, ACT: Australasian Evaluation Society, 2006, 25p, Online only (PDF 136KB)

This paper outlines the People-Centred Evaluation approach which has been used in both overseas development and the Australian public sector. It is well suited for developing practical internal monitoring and evaluation systems for projects that aim to encourage practice change and capacity building amongst target stakeholders. Having a strong program logic element, People-Centred Evaluation helps groups clarify their project design. (Author abstract, edited)

 

PEW: 'dances with the budgies': the development of the Planning and Evaluation Wizard: a cautionary tale.
Aylward, Paul; Murray, Charlie; Cooke, Richard; Martin, Maryanne
Evaluation Journal of Australasia v.5 no.2 2005: 61-66, ill

This article is based on the reflections of the authors of the Planning and Evaluation Wizard (PEW), a software tool produced by the South Australian Community Health Research Unit (SACHRU), which aims to assist primary officers working in primary health care. PEW takes the user through the steps of developing grant applications, project and evaluation plans, including budgets, and project reports. PEW provides practical suggestions as users progress through the stages, based on the experience of project officers, SACHRU staff, and drawing on relevant literature. This paper is a cautionary tale of hope, struggle, compromise and endurance, a tale where the PEW protagonists were drawn into a series of prolonged and contorted dances in the honourable pursuit of developing a software tool to enhance evaluation in primary health care. (Journal abstract, edited)

 

Program evaluation: forms and approaches.
Owen, John M.
Crows Nest, N.S.W. Allen & Unwin, 3rd ed., 2006. 298p.

Everyone involved with policy and program development and delivery is being asked to plan more carefully, reflect more critically and justify their decisions. The key to this is evaluation. This book shows evaluators how to identify appropriate forms, approaches and methods, using an original framework. This third edition has been thoroughly revised to incorporate recent research on evaluation and new examples of good practice.

 

Reconciling tensions between principle and practice in Indigenous evaluation.
Scougall, John
In: Evaluation in emerging areas: Australasian Evaluation Society International Conference 2006. Lyneham, ACT: Australasian Evaluation Society, 2006, 11p, Online only (PDF 92KB)

The author reflects on the difficulties in program evaluation in Indigenous communities, as experienced during the evaluation of the Australian government's Stronger Families and Communities Strategy. Difficulties include the tensions between the ideal participatory research methods and the limited resources of reality, and historic mistrust. The author describes the background of the Strategy, and discusses working with Aboriginal communities and appropriate evaluation practice, where evaluation can balance professional with local knowledge and reporting with capacity building.

 

Reflections on evaluation practice and on the 2005 conference: some observations from a grumpy old evaluator.
Funnell, Sue
Evaluation Journal of Australasia v.6 no.1 2006: 3-10

Concerns about some current practices in evaluation are discussed. These are: monitoring and evaluation as a package deal, which often leads to evaluation missing out; the 'silver bullet' mentality of some programs, which fails to recognise complexity of causality; evaluation frameworks that straight jacket all program participants to achieve similar outcomes; and uncritical and indiscriminate use of data for evaluation purposes.

 

Spiralling up or spinning out: a guide for reflecting on action research practice.
Badham, Richard J; Sense, Andrew J
International Journal of Social Research Methodology: Theory and Practice v.9 no.5 Dec 2006: 367-377, figure

Action researchers face numerous pressures in meeting the demands of the dual customers - practitioners and academics - of an action research process. This paper describes an 'action research spiral' model to guide reflection by researchers on this dual client focus. It argues for a greater degree of reflection on action research practice, and uses action research cases to illustrate those reflective processes.

 

Targeting what matters in health promotion evaluation: using the RE-AIM approach to identify success in real-world settings.
McKenzie, Rosemary; Naccarella, Lucio; Stewart, Andrew; Thompson, Catherine
Evaluation Journal of Australasia v.7 no.1 2007: 19-26

This article examines the RE-AIM evaluation framework, as an approach that can establish the public health impact of a health promotion program. The article presents the practical application of RE-AIM in evaluation of multi-project, multi-setting health promotion programs, illustrated by the evaluation of three statewide programs: a three-year older adult health promotion program, a three-year diabetes prevention program, and a one-year health promotion program in public sector aged care, all set in Victoria. It considers how the RE-AIM approach can be employed to provide insights into real-world program domains that are frequently overlooked in conventional impact evaluations. The article concludes that RE-AIM is an adaptable, easy-to-use evaluation approach suited to multi-project program evaluations that can be used in a range of settings and sectors. (Journal abstract, edited)

 

The cost-effectiveness of homelessness programs.
Flatau, Paul; Zaretzky, Kaylene
Parity v.20 no.7 Aug 2007: 23-26

Cost effectiveness studies have important implications for policy decisions. However, measuring the client outcomes of homelessness programs and assessing the net cost of delivering services to homeless people poses some methodological difficulties. This article reviews the issues for cost effectiveness studies of homelessness programs. It considers the following issues: collaborative research partnerships; client needs; client outcomes; net costs; recurrent funding to programs; and cost offsets.

 

The Success Case Method: a simple evaluation tool to identify critical success factors and program impact.
Bell, Catherine; McDonald, Diane
In: Evaluation in emerging areas: Australasian Evaluation Society International Conference 2006. Lyneham, ACT: Australasian Evaluation Society, 2006, 9p, Online only (PDF 68KB)

This paper shows how the Success Case Method evaluation tool can be used to identify and report on critical success factors and program impact in large-scale evaluations. The authors describe their own experiences using this tool in an evaluation of the Victorian Government's Primary Welfare Officer Initiative, a program for improving the capacity of schools to support students at risk of disengagement. The initial evaluation of this Initiative produced a large volume of unwieldy qualitative data. The Success Case Method was then employed, with benefits in efficiency, targeting, and ease of use. The paper describes the methodology of the Success Case Method, its benefits and limitations, and the evaluation process.

 

Using secondary data for evaluating community programs: opportunities and challenges.
Katz, Ilan
In: Evaluation in emerging areas: Australasian Evaluation Society International Conference 2006. Lyneham, ACT: Australasian Evaluation Society, 2006, 10p, Online only (PDF 88KB)

This paper discusses the benefits and challenges of using secondary data in program evaluation, with some examples from the Australian context. The paper discusses the types of secondary data, its advantages, and possible problems in consistency, sensitivity, timeliness, and degree of proximity to required outcome measures. Administrative data for example is becoming increasingly available, and can be seen as a cheaper and more obviously scientific form of data.

 

Youth participation in evaluation: young people should be seen and heard!
Bell, Catherine; St Leger, Pam
In: Evaluation in emerging areas: Australasian Evaluation Society International Conference 2006. Lyneham, ACT: Australasian Evaluation Society, 2006, 11p, Online only (PDF 2.2MB)

This paper highlights the role of youth participation in the evaluation process of educational initiatives. Youth collaboration and participation is outlined in 2 case studies: the RUMAD? (Are you making a difference?) program, which engaged primary and secondary students in community development, and the Glenormiston Leadership Pilot Program, an initiative for year 9 secondary students in an alpine region. The paper describes the programs; student participation in program design, implementation and evaluation; and the use of photographs by students as a form of evaluation data.

 

 

* You will need an Acrobat Reader which is free from the Adobe Systems Web site: http://www.adobe.com/prodindex/acrobat/readstep.html