Research report no.14 2005
The efficacy of early childhood interventions
by Sarah Wise, Lisa da Silva, Elizabeth Webster and Ann Sanson
11. Evaluations of cost-benefit studies of early childhood interventions
Eight of the 32 early childhood interventions reviewed in this report contained a cost-benefit analysis. These studies are assessed here, in the light of how well they serve as a template for an Australian evaluation of an early childhood intervention. Each evaluation was assessed against ten major criteria that relate to the main steps in conducting a cost-benefit analysis: estimating the net impact if the program; treatment of costs and benefits; and formulae used for calculating net effects.
Perry Preschool Project (Perry)
General comments
This is a well-executed and internally coherent cost-benefit analysis. The analysis relies upon estimates of deterministic relationships between variables such as years of education and earnings from secondary sources. These are not critically examined or qualified. It has a sensible treatment of which people to include as beneficiaries and the scope of costs and benefits included. It uses a discursive method to recognise and treat relevant but imprecise costs and benefits.
The main weakness of the analysis is the very small and non-representative sample and thus the very much reduced value for policy makers who wish to generalise about the effects of extending the intervention. This is not a trivial issue since most of the benefits arise from the reduced crime rate for which we are given little information (that is, the number of criminal acts committed by the evaluation sample). There are no considerations of the effects on the family of participants and other members of the community other than in their role as tax-payers.
Nonetheless, it serves as a good template for future cost-benefit analyses.
Estimating the net impact of the intervention
Controlling for selection bias. Program participation was allocated on the basis of random assignment. There is no discussion of whether there is any selection bias ex post although it is indicated that parents could, and did, opt out of the parental meetings.
The size of the participation and control groups. Only 58 participants were included in the intervention group. This is too small to draw many inferences from.
Tests of statistical significance. Tests for whether the net impacts are significant are given, but it is not clear whether these have controlled for differences in mental retardation between the two groups.
Treatment of costs and benefits
Scope of included costs. This is well done and includes all major program and instruction costs.
Scope of included benefits. The main benefits are the material enhancement of the participants, the reduced costs to society from less remedial schooling, and the reduced crime rate. A discussion is made of other life satisfaction benefits, but these are not used in the cost-benefit calculation. This seems a valid list.
Scope of potential beneficiaries. This is limited to the participants, the counterfactual victims of crime and the taxpayer. This seems appropriate.
Time scale considered. Real data is used to age 27 and estimates are extrapolated to retirement age. This is a valid method.
Method used to monetise non-pecuniary benefits. The only non-pecuniary benefit which has been monetised, and which contributes over half of the net present value of benefits, is the effects on the potential victims of crime. The monetary value of the pain and suffering etc that a victim of crime incurs is cited in a footnote and not discussed or rationalised. This is a weakness of the analysis.
Formula used for calculation
Method employed. The analysis uses the net present value (NPV) method, at a 3 per cent rate of time preference (discounted back to the start of the program).
Use of sensitivity analysis. A reasonable use of sensitivity analysis is made with respect to the rate of time preference and the effects of crime on victims.
Cost-benefit findings
The cost-benefit analysis indicated that the benefits totalled $108,002 per child while costs totalled $12,356 per child. This is equal to a saving of $8.74 for every $1 spent. The cost-benefit analysis also indicated that the net benefits remained large even when any one of the benefits was excluded, or if all benefits were reduced by half.
Bolivia Integrated Child Development Program (PIDI)
General comments
Participants in this intervention, which aims to increase the cognitive and physical development of undernourished children in Bolivia, were self-selected. Most of the analysis is devoted to the treatment of the selection issues based on observable characteristics, as the data do not permit the evaluators to control for unobservables. There is no data on the pre-intervention cognitive and physical condition of the children. It does not appear that the intervention was set up with evaluation in mind and this compromises the robustness of the findings.
The cost-benefit analysis is rather scantily done and some of the assumptions are weak. In particular, the assumption that four years of program provision will have a permanent effect on cognitive and physical development has not been justified. The exposé does not make it clear how many observations are included in each regression used to derive the net impacts.
This is not a good example of a cost-benefit analysis.
Estimating the net impact of the intervention
Controlling for selection bias. The authors use a Heckman-type estimation to control for selection issues, but this can only conditioned on observables.
The size of the participation and control groups. The description of the sample size is confusing and it is not clear which size has been used for each regression.
Tests of statistical significance. Tests of significance are provided.
Treatment of costs and benefits
Scope of included costs. Costs are limited appropriately to the intervention costs for the four years. Scope of included benefits. Benefits are limited to the net effect on earnings. This is narrow. It is assumed that changes in height affect earnings even though the net impact on height was not significant at the 10 per cent level.
Scope of potential beneficiaries. Only the intervention participants are included - this is narrow.
Time scale considered. The time scale is limited to two rounds of data gathering. It is not clear what the time period was between these measurement points and the commencement or termination of the intervention.
Method used to monetise non-pecuniary benefits. All intervention data give intermediate outcomes (measures of cognitive and physical development) and these are converted into future earnings using the assumption that all changes are permanent and reference to secondary data linking cognition and physic to earnings. This is a reasonable method.
Formula used for calculation
Method employed. NPV.
Use of sensitivity analysis. Different levels of time preference and base level educational attainment are used to calculate the cost-benefit ratios.
Cost-benefit findings
The cost-benefit analysis estimated that the program cost approximately $43 per child per month (per capita annual GDP is $800). Forty per cent of this cost is consumed by providing children with their nutritional needs. Benefit to cost ratios were found to range from 1.7:1 to 3.7:1 (where benefits in terms of future earnings were the focus).
Chicago Child-Parent Center (CPC)
General comments
This analysis follows the format established by Barnett (1993a, 1993b) in the Perry Preschool Project evaluations. The major differences are, on the plus side, a considerably larger sample (989 in the program and 550 in the control group) but, on the negative side, non-randomised selection into the program. This means that the two comparison groups can vary in unobservable ways. In particular, it is expected that more motivated and determined parents will enrol in the program. This will tend to overstate the benefits from the intervention as control and intervention groups are not alike except for their program participation. Similar methods to the Barnett studies are used to measure costs and benefits.
Estimating the net impact of the intervention
Controlling for selection bias. Selection into the intervention is not random and it is not clear whether the determinants are limited to parental interest, other than the normal income and disadvantage requirements. No attempt is made in the regression analysis to adjust for selection bias, as in the cost-benefit analysis for PIDI.
The size of the participation and control groups. Over 1000 cases are included in the evaluation. This is a reasonable sample.
Tests of statistical significance. Tests of significance are reported and the significant results are used in the cost-benefit analysis.
Treatment of costs and benefits
Scope of included costs. Same set of costs as Barnett above are included. This is adequate.
Scope of included benefits. The same set of benefits are included as Barnett, but like Barnett, some discussion is made of the unmeasured benefits. Similar to other studies, seemingly reputable secondary sources are used to convert intermediate outcomes, such as education, into earnings.
Scope of potential beneficiaries. Same set of beneficiaries are included as Barnett.
Time scale considered. Evaluation data appears to extend to the age of 21 years which is a reasonable length.
Method used to monetise non-pecuniary benefits. Only the public expenditure costs associated with criminal activity, policing, judicial and property costs etc, are included in the analysis so there are no monetary values of non-pecuniary benefits. A similar treatment is made of the costs of child abuse - these benefits are limited to public expenditure savings.
Formula used for calculation
Method employed. NPV.
Use of sensitivity analysis. Several different rates of time preference are used.
Cost-benefit findings
The cost-benefit analysis estimated the cost of the program to be US$6,730 (1998 dollars) for 1 and half years, with a return of US$47,759 per child. Overall, $7.10 was returned to society for every dollar spent (benefits to society were $3.83 for every dollar and government savings was $2.88 per dollar).
Carolina Abecedarian Project (Abecedarian)
General comments
A more expensive and intensive intervention than Perry and Head Start, the Abecedarian involved full-time care for children aged up to five years. The number of children involved in the intervention was very small (104) and the analysis thus suffers from the same difficulties as Barnett's other studies on the Perry Preschool Program. Unlike other studies, this evaluation extends to more speculative benefits such as the effects on future generations and life expectancy of the participants. This is not entirely wrong, but because there is a large variance on the size of the monetised estimates of these benefits, it is good practice to present both conservative as well as the more far-reaching estimates.
Estimating the net impact of the intervention
Controlling for selection bias. Participation was by random assignment, but given the considerable commitment required from parents (relinquishing their child for 40 hours a week from birth), there are likely to be considerable ex post selection issues. This is not discussed.
The size of the participation and control groups. At 104 in both intervention and control groups combined, the numbers are too small for strong generalisations.
Tests of statistical significance. These are presented.
Treatment of costs and benefits
Scope of included costs. Usual program costs are included in addition to the costs of child care for the control group.
Scope of included benefits. These are considerably broader than other studies as they evaluate, using secondary sources, the effects on the education of the next generation and the effects of the significant reduction in smoking on the value of additional years of life to the participant. Estimates are used from the secondary literature to estimate earnings to the age of retirement.
Scope of potential beneficiaries. Unlike other studies, this study includes the effects on the mothers' educational attainment during the intervention period.
Time scale considered. Survey data and school data has been used up to the age of 21 years. This is a reasonable period.
Method used to monetise non-pecuniary benefits. The main non-pecuniary benefit monetised is the value of life. The analysis refers to value of life literature that make this assessment based on how much people are willing to spend in order to reduced their risk of death by a designated percentage. This is a common, but controversial, method used in economics.
Formula used for calculation
Method employed. NPV, but rates of return are also discussed.
Use of sensitivity analysis. Discount rates from 0 to 7 per cent are used. Net present benefits can be decomposed to exclude benefits affecting future generations and those that arise from reduced smoking.
Cost-benefit findings
The cost-benefit analysis found that the average annual cost was about US$13,900 (2002 dollars) per child. The final cost-benefit findings were that the benefits outweighed the costs by $4 to every $1 spent.
Starting Early Starting Smart (SESS)
General comments
This intervention integrates preschool education with programs to reduce parents' mental illness and substance abuse. It began in 1997, and the evaluation is still in the early data collection phase. While there has only been partial random assignment, the numbers of children involved in the evaluation is large. The evaluators are intending to collect extensive detail on program costs and cover abroad scope of potential benefits. No outcomes are presented in the article.
Estimating the net impact of the intervention
Controlling for selection bias. Participation was by random assignment in some centres but in others a control group was constructed. A broad range of control characteristics has been collected. These range from parent-child interaction, home environment, child behaviour as well as the more traditional individual and family characteristics.
The size of the participation and control groups. There are 1900 children in the sample, which is large. Tests of statistical significance. Not applicable.
Treatment of costs and benefits
Scope of included costs. Unusual detail made of program costs, including classifications into fixed and variable, consumable and non-consumable, investment and operating and stakeholder group costs. It is not clear what can be gained from this level of detail.
Scope of included benefits. An extensive set of benefits are proposed to be collected. In addition to the usually educational intermediate outcomes, there will be data on welfare dependence arrests, emergency room visits, family violence and mental illness.
Scope of potential beneficiaries. Unlike other studies, this study includes the effects on the participants' families.
Time scale considered. Survey data and school data will be collected through to adulthood. Method used to monetise non-pecuniary benefits. Not stated.
Formula used for calculation
Method employed. Not decided yet.
Use of sensitivity analysis. Not decided yet.
Cost-benefit findings
The cost-benefit analyses have not yet been conducted.
Florida Family Transition Project (FTP)
General comments
This family based intervention delivers health, education and social services to adults with the aim of reducing welfare dependence and increasing employment income. There is random assignment and a large evaluation sample. Conservative measures are taken on costs and benefits, which are limited to government payments and earnings.
Estimating the net impact of the intervention
Controlling for selection bias. Participants allocated through random assignment and participation was deemed compulsory. There was considerable attrition over the evaluation period, but unless there are reasons to believe that there is bias in the attrition, this is not a problem.
The size of the participation and control groups. There were 2800 people in the evaluation, split evenly between the intervention and control group. This is a large number.
Tests of statistical significance. Not performed.
Treatment of costs and benefits
Scope of included costs. Limited to government welfare costs, and the cost of delivering the program services.
Scope of included benefits. Limited to earnings and fringe benefit differentials (adjusted for tax payments). Estimated by regression analysis (based on some individual characteristics).
Scope of potential beneficiaries. The participant and the taxpayer.
Time scale considered. Five years post intervention. This is a long time period for an employment program evaluation.
Method used to monetise non-pecuniary benefits. No non-pecuniary benefits are monetised but some are discussed.
Formula used for calculation
Method employed. NPV.
Use of sensitivity analysis. Not clear.
Cost-benefit findings
The cost-benefit analysis indicated that the program costs were approximately US$12,500 per family member over the five-year period. The net costs, over and above what was spent on the usual welfare program were US$8,000 per family. The cost-benefit analysis found that the FTP produced a net loss to the government of US$6,300 per family.
Triple-P Positive Parenting Program (Triple P)
General comments
This is a well-conducted and thorough evaluation with a reasonable sized intervention and control groups. Unlike other analyses, it uses a cost effectiveness method but in order to capture some of the wider benefits caused by an intervention it recasts them as cost savings. The short time frame for measuring the benefits of the intervention is a limitation.
Estimating the net impact of the intervention
Controlling for selection bias. Participants were allocated through referral and there were several levels of intensity of treatment. In most cases the control group was drawn from families on the waiting list for the intervention. However, it was not clear how long the waiting lists were. There would have to be very long waiting lists for good counterfactual data to be collected. There was considerable attrition over the evaluation period, but unless there are reasons to believe that there is bias in the attrition, this is not a problem. Children with additional developmental or health problems were excluded. A comparison of the family background characteristics showed that the intervention and control groups were not significantly different.
The size of the participation and control groups. Five separate 'randomised' trials were conducted involving 567 children. This is a reasonable number if the random assignment is truly random. No details on refusal rates for parents who did not want to participate, but it is implied that all participants and the control groups are self selected. This may be appropriate for this type of intervention that requires a high level of parental cooperation. Attrition rates are provided.
Tests of statistical significance. Provided.
Treatment of costs and benefits
Scope of included costs. Program costs such as fees for counselling sessions, the prices for workbooks and materials are included but the time and medial costs (can be negative) affecting the family are excluded. Because of the limited scope for including benefits in a cost effectiveness formulae, some benefits are included as cost savings, such as education, health, foster care and crime costs associated with fewer people, up to the age of 28, with conduct disorders. These estimates are derived from secondary literature.
Scope of included benefits. Whether the child developed or maintained a conduct disorder was the only benefit. Only one benefit is permitted in costs effectiveness studies but one way around this is to include other benefits in the cost side as cost savings.
Scope of potential beneficiaries. The child.
Time scale considered. Six months to three year follow-ups undertaken depending on the trial.
Method used to monetise non-pecuniary benefits. Not applicable as a cost effectiveness method used.
Formula used for calculation
Method employed. Cost effectiveness. The benefit of the intervention is the number of child conduct disorders averted. The evaluators considered the number of disorders that need to be prevented in order for the intervention to cover costs. No account taken of reduced mental health problems in parents (considered to be a secondary benefit of the intervention).
Use of sensitivity analysis. Conducted.
Cost-benefit findings
Triple P costs range from 75c at Level 1 to $422.45 at Level 4 (individual) in Australian 2003 dollars. The cost-effectiveness analysis indicated that the intervention would pay for itself if it averted less than 1.5 per cent of conduct disorder cases and that an aversion rate of 7 per cent or more would result in a cost saving.
Elmira Prenatal and Early Infant Project (PEIP)
General comments
The cost-benefit analysis appears to include both costs and benefits affecting the mother and child as well as transfer payment between the family and the government. This appears to include double counting. Ideally, a cost-benefit analysis should be separated from a statement of government accounts.
Estimating the net impact of the intervention
Controlling for selection bias. It has been described as a randomised trial, but few details are provided.
The size of the participation and control groups. Three hundred in combined intervention and control groups.
Tests of statistical significance. No.
Treatment of costs and benefits
Scope of included costs. Program costs and other savings to government are included.
Scope of included benefits. Mother behaviours including smoking, attendance at child related classes, nutrition, child abuse and neglect rates, education, substance abuse and criminal activity. Child effects included IQ, and criminal activity.
Scope of potential beneficiaries. Includes both mother and child.
Time scale considered. Benefits monitored up until the age of 15.
Method used to monetise non-pecuniary benefits. Only a subset of the benefits were monetised, including emergency room visits, use of welfare, and use of the criminal justice system.
Formula used for calculation
Method employed. This is an unusual combination of costs and benefits to the child, mother and family, as well as measures of transfer payments between the family and government (taxes and welfare payments).
Use of sensitivity analysis. No.
Cost-benefit findings
The cost-benefit analysis estimated that the costs of the program were US$3,300 in 1980 dollars and US$6,700 in 1997 dollars per child for two and a half years of service. The analysis also indicated that investment was recovered before the children turned four years old and the intervention saved US$4 for every US$1 spent. However, the benefits exceeded the costs only for families where the mother was of low income and unmarried.
Effectiveness of early childhood interventions with a cost-benefit analysis
Cost-benefit analyses were available for at least one program from each of the five intervention clusters specified in this report. This section summarises the interventions in each of the clusters according to the adequacy of the evaluation design, program efficacy and cost effectiveness.
Cluster 1: targeted, child focused, centre based, preschool age
Three of the interventions in cluster 1 included a cost-benefit analysis in their evaluation. Two of these cost-benefit analyses (for Perry and CPC) were appropriately executed. However, the cost-benefit findings of PIDI need to be interpreted with caution due to a number of issues discussed above.
A saving of US$8.74 was found for every dollar spent on Perry, a saving of US$7.10 for every dollar spent on CPC and cost-benefit ratios ranging from 1.7:1 to 3.7:1 for PIDI. This suggests that interventions in cluster 1 provide a good return on investment. In addition, Perry and CPC were well implemented and the evaluations of both were well-designed. Effect sizes for Perry ranged from large in the short-term to medium in the intermediate term, while effect sizes for CPC ranged from small in the intermediate term to small in the long-term. This suggests that the cost-benefit analyses of Perry and CPC adequately represent the intended interventions.
Cluster 2: targeted, parent focused, home visits, all ages
Cost-benefit analyses were conducted for the Elmira PEIP intervention. This analysis indicated a saving of US$4 for every dollar spent - however, this only applied to low-income and single parent families. In addition, the calculation of the costs and benefits was unconventional. The evaluation design of the PEIP was excellent and effect sizes ranged from large in the short-term to medium in the intermediate term. Although the PEIP demonstrated good return on investment for a select group, it is not possible to generalise this finding to other parent focused, home visitation-type interventions.
Cluster 3: targeted, family economic/welfare focused, all ages
A cost-benefit analysis was conducted on the FTP and found a loss, as opposed to a saving. Although the evaluation of FTP was well designed, effect sizes were not published. It is therefore inappropriate to comment whether the poor return on investment demonstrated by the FTP generalises to other interventions with an economic or welfare focus.
Cluster 4: targeted, holistic, various locations, all ages
A cost-benefit analysis was conducted on the Abecedarian project. A saving of US$4 for every dollar spent was found. Adding further strength to this finding is the adequate cost-benefit techniques, good implementation of the intervention and well-designed evaluation. Again, it is difficult to generalise this finding to a range of targeted, holistic early childhood interventions.
Cluster 5: universal, various foci, various locations, all aged
Triple P was the only intervention in cluster 5 that produced a cost analysis, however this was only a cost-effectiveness analysis, which does not include an extensive calculation of program benefits. Therefore, it is difficult to make any conclusions about return on investment for universal interventions.
Summary
Focusing narrowly on the limited cost-benefit data for early childhood interventions reviewed here, there is some indication that interventions that involve children as participants, or that focus on enhancing parenting efficacy, and that are intensive in nature, have greater cost savings potential than interventions that focus solely on familial economic circumstances. However, reliable conclusions about the relative cost savings of early childhood interventions require additional cost-benefit data on a more representative sample of programs.