Upcoming cause

Pre-K day care programs

This page gives a brief overview of what we know about pre-K day care programs and their effect on later life outcomes for the children they serve. We believe that there are strong reasons to believe these programs have great potential; however, of the organizations we have reviewed in this area, none has given us strong reason to believe that it stands above the others, and none has shown the level of self-monitoring and -evaluation we would need to feel confident in it (either through directly monitoring long-term outcomes or demonstrating fidelity to a proven model). We found a literature review by W.S. Barnett, "Does Head Start Have Lasting Cognitive Effects?", particularly useful in summarizing the conclusions of relevant research. We read several of the papers it refers to and found its assessment broadly similar to our own; in addition, Barnett lays out clear, firm criteria by which he chose which papers to examine (his criteria are based entirely on methodology, not conclusions) and provides tables with the research design and results of each paper, so that the reader need not rely on his subjective conclusions. Unfortunately, to our knowledge this essay is only available in hard copy, as Chapter 16 of The Head Start Debates (edited by Edward Zigler and Sally Styfco, published in April 2004). Types of Pre-K day care programs We focused our research for this cause on New York City; below we detail a few relevant terms for different types of day care programs.
  • Model programs is a term frequently used in the academic literature (including the Barnett paper) to refer to programs that are developed, executed, and closely monitored by experts and researchers, generally with the express intent of demonstrating what child care is capable of. Such programs are often very high-intensity: for example, the Carolina Abecedarian Program was a full-day, year-round program that lasted from infancy through the age of five.
  • Head Start refers to a federal program - legal specifications available here - that directly reviews applications from child care programs, and assigns funding (in addition to managing ongoing assessment of programs). As outlined in this overview, Head Start is a "comprehensive child development" program, targeting children below the federal poverty line, that includes "educational, health, nutritional, social and other services to enrolled children and families." Head Start funding is generally used for part-year, part-day programs for 4-year-olds; more details, and examples, are available in the Barnett paper.
  • Universal Pre-K is a New York State program (legal specifications here) that, like Head Start, directly funds child care programs and carries out its own evaluations. Funding can be used in combination with other sources - for example, to provide full-day instead of part-day care. More details are available here. The term "Universal Pre-K" can also refer to other state programs, but we have focused on its New York context.
  • New York child care centers have a variety of other funding sources available to them (one overview is available in the early pages of this document from United Way). One such program is the state-run Universal Pre-K program, which conducts its own instructional reviews; many of our applicants have their written reviews available, providing an additional third-party opinion on the quality of a program.
Evidence of effectiveness: model programs The Barnett paper cited above summarizes the results of 15 studies of model programs. Each of these studies tracked both a "treatment group" of children who participated in model programs and a "comparison group" of children who did not, and compared the two groups after several years had passed (each study goes at least as late as third grade). About half employed experimental design: families pre-applied to the programs, and participants were chosen by random lottery, meaning that any differences between the two groups could only be attributed to the effect of the program or to random chance. Out of 15 such studies of such programs, 9 showed the treatment group as statistically significantly better off, as of 3rd grade or later, by at least one of the following measures: IQ (rare), special education and grade retention, achievement test scores and statistically equivalent on the others; 5 didn’t show strong effects one way or the other; in one, the treatment group had lower IQ. The studies that were methodologically strongest, and most careful in their selection of at-risk children, showed the most impressive effects. These include the Perry Preschool project, which found preschool participants to have significantly higher achievement test scores and high school graduation rates at age 18, and the Carolina Abecedarian project, which found preschool participants to have significantly superior achievement test scores and college enrollment rates at age 21. These two programs are frequently cited as examples of how much preschool programs can accomplish. See pages 226-229 of the Barnett study for more details as well as full citations. We feel that these results give strong reasons to be optimistic about what quality preschool can accomplish. However, because these programs were designed to be "exemplary" - and were often better funded and more intense than typical Head Start programs - we feel it would be a mistake to generalize their results to all preschool programs. Evidence of effectiveness: Head Start and other large-scale programs Head Start comes with its own assessment program, and participants have been observed to meet certain quality guidelines relatively consistently (see page 222 of the Barnett paper for an overview of the research on implementation, along with citations). Therefore, a broad enough survey of outcomes may be somewhat generalizable. A large-scale impact study is currently underway, but no data on later life outcomes is yet available. Barnett reviews 24 studies of Head Start or similar programs ("12 studied the effects of Head Start programs, 4 studied a mix of Head Start and state or local programs, and 8 studied state or local programs"). Like the studies of model programs cited above, these studies track both a "treatment group" of children who participated in model programs and a "comparison group" of children who did not, and compared the two groups after several years had passed (each study goes at least as late as third grade). However, none of these studies employed experimental design; most simply compared children who had attended preschool to those who had not, raising the concern that any differences could reflect pre-existing differences in families - related to the fact that some families sent their children to preschool, and others didn't - rather than actual effects of the program. The only two studies that made any attempt to address this issue suffered from high attrition, i.e., many of their subjects dropped out of the study before it was complete. 15 of these 24 studies (including 1 of the 2 that seems to have attempted to control for selection bias) showed Head Start children with a statistically significant advantage as of grade 3 or later (and in the last year of the study) on at least one of: high school graduation, achievement tests, special education, grade retention. 9 showed mixed or non-statistically significant effects. See pages 233-236 of the Barnett paper for details. Barnett also observes that differences between Head Start and non-Head Start children may be understated in some cases, because groups with weaker academic performance also lost more students to grade retention and special education, causing their average test scores (as measured) to rise artificially. While we are concerned about potential sources of bias, we believe these studies give some reason to be cautiously optimistic overall, and we look forward to learning more from the large-scale impact study currently in progress; but knowing what we know, we are not confident in the lasting effects of non-model preschool programs. Our applicants We had a total of 9 applicants running preschool programs for disadvantaged children, but none provided us with either:
  • A compelling case that their program resembles an already proven "model program."
  • Independent evidence for their program's effect on later life outcomes.
We were in some cases sent reports on children's cognitive development, but these reports either did not show consistent progress for enrolled children, or did not use measures that allowed us to compare the progress to what might be expected in the absence of a preschool program. In addition, none of these tracked children after their enrollment in kindergarten. This is a particular concern since there are many studies of preschool programs showing short-term improvement that fades out entirely - or weakens considerably - over time (the studies mentioned above, summarized in the Barnett paper, provide many examples of this phenomenon). All of the reports in this category were sent in hard copy only. Below, we list all of the applications we received from day care programs, and mark which ones receive funding from Head Start and Universal Pre-K.

Nurse-Family Partnership (NFP)

In a nutshell:

What do they do? The Nurse Family Partnership supports programs offering home visits (from trained registered nurses) to low-income, first-time mothers. Nurses advise the mothers on prenatal health, child care, and personal life management (including birth spacing and employment counseling). Does it work? Yes. The program has been the subject of several formal, rigorous evaluations that imply measurable impacts on both mothers (particularly on how long they go between births) and children (particular cognitive and behavioral measures). NFP stands above any other developed-world organization we know of in terms of its commitment to replicating a proven model. What do you get for your dollar? We estimate the cost of the full 2.75-year program as ranging from $10,000-$14,000 per family. We do not have the data to say how often the program makes a significant difference in a particular child's life, but across large numbers of children, a statistical difference is noticeable.

The Details:

What do they do? The NFP model consists of having trained registered nurses visit low-income, single mothers – starting with weekly visits early in pregnancy, and progressing toward monthly visits until the child's second birthday – attempting to help with prenatal health (nutrition; reducing alcohol/tobacco/drug use during pregnancy; obtaining prenatal care), child care (creating a safe and supportive home environment), and personal life management (birth spacing as well as taking steps toward education and employment). A summary of topics covered, as well as frequency of visits, is available at the Home Visit Experience page on NFP's website. NFP's national office primarily provides a consulting (as opposed to funding) role, partnering with regional organizations in order to carry out its model. Attachment [[Program descriptions]] lays out the national office's activities, along with the associated 2007 expenses (which we cite below as a proportion of the organization's total expenses):
  • Program development (19%): helping regional organizations raise funding.
  • Nursing practice (12%): training and supporting participating nurses.
  • Program quality support (28%): ongoing evaluation and consultation with regional programs. The national office collects extremely thorough and detailed data from its implementing programs, which it then compares to benchmarks, monitoring compliance and providing advice; see Attachment [[example of monitoring]] for a sample review.
  • Federal policy & program finance (5%): advocacy for more government funding of programs on the NFP model.
  • Other (37%): marketing, administrative and management costs; a small amount of direct funding for regional programs.
Does it work? The primary evidence for the NFP model's effectiveness consists of three different studies - each of which has been reported on several times - in which low-income mothers were (or were not) assigned nurses based on lottery, and then researchers followed both those who did and did not receive nurse visits, collecting a broad range of data. We find the evidence from these studies to provide a compelling case that the model in question - a model that NFP's national office exists specifically to replicate - likely improves later life outcomes for children, across a variety of regions and population profiles. This opinion is confirmed by a literature review of home visitation programs (see Attachment B-1 Pg 2), which points to NFP as a standout among such programs in terms of the methodological strength and encouraging implications of its studies. Formal studies of the NFP model The first trial of the NFP model, in Elmira County, NY, followed families for 15 years after birth and found statistically significant effects on children’s disciplinary records (though few differences in their reported behavior, such as sex and drug use) (Attachment B-4 Pg 1241). The second, in Memphis, has so far published 3-, 6-, and 9-year follow-ups (and found the children of visited mothers with low economic and psychological resources to have superior scores on a variety of tests including vocabulary, arithmetic, and a mental processing composite at age 6 (Attachment B-3 Pgs 1554, 1556) and higher GPA and academic achievement at age 9 (Attachment B-7 Pg 838). The third, in Denver, published the results of its 4-year followup in 2004, and found the children of visited mothers with low economic and psychological resources to have superior scores on a variety of tests including language and behavioral adaptation in testing (Attachment B-6 Pg 1565-1566). We complied a summary of all results in this table. Other studies of similar programs have found much weaker results; the hypothesis we have seen advanced in the literature is that NFP is distinctive partly for its insistence that registered nurses conduct the visits (Attachment B-1 Pg 40). The Denver study of NFP provides preliminary direct evidence for this, by not only comparing NFP to "no treatment," but also looking at an "in-between" option in which an NFP-like program was carried out by paraprofessionals rather than registered nurses (Attachment B-6 Pg 1563). While both appeared to have some positive effects on both mothers and children, paraprofessional effects were reported to be about half as large (Attachment B-5 Pgs 492-493). On one hand, it is important to emphasize that the effects found by these studies are often small in aggregate. In many cases, the studies look at many variables and find a few pointing strongly in the direction of the treated group, with the remaining variables showing no statistically significant differences. On the other hand, we feel it is impressive to see even these effects given the nature of the evaluations: examining the impact of a relatively low-intensity program, anywhere from [[2]] to [[15]] years after program ends, on a variety of different populations. Replication There is reason for caution in extrapolating from the results of these studies to the expected impact of the NFP program as a whole; the question is how well regional programs, working with the national office's help, can replicate the most important aspects of the program. We are optimistic for the following reasons:
  • NFP's national office was established with direct involvement from David Olds, the lead researcher on all of the above studies, specifically to replicate the program under discussion ([[History]]).
  • The national office has clear, measurable criteria for adherence to the program (see [[Fidelity attachment]]), including the requirement that registered nurses carry out the program according to the (proprietary) NFP curriculum.
  • The national office is extremely thorough in its data collection. It records information from each regional partner on demographics; attrition; implementation (number and content of visits); and a broad range of outcomes (maternal smoking/alcohol consumption, life situation, children’s test scores, etc.) ([[Data collection example]]). It provided us with a sample report containing all of this data, and national aggregates; it was not able to provide us site-by-site data due to confidentiality issues ([[Data collection examples]]).
  • As shown in the studies above, empirical evidence for the model's effectiveness holds across very different populations.
Costs NFP’s website puts the cost of the program at a total of $10,000 per family, for the complete 2.75-year program ([[Attachment C-1 Pg 1]]). Its sample startup budget ([[Attachment D-1]]) is around $500,000 per year for 100 families, which implies a total cost of about $14,000 for the full program. Note that most of these costs are paid by the individual agencies; the role of the national organization is to provide training, support, and oversight, and its costs are negligible next to regional costs: around $10 million for 75,000 families next year ([[Attachment D-2]]). The Organization Financials. The following data is from NFP's 2003-2005 IRS Form 990s (available via GuideStar) and Attachment D-2.
YearRevenues (in thousands)Expenses (in thousands)
2004$2,035$1,439
2005$5,562$3,162
2006$2,251$5,594
2007$12,797$6,840
2008 (exp)$11,114$9,951
The financials we were sent do not include statements of assets, so we tried to get an idea of how much NFP holds by looking at the Form 990 for 2006, the last one we have access to. At that time, NFP held $495,381 in assets. This is low in relation to that particular year's expenditures ($5,594,000); however, we don't think that such a low level of assets is characteristic of the organization, given that this number was calculated during a period of growth in which expenses were out-pacing revenues. We further expect that the NFP's assets have since increased. Since NFP is a not-for-profit, its projected $7,120,000 in surplus revenues for 2007 and 2008 (see table above) must be held in assets. Even assuming high levels of depreciation in the organization's assets, we should be safe in assuming that NFP currently holds at least $3,420,000 (50% of its 2007 expenditures) in assets, and will hold at least $4,975,500 (50% of its 2008 expenditures) in assets by the end of 2008. Board of directors. Half of NFP's ten member board of directors come from high-level positions in the private sector. The remaining directors have a mix of academia, not-for-profit, and public sector backgrounds. Conclusions The NFP model has been shown to positively affect the lives of participating mothers and their children in several rigorous studies, and grants to the national NFP organization are directed towards replicating this proven program. This organization stands out from any other we have examined, in any cause, for its clarity of strategy, commitment to replicating what is proven, and ability to continue learning about what works (both through ongoing randomized trials and through the data collected from all its sites). We highly recommend it as a proven and scalable early childhood intervention.

Attachments:

A. Application and response:

B. Program related attachments:

C. Organization related attachments:

D. Financials:

Applicants

How we found applicants describes the process by which we invited organizations to apply for grants in all five of our causes. We invited 107 to apply within our international causes (saving lives and global poverty). 59 completed our Round 1 application; 48 did not apply. Full details of how we found applicants, for all five of our causes, are available here.

Overview: Global poverty (focus on Africa)

[Print this page] - [Discuss this page]

According to the 2007 UN Human Development Report, 2.6 billion people - 40% of the world's population - live on less than US$2 per day, and 1 billion lived on less than $1 a day ("at the margins of survival" - Pg 25).

While our saving lives cause emphasizes health, for this cause we looked for organizations that specifically focus on helping people in extreme poverty to directly and sustainably improve their income, job situation, and general standard of living. We examined both microfinance - aid through financial services - and more traditional economic assistance programs focused on training. Our main finding was that large organizations in this area are generally unwilling or unable to share detailed empirical information about the people they serve, the services they provide, and their programs' effects on clients' standards of living. Given the complexity of this goal - bringing about lasting changes in the lives of struggling people from other cultures - we do not have strong confidence in any program we've seen to date.

While no economic empowerment program has been widely documented as effective, we are more confident in microfinance than other interventions, knowing what we know. Microfinance is a relatively simple intervention that aims to help people manage their own lives (by providing financial services such as loans, savings, and insurance), rather than prescribing particular activities for them; and the literature on its effectiveness provides stronger and more encouraging evidence (although still not enough to give us great confidence) than the literature we've found on any other intervention in this area. Our full review of the logical and empirical case for microfinance is here.

We therefore awarded our $25,000 grant to Opportunity International, a microfinance program that provided significantly more information on the details of its activities than any other organization we've examined.

We hope to study this cause more in the future; we find the idea of economic assistance for the extremely poor to be highly appealing, but so far we have not found an organization that we can be highly confident in.

Reports

Round 1 materials for all applicants

Nurse-Family Partnership (NFP)

[Print this page] - [Discuss this page]

In a nutshell:

What do they do? The Nurse-Family Partnership supports programs offering home visits (from trained registered nurses) to low-income, first-time mothers. Nurses advise mothers on prenatal health, child care, and personal life management (including birth spacing and employment counseling).

Does it work? The program has been the subject of several formal, rigorous evaluations that imply measurable impacts on both mothers (particularly on how long they go between births) and children (particular cognitive and behavioral measures). NFP stands above any other developed-world organization we know of in terms of its commitment to replicating a proven model.

What do you get for your dollar? We estimate the cost of the full 2.4-year program as ranging from $10,000-$12,000 per family. We do not have the data to say how often the program makes a significant difference in a particular child's life, but across large numbers of children, a statistical difference is noticeable.

The Details:

Table of Contents

What do they do?

The NFP model consists of having trained registered nurses visit low-income, first-time mothers – starting with weekly visits early in pregnancy, and progressing toward monthly visits until the child's second birthday – attempting to help with prenatal health (nutrition; reducing alcohol/tobacco/drug use during pregnancy; obtaining prenatal care), child care (creating a safe and supportive home environment), and personal life management (birth spacing as well as taking steps toward education and employment). A summary of topics covered, as well as frequency of visits, is available at the Home Visit Experience page on NFP's website.

NFP's national office primarily provides a consulting (as opposed to funding) role, partnering with regional organizations in order to carry out its model. Attachment B-13 lays out the national office's activities, along with the associated 2007 expenses (which we cite below as a proportion of the organization's total expenses):

  • Program development (19%): helping regional organizations raise funding.
  • Nursing practice (12%): training and supporting participating nurses.
  • Program quality support (28%): ongoing evaluation and consultation with regional programs.
  • Federal policy & program finance (5%): advocacy for more government funding of programs on the NFP model.
  • Other (37%): marketing, administrative and management costs; a small amount of direct funding for regional programs.

NFP notes that individuals or communities interested in learning more about the feasibility of starting NFP for families in your city should call NFP at 1-800-864-5226.

Does it work?

The primary evidence for the NFP model's effectiveness consists of three different studies - each of which has been reported on several times - in which low-income mothers were (or were not) assigned nurses based on lottery, and then researchers followed both those who did and did not receive nurse visits, collecting a broad range of data. We find the evidence from these studies to provide a compelling case that the model in question - a model that NFP's national office exists specifically to replicate - likely improves later life outcomes for children, across a variety of regions and population profiles. A thorough literature review of home visitation programs (see Gomby 2005 Pg 2) points to NFP as a standout among such programs in terms of the methodological strength and encouraging implications of its studies.

Formal studies of the NFP model

The first trial of the NFP model, in Elmira County, NY, followed families for 15 years after birth and found statistically significant effects on children’s disciplinary records (though few differences in their reported behavior, such as sex and drug use) (Olds 1998 Pg 1241). The second, in Memphis, has so far published 3-, 6-, and 9-year follow-ups (and found the children of visited mothers with low economic and psychological resources to have superior scores on a variety of tests including vocabulary, arithmetic, and a mental processing composite at age 6 (Olds and Kitzman 2004 Pg 1554, 1556) and higher GPA and academic achievement at age 9 (Olds 2007 Pg 838). The third, in Denver, published the results of its 4-year follow-up in 2004, and found the children of visited mothers with low economic and psychological resources to have superior scores on a variety of tests including language and behavioral adaptation in testing (Olds, Robinson and Pettitt 2004 Pg 1565-1566). We compiled a summary of all results in this table.

Other studies of home-visit programs have found much weaker results; the hypothesis we have seen advanced in the literature is that NFP is distinctive partly for its insistence that registered nurses conduct the visits (Gomby 2005 Pg 40). The Denver study of NFP provides preliminary direct evidence for this, by not only comparing NFP to "no treatment," but also looking at an "in-between" option in which an NFP-like program was carried out by paraprofessionals rather than registered nurses (Olds, Robinson and Pettitt Pg 1563). While both appeared to have some positive effects on both mothers and children, paraprofessional effects were reported to be about half as large (Olds 2002 Pg 492-493).

On one hand, it is important to emphasize that the effects found by these studies are often small in aggregate. In many cases, the studies look at many variables and find a few pointing strongly in the direction of the treated group, with the remaining variables showing no statistically significant differences. On the other hand, we feel it is impressive to see even these effects given the nature of the evaluations: examining the impact of a relatively low-intensity program, anywhere from 2 to 15 years after program ends, on a variety of different populations.

Replication

There is reason for caution in extrapolating from the results of these studies to the expected impact of the NFP program as a whole; the question is how well regional programs, working with the national office's help, can replicate the most important aspects of the program. We cannot be fully confident about the quality of replication, but are cautiously optimistic for the following reasons:

  • NFP's national office was established with direct involvement from David Olds, the lead researcher on all of the above studies, specifically to replicate the program under discussion (see NFP's website).
  • The national office has clear, measurable criteria for adherence to the program, including the requirement that registered nurses carry out the program according to the (proprietary) NFP curriculum. (Unfortunately, Attachment B-14, which gives a good sense of these criteria, is currently confidential; a brief overview is available at this page on NFP’s website.)
  • The national office appears thorough in its data collection, although it unfortunately keeps much of this information confidential. We were sent a sample report (Attachment B-12, not cleared for public release) with both national aggregate data and data for an anonymous local agency on demographics; attrition; implementation (number and content of visits); and a broad range of outcomes (maternal smoking/alcohol consumption, life situation, measures of children's development, etc.) NFP has told us that a similar data set, at least at the national level, may be cleared for public release by September 2008; in the meantime, it has cleared Attachments B-17 and B-18 for public release, though these do not include as much data as the confidential Attachment B-12.
  • As shown in the studies above, empirical evidence for the model's effectiveness holds across very different populations.

Costs

NFP puts the cost of the program at a total of $10,000 per family, for the complete 2.4-year program (Attachment B-8). Its sample startup budget is around $500,000 per year for 100 families (this sample budget, Attachment B-15, is unfortunately not yet cleared for public release), which implies a total cost of about $12,000 for the full program. Note that most of these costs are paid by the individual agencies. The role of the national organization is to provide training, support, and oversight, and its costs are negligible next to regional costs: around $10 million (Attachment D-1) to serve the implementing agencies, which together serve 14,800 enrolled families as of 5/31/2008 (Attachment C-1 Pg 2).

The Organization

Financials. The following data is from NFP's 2003-2005 IRS Form 990s (available via GuideStar) and Attachment D-2.

Year Revenues (in thousands) Expenses (in thousands)
2004 $2,035 $1,439
2005 $5,562 $3,162
2006 $2,251 $5,594
2007 $12,797 $6,840
2008 (exp) $11,114 $9,951

The financials we were sent do not include statements of assets, so we tried to get an idea of how much NFP holds by looking at the Form 990 for 2006, the last one we have access to. At that time, NFP held $495,381 in assets. This is low in relation to that particular year's expenditures ($5,594,000); however, we don't think that such a low level of assets is characteristic of the organization, given that this number was calculated during a period of growth in which expenses were out-pacing revenues.

We further expect that the NFP's assets have since increased. Since NFP is a not-for-profit, its projected $7,120,000 in surplus revenues for 2007 and 2008 (see table above) must be held in assets. Even assuming high levels of depreciation in the organization's assets, we should be safe in assuming that NFP currently holds at least $3,420,000 (50% of its 2007 expenditures) in assets, and will hold at least $4,975,500 (50% of its 2008 expenditures) in assets by the end of 2008.

Board of directors. Half of NFP's ten-member board of directors come from high-level positions in the private sector. The remaining directors have a mix of academia, not-for-profit, and public sector backgrounds.

Conclusions

The NFP model has been shown to positively affect the lives of participating mothers and their children in several rigorous studies, and grants to the national NFP organization are directed towards replicating this proven program. This organization stands out from any other we have examined, in any cause, for its clarity of strategy, commitment to replicating what is proven, and ability to continue learning about what works (both through ongoing randomized trials and through the data collected from all its sites). We would like to see NFP share more of its materials publicly, but we nonetheless strongly recommend it as a proven and scalable early childhood intervention.

References (academic)

Elmira Trial

  • Olds DL, Henderson CR Jr, Kitzman H. Does prenatal and infancy nurse home visitation have enduring effects on qualities of parental caregiving and child health at 25 to 50 months of life? Pediatrics. 1994 Jan;93(1):89-98.
  • Olds DL, Eckenrode J, Henderson CR Jr, Kitzman H, Powers J, Cole R, Sidora K, Morris P, Pettitt LM, Luckey D. Long-term effects of home visitation on maternal life course and child abuse and neglect. Fifteen-year follow-up of a randomized trial. JAMA. 1997 Aug 27;278(8):637-43.
  • Olds D, Henderson CR Jr, Cole R, Eckenrode J, Kitzman H, Luckey D, Pettitt L, Sidora K, Morris P, Powers J. Long-term effects of nurse home visitation on children's criminal and antisocial behavior: 15-year follow-up of a randomized controlled trial. JAMA. 1998 Oct 14;280(14):1238-44. Available online.

Memphis Trial

  • Kitzman H, Olds DL, Henderson CR Jr, Hanks C, Cole R, Tatelbaum R, McConnochie KM, Sidora K, Luckey DW, Shaver D, Engelhardt K, James D, Barnard K. Effect of prenatal and infancy home visitation by nurses on pregnancy outcomes, childhood injuries, and repeated childbearing. A randomized controlled trial. JAMA. 1997 Aug 27;278(8):644-52.
  • Kitzman H, Olds DL, Sidora K, Henderson CR Jr, Hanks C, Cole R, Luckey DW, Bondy J, Cole K, Glazner J. Enduring effects of nurse home visitation on maternal life course: a 3-year follow-up of a randomized trial. JAMA. 2000 Apr 19;283(15):1983-9. Available online.
  • Olds DL, Kitzman H, Cole R, Robinson J, Sidora K, Luckey DW, Henderson CR Jr, Hanks C, Bondy J, Holmberg J. Effects of nurse home-visiting on maternal life course and child development: age 6 follow-up results of a randomized trial. Pediatrics. 2004 Dec;114(6):1550-9. Available online.
  • Olds DL, Kitzman H, Hanks C, Cole R, Anson E, Sidora-Arcoleo K, Luckey DW, Henderson CRJ, Holmberg J, Tutt RA, et al. Effects of nurse home visiting on maternal and child functioning: age-nine follow-up of a randomized trial. Pediatrics 2007;120(4):832-45. Available online.

Denver Trial

  • Olds DL, Robinson J, O'Brien R, Luckey DW, Pettitt LM, Henderson CR Jr, Ng RK, Sheff KL, Korfmacher J, Hiatt S, Talmi A. Home visiting by paraprofessionals and by nurses: a randomized, controlled trial. Pediatrics. 2002 Sep;110(3):486-96. Available online.
  • Olds DL, Robinson J, Pettitt L, Luckey DW, Holmberg J, Ng RK, Isacks K, Sheff K, Henderson CR Jr. Effects of home visits by paraprofessionals and by nurses: age 4 follow-up results of a randomized trial. Pediatrics. 2004 Dec;114(6):1560-8. Available online.

Misc.

  • Gomby D. Home visitation 2005: outcomes for children and parents. Committee for Economic Development, Invest in Kids Working Group. 2005 Jul. Available online.

Attachments

A. Application and response

B. Program related attachments

  • Attachment B-1: Gomby 2005 Literature Review
  • Attachment B-2: Kitzman 2000 Memphis Paper
  • Attachment B-3: Kitzman 2003 Memphis Paper
  • Attachment B-4: Olds 1998 Elmira Paper
  • Attachment B-5: Olds 2002 Denver Paper
  • Attachment B-6: Olds 2004 Denver Paper
  • Attachment B-7: Olds 2007 Memphis Paper
  • Attachment B-8: Cost-Benefit Analysis
  • Attachment B-9: Data Collection and Reporting Summary (currently classified as confidential by NFP; may later be released)
  • Attachment B-10: Brief History of NFP
  • Attachment B-11: Numbers and Locations of Nurse Visits 2003 - 2007 (currently classified as confidential by NFP; may later be released)
  • Attachment B-12: Sample Quarterly Nurse Visit Data Tables (currently classified as confidential by NFP; may later be released)
  • Attachment B-13: National Office Program Descriptions and Budgets
  • Attachment B-14: Example Implementing Agency Evaluation Report (currently classified as confidential by NFP; may later be released)
  • Attachment B-15: 2008 Sample Budget for Implementing Agency (currently classified as confidential by NFP; may later be released)
  • Attachment B-16: Description of Implementing Agency Costs (currently classified as confidential by NFP; may later be released)
  • Attachment B-17: Sample Report I
  • Attachment B-18: Sample Report II
  • Attachment B-19: Public statement on NFP Curriculum

C. Organization related attachments

D. Financials

Opportunities Industrialization Centers International

In a nutshell

OICI implements a diverse set of programs aimed at improving health and reducing poverty in the developing-world. In trying to examine the organization as a whole, we are unable to gain confidence in a large enough portion of its activities, and therefore cannot confidently recommend the organization.

The Details

What do they do? OICI implements a variety of public health and economic development programs in the developing world, including:
  1. Agricultural training, i.e., providing farmers with technology and training aimed at improving their output.
  2. Entrepreneurship training for self-employed business owners aiming to improve their ability to support themselves.
  3. Child health and nutrition programs, often based on providing food packages directly.
  4. Water and sanitation programs, including well construction and hygiene training.
OICI submitted a full report of its activities for 2006 (Attachment B-1). We have very little sense of the organization's high-level strategy in picking and prioritizing programs. The table below provides a complete accounting of what we know about these activities; the total cost of each project comes from OICI’s 2006 IRS Form 990 (available via GuideStar). Numbers in the “Activities” column refer to the four activities listed above.
Location Activities 2006 costSource
Ghana 1, 2, 3, 4 & focus on well-being of those suffering from HIV/AIDS$2,890,290Attachment B-1 Pgs 9-25
Guinea 1, 3 & building roads and bridges$999,036Attachment B-1 Pgs 27-37
Cote D'Ivoire 1, 2, 3$903,781Attachment B-1 Pgs 89-105
Nigeria Provides access to microcredit, trains youth and connects workers to potential employers$883,803Attachment B-1 Pgs 61-71
Ghana, Guinea, Mali, and Nigeria 1$583,622Attachment B-1 Pgs 39-53
Togo 1, 2, 4$374,270Attachment B-1 Pgs 107-129
Cote D'Ivoire, Ghana, Guinea, Togo Training for OICI staff on the ground$277,224Attachment B-1 Pgs 55-59
US Encourage American youth to pursue careers in international development$121,482Attachment B-1 Pgs 83-87
Ethiopia Youth education$95,940Attachment B-1 Pgs 73-81
Ethiopia Employment skills and reproductive health training$71,184Attachment B-1 Pgs 131-153
Does it work? Many of the programs above are focused on training, and thus on changing behavior, which we would believe is far from straightforward (especially when dealing with another culture). Given this concern, we would require significant empirical evidence to have high confidence in OICI - evidence that convincingly demonstrates not just what programs were carried out, but how behavior and outcomes (such as income, standard of living, etc.) changed. OICI provided some data on changes in incomes and measures of nutrition, but this data was neither broad enough (i.e., covering many programs) nor compelling enough (i.e., ruling out alternate hypotheses for observed changes - see our overview of microfinance research for a discussion of many potential problems with simple outcomes data) to give us confidence regarding OICI's effect on its clients. Conclusion Ultimately, we cannot confidently recommend OICI because we have too little information about the organization as a whole: we have neither comprehensive evidence on outcomes, nor an overall view of the organization's strategy. While many of OICI’s programs are intuitively attractive – providing comprehensive economic and health aid to a village – we have little sense of what to expect from this organization if and when it brings in more donations. Attachments A. Application materials B. Program related materials C. Organization related materials D. Financials

Opportunity International

In a nutshell

What do they do? Opportunity International is a microfinance organization that provides financial services to low-income people in the developing world (close to 30 countries in Africa, Eastern Europe, Latin America, and Asia). Does it work? We have little to go on besides our independent analysis of microfinance in general; our knowledge of Opportunity International's high overall repayment rate (one of the conditions we feel is important for microfinance programs, as explained in our independent analysis); and more detail on one of Opportunity International's programs in Mozambique. Based on this information, we would guess - without much confidence - that Opportunity International's programs are generally improving lives.

The Details

What do they do? Opportunity International seeks to improve the economic well-being of low-income people in the developing-world by providing financial services: loans, savings accounts, and insurance. We have only the most general information on their programs as a whole (see Attachment B-4 Pg 6-7, which gives an overview of these three different types of services). Below we give an overview of the one region, Mozambique, that we were able to obtain more detailed information on. Loans program (Mozambique) The information below comes from Opportunity International’s “Credit Policy, Guidelines and Procedures” documents for its bank in Mozambique (Attachment B-3). Unless otherwise noted, our statements refer to these guidelines (i.e., they are statements about Opportunity International's stated principles, not reports about actual execution). Opportunity International provides three types of loans:
  • A trust bank loan is a small loan to a group of 8-30 people (Attachment B-3 Pg 6). The group approves the loan and its use, which is subsequently monitored by a bank officer (Attachment B-3 Pg 7), and the group shares collective responsibility for repayment. All members of the group must contribute 10% of loan funds to a Loan Guarantee Fund, which serves simultaneously as funds to support loans that aren’t repaid and as a savings vehicle for group members, which they can withdraw from at the end of a loan cycle when all debts have been repaid (Attachment B-3 Pg 7). The minimum loan is $20 per member and the maximum is $6,000 per group (Attachment B-6 Pg 1). In the first half of 2007, the average loan size was $107 (Attachment B-2). The loan term is 12-27 weeks with an annualized interest rate of 72% (Attachment B-6 Pg 1).
  • A solidarity group business loan is a larger loan (Attachment A-1 Pg 2) made to a group of 3-8 people (Attachment B-3 Pg 9). The minimum loan is $20 per member and the maximum is $2,000 per group (Attachment B-6 Pg 1). In the first half of 2007, the average loan size was $146 (Attachment B-2). The loan term is 12-35 weeks with an annualized interest rate of 72% (Attachment B-6 Pg 1).
  • An individual business loan is a larger loan, made to an individual who can provide a guarantor or collateral for the loan (Attachment B-3 Pg 13). The loan must be used to expand the borrower’s business (Attachment B-3 Pg 13). A loan officer monitors the use of the loan at the borrower’s business or residence to ensure that the loan is used as it was intended (Attachment B-3 Pg 14). Loan sizes are $200-80,000 (Attachment B-6 Pg 1). In the first half of 2007, the average loan size was $617 (Attachment B-2). The loan term is 4-12 months with an annualized interest rate of 66% (Attachment B-6 Pg 1).
To be eligible for any of these loans, an applicant must meet a set of requirements outlined in Attachment B-3 Pg 4. These requirements include age restrictions (applicants must be between 18 and 65 years old) and a requirement that applicants be able to own and operate a business. In addition, although Opportunity International has no formal gender criteria, it reports that 85% of its clients are women (Attachment B-4 Pg 4). Savings Opportunity International provides two types of savings accounts to its clients (Attachment B-5 Pg 1):
  • Transaction accounts require $4 to open, yield no interest, and charge a monthly maintenance fee of $0.04. The first four withdrawals each month are free, but clients must pay $0.40 for each subsequent withdrawal. Clients are also charged a fee of $0.40 each time the account’s balance falls below $4 (the minimum) and must pay $2 to close the account.
  • Savings accounts require $4 to open, yield 0%-9% interest (depending on the amount deposited), and charge a monthly maintenance fee of $0.04. The first withdrawal each month is free, but clients must pay $0.40 for each subsequent withdrawal. Clients are also charged a fee of $0.40 each time the account’s balance falls below $4 (the minimum) and must pay $2 to close the account.
Does it work? We believe, from our analysis of microfinance in general, that a microfinance program is likely to improve lives when it is serving people who face considerable economic uncertainty, lack access to traditional financial services, and are willing and able to "pay" for financial services (for example, repaying loans with interest). Opportunity International reports a 98% repayment rate on loans made to Trust Groups (Attachment B-4 Pg 6), implying that the third of these conditions is generally being met. We have little other substantive information on its clients in general, although it did give us a more detailed picture of its operations in Mozambique (see especially Attachments B-1 and B-2). Note that most of its clients in Mozambique are generally under an "extreme poverty line" (US $1/day) that, according to one independent study, is often connected with economic uncertainty and lack of access to traditional financial services (these issues are discussed more in our general overview of microfinance). Knowing what we know, we would guess that Opportunity International is generally improving its clients' lives, but would need far more detailed information on its operations around the world to have confidence. The Organization Financials. The following is from Opportunity International’s 2004-2006 IRS Form 990s (available via Guidestar) and Attachment D-3.
YearRevenues (in thousands)Expenses (in thousands)
2004$28,250$25,363
2005$24,440$22,645
2006$45,201$29,627
2007 (est)$37,450$35,825
As of the end of 2006, Opportunity International held $109 million in assets, the equivalent of approximately 2.5 years of operating expenses (Attachment D-2 Pg 2). This constitutes a large increase over its 2005 assets ($33 million) is likely due to the fact that revenue grew much faster than expenses that year. In 2007, expenses rose to where they are nearly in line with revenues. Board of directors.The names of Opportunity International's Board of Directors are available in Attachment B-4 Pg 19. However, Opportunity International did not provide information on their affiliations. Conclusions We chose to award Opportunity International because:
  • Without strong evidence on the effectiveness of any economic empowerment program, we chose to award an organization working in microfinance. Our general analysis of microfinance implies that it is a helpful intervention; the evidence is not strong, but it is stronger than the case for other programs we're aware of.
  • Opportunity International was the only microfinance organization to give us a detailed picture of even one of its regions (Mozambique in this case). That said, we have not been able to get a clear bird's-eye view of the organization's operations and programmatic finances (although we do have standard financial information of the form provided in auditors' reports and Form 990s).
For a donor committed to economic empowerment programs, we recommend Opportunity International with major reservations. We hope in the future to gain a far more detailed understanding of the organization (or other organizations doing similar work). Attachments A. Application and response B. Program related attachments C. Organization related attachments D. Financials

KickStart

In a nutshell

KickStart develops and markets products aimed at improving production and reducing poverty in the developing world. These products include irrigation pumps that can provide water for crops during dry season; the aim is to improve farm production, improving farmers' incomes as well as increasing the supply of food locally. Unfortunately, we do not have enough detail and empirical evidence to be confident in KickStart's effectiveness.

The Details

What do they do? KickStart produces, markets, and (at subsidized prices) sells technology aimed at improving people's lives in sub-Saharan Africa (Attachment A-1 Pg 1). Their application focuses on human-powered irrigation pumps, which we believe to constitute the majority of their operations (Attachment A-2 Pgs 4-6). The "MoneyMaker Irrigation Pump" a human-powered pump capable of irrigating up to two acres of land with water gathered from existing sources such as ponds, rivers, and wells (Attachment A-1 Pgs 2-3). The aim is to give farmers year-round irrigation, allowing them to grow water-intensive high-value crops like fruits and vegetables, optimize watering volumes for any crop, and produce output during the dry season (Attachment A-1 Pg 2). Increased production then hopefully leads to higher and more stable farm revenues, as well as lower and more stable food prices. Note that three of the countries in which KickStart markets this product have dry seasons lasting for about half the year (according to NOAA's seasonal rainfall analysis - see rainfall analysis for Tanzania, Kenya, and Mali). While the most of production, distribution, and marketing costs for KickStart's irrigation systems are subsidized by donors, clients pay a significant portion of the total cost (between $35 and $95) for each system (Attachment A-1 Pg 3). So far, 82,000 pumps have been sold, 40,000 of those in Kenya (Attachment A-1 Pg 3). Does it work? We have relatively little hard evidence on KickStart products' impacts on life outcomes. Attachment B-1 includes a summary of a client survey KickStart conducted, finding major improvements over time in income, ownership of land and livestock, and social status (Pgs 40-41); however, KickStart notes that the set of people surveyed was quite small and not geographically representative (Pg 5). KickStart states elsewhere (Attachment A-1 Pg 3) that "The average net farm income of the farmers in Kenya using our pumps increases by ten fold – from about $100 per year to over $1,000 per year, while total family incomes more than double (Virtually all families have some other income source: 1 in 3 adults is also involved in some kind of informal sector employment)." However, we have been unable to get more documentation of the data behind this statement, aside from a general description (Attachment A-2 Pg 6) of how data is collected. Much as with microfinance, we find KickStart's products to be fairly logical and intuitive ways of helping people, if and when they are marketed to the people most likely to benefit from them (in the case of the MoneyMaker pump, farmers who cannot irrigate during the dry season, could do so with better technology, and would benefit significantly from year-round irrigation). However, without strong empirical evidence about life outcomes - or clear information about where (and to whom) different products are sold - we have only limited confidence. Conclusion As with microfinance, we believe that KickStart's activities logically have enormous potential to help improve people's lives, but we lack the detail and empirical evidence to have high confidence. Knowing what we know, we are very slightly more confident in the approach of microfinance programs, which take a more general and flexible approach (providing financial services to help people manage their own lives, rather than supporting a specific activity) and have slightly more empirical evidence behind them. Attachments A. Application materials B. Program related attachments C. Organization related attachments D. Financials

The LEDA Scholars program

In a nutshell

What do they do? The LEDA Scholars Program is a college preparatory program serving economically disadvantaged high school students. The program consists of a full-time, seven-week summer program (following 11th grade) and college advising and assistance during 12th grade. Applicants to LEDA's program undergo a rigorous application process that includes academic evaluation, multiple recommendations, and an interview. Does it work? We aren't confident that LEDA is significantly impacting its students, i.e., causing them to perform better than they would without its help. The only empirical study we have access to is highly ambiguous. It follows a group of LEDA scholars as well as a group of students who were rejected from the program in the final round, and finds that the former matriculated more frequently (and to more selective schools); but we believe that those who gain acceptance into the program are likely much better off to begin with than those who are rejected, and it is not intuitively or empirically clear to us that LEDA is adding significantly to these already accomplished students' opportunities.

The details

What do they do? Applicants to LEDA's program undergo a highly selective application process; those who are selected go through LEDA's full-time, seven-week summer program (following 11th grade) and receive college advising and assistance during 12th grade. Recruitment and selection of scholars LEDA targets students from minority (African-American, Latino, and Native American) backgrounds (Attachment A-2 Pg 1) and those who come from relatively low-income families (families of the first LEDA cohort had an average income of $40,000 per year, as stated in Attachment B-9). Over the 4 year history of the program, LEDA has served 250 students, 120 of whom have come from New York City (Attachment A-2 Pg 1). Applicants to LEDA participate in a rigorous application process, which assesses students' academic records (grades and difficulty of courses taken), writing ability (through one essay prepared specifically for LEDA and three additional writing samples), four recommendations, and interviews with applicants and their parents (Attachment A-2 Pg 1-2). The program In the summer preceding scholars' senior year of high school, LEDA holds a seven-week program at Princeton University. The program consists largely of college-style academic work that aims to prepare scholars academically and socially for college (Attachment A-2 Pg 3). In addition, the program offers student-specific guidance to help scholars prepare for the college application process (Attachment A-2 Pg 3). During students' senior year of high school, LEDA provides college admissions assistance, including help navigating the financial aid process (Attachment A-2 Pg 3) and direct advocacy of scholars to admissions and financial aid officers (Attachment A-2 Pg 3). Does it work? Because LEDA has a selective admissions process, we believe its scholars may be students who are already positioned to succeed in the college admissions process; the question is how much better they do with LEDA's help than they would without it. We have little to go on, either intuitively or empirically, to answer this question. LEDA provided us with a study (Attachment B-1) that attempts to gauge impact by following both the students who went through its program and a "control group" comprised of students who were denied admission in the final stage of the process. The former group enrolled more frequently in college (and at more selective schools) than the latter:
ComparisonLEDA scholarsControl group
% in college96%93%
% in selective college (Top 145 schools acc. to Kahlenberg, 2004)95%26%
% in Ivy League schools41%2%
If these two groups had been largely similar to begin with (i.e., prior to their interaction with LEDA), such a difference might imply that LEDA had a positive impact. However, we believe that the two groups are not comparable: LEDA scholars were significantly more academically accomplished than control group students, before ever enrolling in LEDA's program. The table below illustrates this by comparing background characteristics of LEDA scholars and the control group.
ComparisonLEDA scholarsControl group
SAT12521137
PSAT (SAT equivalent scale)12141067
Rank817
Rank %2.8%6.2%
Family income$40,084$43,884
LEDA scholars had noticeably higher SAT and PSAT scores, and higher class rank. These differences almost certainly cannot be attributed to the LEDA program's effects: students take the PSAT in the fall of their junior year of high school, and the difference between LEDA scholars and the control group is noticeable by this measure. We find it highly plausible that the superior performance of the LEDA students in question (shown in the first table) was a function of their background characteristics (shown in the second table), and not of the LEDA program itself. Conclusion Although LEDA's scholars are successful in gaining admission to selective colleges, we are not confident that much of this success can be attributed to LEDA, as opposed to scholars' existing academic credentials. We would need to see a more compelling empirical case in order to consider this program a proven way of helping disadvantaged students. Attachments: A. Application and response B. Program related attachments
  • Attachment B-1: “Affirmative/Quantifiable Action?” – Collins 2007
  • Attachment B-2: Hispanic Initiative Evaluation (Liza’s Study)
  • Attachment B-3: Admitted Students’ schools, zip codes, gender, ethnicity, and income 2004/05 (No Names)
  • Attachment B-4: Admitted Students’ schools, zip codes, gender, ethnicity, and income 2005/06 (No Names)
  • Attachment B-5: Liza’s Study: Control Group Final Graphs
  • Attachment B-6: Liza’s Study: Control Group Admissions Information (No Names)
  • Attachment B-7: Liza’s Study: LEDA’s Latino Students’ Admissions Information (No Names)
  • Attachment B-8: List of LEDA Scholars Current College Enrollment (By School)
  • Attachment B-9: 1st LEDA Cohort’s Performance Data (SAT, GPA etc.) And College Attended (No Names)
  • Attachment B-10: Percentage of 1st Cohort With Family Income > and < $60,000
  • Attachment B-11: Number of 1st Cohort Families In Each Income Quintile
  • Attachment B-12: Percentage of 2nd Cohort With Family Income > and < $60,000
  • Attachment B-13: Number of 2nd Cohort Families In Each Income Quintile
  • Attachment B-14: Number of 1st Cohort Students Eligible for Free Lunch Program
  • Attachment B-15: Number of 2nd Cohort Students Eligible for Free Lunch Program
  • Attachment B-16: Number of 1st Cohort Students Eligible for Reduced Lunch Program
  • Attachment B-17: Number of 2nd Cohort Students Eligible for Reduced Lunch Program
  • Attachment B-18: Description of LEDA’s Recruitment Process
  • Attachment B-19: SAT Composites-#s 3-19-07 [Note: student names removed.]
  • Attachment B-20: Comma Delimited Student Performance Data in 2006 (No Names)
  • Attachment B-21: Comma Delimited Student Performance Data in 2007 (No Names)
  • Attachment B-22: Comma Delimited Student Performance Data in 2008 (No Names)
C. Organization related attachments D. Financial related attachments
Syndicate content