“I have the report- now what?” 7 Questions to Ask When Reading Program Evaluations

Program evaluations provide answers to a key question: “Did this program work in a particular population and setting?” For funders who haven’t been trained in reading evaluations, wading through such reports can be confusing and time-consuming. So, to get the most out of program evaluations, here are 7 questions to ask when reading them.

Categories: Program Level Learning and Evaluation

Tags: , ,









35 Keys to Effective Evaluation

This tool will help you make sure grant money is spent as effectively as possible and how to evaluate the consequences of grant supported activities  

Categories: Program Level Learning and Evaluation

Tags: , , ,









6 Steps to Evaluating your Program

This tool will help you to create an accurate process for documenting and evaluating the entire effort.

Categories: Program Level Learning and Evaluation

Tags: ,









A Model for Multilevel Advocacy Evaluation

The Colorado Trust provided three years of general operating support to nine advocacy organizations working to increase access to health through policy change work. The nine grantees had a variety of goals and strategies and had different levels of organizational capacity, but were evaluated using a uniform evaluation approach. The evaluation was designed to build grantees’ own evaluation capacity to incorporate real-time feedback, monitor progress toward goals, and to assess growth in the overall health advocacy community in Colorado. Individual grantees identified short- and intermediate- term outcomes related to The Trust’s intermediate outcomes, which were in turn related to the Read more

Categories: Program Level Learning and Evaluation

Tags: ,









A Road Made by Walking: Participatory Evaluation and Social Change

This article describes how participatory evaluation was used in a Ford Foundation–funded project to promote mixed-income housing in Atlanta. The project resulted in an increase in mixed income housing, but also in social outcomes such as increased knowledge about housing issues. Validity and reliability of the findings are demonstrated through feedback from the community members, rather than through statistical methods.

Categories: Program Level Learning and Evaluation

Tags:









Alliance Breakfast Club- Markets for good: removing the barriers

In this video, the participants are discussing why so few market-based solutions to poverty are getting to scale, what can be done so they can deliver meaningful benefits to the poor, and more.

Categories: Program Level Learning and Evaluation

Tags: , ,









Beldon Fund Evaluating Impact

The Beldon Fund relied heavily on independent external evaluations to make early and mid-course corrections to our program strategies and to develop benchmarks to measure progress. EXTERNAL EVALUATIONS Each assessment includes a summary of key findings as well as the full report. Evaluation of Beldon’s Program Strategies PDF A qualitative evaluation, based on confidential interviews, of the impact of our program strategies. Beldon Grantee Perception Report PDF Anonymous survey of Beldon grantees. Beldon Grant Applicant Perception Report PDF Anonymous survey of Beldon grant applicants. EVALUATION BENCHMARKS Evaluating Policy Advocacy Grant-Making Strategies

Categories: Program Level Learning and Evaluation

Tags:









Beldon Fund Final Impact Assessment

This final impact assessment provides an inventory of Beldon’s legacy, five years after its grantmaking ended. It tells the story of a foundation that brought innovative ideas to its grantees, built capacity and infrastructure, and planted seeds that continue to bear fruit. As a spend-out foundation, it struggled with challenges that other spend outs share, and the lessons learned from this assessment are intended to help others embarking on similar paths.

Categories: Program Level Learning and Evaluation

Tags:









Building and Restoring Civic Capacity: The Obama Administration’s Federal-Local Partnership with Detroit

This report focuses on what the partnership contributed to Detroit’s revitalization efforts and how it helped build local capacity. Initiatives and projects detailed in the report in which The Kresge Foundation was involved include the Blight Task Force, the Detroit Home Mortgage program, creation of the city’s global engagement strategy and attracting support for sustainable recreation at Historic Fort Wayne.

Categories: Program Level Learning and Evaluation

Tags: , , , , ,









Building the Capacity of Networks to Achieve Systems Change

Networks have historically played an essential role in promoting progress in areas such as social justice, political reform, environmental protection, and public health. Foundations are increasingly recognizing the power of networks and looking for strategies to help networks achieve their potential. The most common strategies are: a) convene a new network around a mission in line with the foundation’s interests, or b) make grants to an existing network whose interests align with the foundation’s. Each strategy has practical limitations. This paper analyzes an alternative strategy developed by the Mary Reynolds Babcock Foundation (MRBF). In addition to providing networks with grants, Read more

Categories: Program Level Learning and Evaluation

Tags: , , ,









Consumer Voices for Coverage: Advocacy Evaluation Toolkit

The Advocacy Evaluation Toolkit contains the instruments Mathematica used to collect data for evaluating the Consumer Voices for Coverage program. It explains how the instruments were developed, what each was designed to measure, and how Mathematica used them for the evaluation. Although the instruments in the toolkit were designed to collect data for the grant program and reflect its structure and goals, they can be adapted for other situations and uses, ranging from an organization’s informal self-assessment to shape its activities to a comprehensive evaluation. The toolkit suggests some of these adaptations. For people who might not be familiar with evaluation methods, Read more

Categories: Program Level Learning and Evaluation

Tags: , ,









Defining, Building, and Measuring Capacity: Findings From an Advocacy Evaluation

This article reports on results from Mathematica Policy Research’s evaluation of Consumer Voices for Coverage, a program funded by the Robert Wood Johnson Foundation to support the role of consumer health advocacy coalitions in 12 states. The foundation based the program on a study that identified six core advocacy capacities, and designed it to strengthen these capacities. The evaluation found that the level of funding, substantial and targeted technical assistance, and the three-year time frame of the program contributed to the observed increases in five capacities. Fundraising remained the lowest-rated capacity for most of the coalitions and may require different Read more

Categories: Program Level Learning and Evaluation

Tags: , , ,









Demonstrating the Value of Social Service Programs: A Simplified Approach to Calculating Return on Investment

In 2008, as charitable giving dropped by $6.4 billion, 54 percent of human service programs saw an increase in the need for their services. Additionally, 74 percent of programs specifically serving children and youth reported being underfunded or severely underfunded. As government and foundation grantmakers transition from charitable giving to social investment, a Gates Foundation report on eight methodologies to assist measuring social value creation finds the methodologies are many years away from being suitable for both nonprofits and grantmakers. To better recognize and communicate the work of frontline practitioners, there is a need to change the orientation of our Read more

Categories: Program Level Learning and Evaluation

Tags: , ,









Evaluating Complexity

While evaluation has traditionally focused on assessing programmatic impact according to pre-determined indicators, a new approach is needed for evaluating complex initiatives, as well as initiatives operating in complex environments where progress is not linear, predictable, or controllable. 9 propositions can help evaluators navigate the unique characteristics of complex systems, improve their evaluation practice, and better serve the needs of the social sector.

Categories: Program Level Learning and Evaluation

Tags: , , , ,









Evaluating the Kaiser Permanente Community Fund’s Social Determinants of Health Portfolio

Research over the past two decades repeatedly demonstrates the relationship between poor health outcomes and socioeconomic factors such as poor housing, poverty, racism, and structural inequity. In 2005, the Northwest Health Foundation, supported by the Kaiser Permanente Community Fund, began an initiative to address these social determinants of health (SDOH). A variety of projects – short- and long-term, large and small – were supported over the five-year period for a total of $12.4 million. The mean project-implementation grant was $175,350 and 2½ years in length; capacity-building grants averaged $50,000 for 1½ years. In all, 323 social-determinant accomplishments were identified. The Read more

Categories: Program Level Learning and Evaluation

Tags: , , , , ,









Evaluation at Sunset: Considerations When Evaluating a Program as It Concludes

This article uses the evaluation of the Orfalea Foundation’s initiative to provide a case example of a rigorous and useful sunset evaluation, and discusses other possible extensions of these kinds of methods.

Categories: Program Level Learning and Evaluation

Tags: , , ,









Evaluation Basics for Grantmakers

Five Guidebooks for Grantmakers released in 2011, these concise guides provide resources for developing and expanding the use of evaluation and evaluative thinking in Grantmaking organizations. Basic Concepts for Grantmakers Using Logic Models Evaluation Data Collection Evaluative Thinking for Grantmakers Commissioning Evaluation

Categories: Program Level Learning and Evaluation

Tags:









Evaluation Capacity Building: Examples and Lessons from the Field

Innovation Network developed these three introductory evaluation documents as part of Building Nonprofit Capacity to Evaluate, Learn, and Grow Impact, a workshop we presented in partnership with Grantmakers for Effective Organizations’ Scaling What Works initiative.  

Categories: Program Level Learning and Evaluation

Tags: , ,









Evaluation Capacity Building: Funder Initiatives to Strengthen Grantee Evaluation Capacity and Practice

The three case studies presented in this paper share Innovation Network’s experiences with funder initiatives to strengthen grantee evaluation capacity and practice. Each case study includes a description of the overall grantmaking initiative, followed by a discussion of the Evaluation Capacity Building (ECB) services requested by grantees and/or provided by Innovation Network. The paper concludes with a reflection on lessons learned and recommendations for funders considering ECB for their grantees.  

Categories: Program Level Learning and Evaluation

Tags: , ,









Evaluation for Models and Adaptive Initiatives

Although there has been a growing emphasis on use of experimental designs in evaluation, there is also increasing agreement that evaluation designs should be situation specific. The nature of the program is one of the key factors to consider in evaluation design. Two types of programs – models, which provide replicable or semi-standardized solutions, adaptive initiatives, which are flexible programming strategies used to address problems that require unique, context-based solutions – require different evaluation designs. Evaluation of models requires understanding the stage of development of the model program, with summative evaluation done only when the model is fully developed. Adaptive Read more

Categories: Program Level Learning and Evaluation

Tags: , , ,









Evaluation Use in Philanthropy: Using Logic Models

This guide will show you the difference between a Logic Model and the Theory of Change and when it is best to use these tools.

Categories: Program Level Learning and Evaluation

Tags: , ,









Finding the Win in Wicked Problems: Lessons From Evaluating Public Policy Advocacy

Many of the social issues private foundations and other philanthropies attempt to address — poverty, homelessness, global climate change — are wicked problems. That is, they defy easy definition, lack permanent solutions, and have multiple stakeholders. The wicked problems framework helps make explicit the challenging nature of the issue to be addressed, requires an inclusive style of leadership that seeks stakeholder involvement, and demands candid exchange among stakeholders about the nature of the problem and effectiveness of efforts to address it. A wicked problems framework provides a set of criteria and questions for evaluators of advocacy efforts to ask all Read more

Categories: Program Level Learning and Evaluation

Tags: ,









Foundation Evaluation Startup: A Pause for Reflection

This article reports on the accomplishments, challenges, and lessons learned in creating a new Department of Research and Evaluation at the California HealthCare Foundation. Different tools were developed to address each of three key areas: performance assessment, organizational learning, and program evaluation. These new processes and tools have been well received by both staff and the board, and have become increasingly important as resources become more scarce, making understanding and maximizing the impact of investments even more critical. Fostering a culture of evaluative inquiry in a fast-paced, payout-oriented environment is a significant challenge – program staff often feels pressured to Read more

Categories: Program Level Learning and Evaluation

Tags: , , , ,









Foundations as Network Strategists, Weavers, and Managers: Learning From One Foundation’s Journey and Results

This article shares insights from a five-year evaluation of the Oral Health 2020 network, an effort by the DentaQuest Foundation to align and strengthen efforts in service of a national movement to improve oral health. The evaluation helped to place the foundation’s journey in the context of a broader field seeking new approaches to achieve deep and sustainable social change.

Categories: Program Level Learning and Evaluation

Tags: , ,









Getting to Results: A Tool and Lessons from the Annie E. Casey Foundation’s K-12 Education Portfolio

In 2002, the Annie E. Casey Foundation adopted a results-based accountability (RBA) framework to track and report on the results of their philanthropic investments. The RBA tool was piloted in a few program areas, including its K-12 education portfolio. Grantees were highly engaged in an iterative process to determine appropriate measures, refine the theory of change, and how to track progress. Overall, the RBA tool enabled staff to get a sense of how grantees were doing and therefore how the foundation was doing in a way that hadn’t been possible before. The K-12 program got a much clearer sense of Read more

Categories: Program Level Learning and Evaluation

Tags: , , , , ,









Goal-Free Evaluation: An Orientation for Foundations’ Evaluations

Goal-free evaluation (GFE), in program evaluation, is a model in which the official or stated program goals and objectives are withheld or screened from the evaluator. Several obstacles must be overcome in persuading foundations and programs to consider GFE as a viable option, because both tend to view goal attainment as intuitively and inextricably linked to evaluation. This article presents the case for GFE as a perspective that belongs in a foundation’s toolbox. In particular, this article demonstrates GFE’s actual use, highlights aspects of its methodology, and details its potential benefits.

Categories: Program Level Learning and Evaluation

Tags: , ,









Going Deeper: Can Investigative Reporters Add Value to Assessment and Evaluation?

The John S. and James L. Knight Foundation supplemented its standard evaluation approach by engaging professional journalists to elaborate on evaluation findings. The resulting reports are more direct, even critical, than any prior Knight Foundation attempt to evaluate and assess. It produced deeper looks into the intent and outcome of major initiatives, analyzing and addressing flaws in the theories of change underlying initiatives. The goal of reaching external audiences was not achieved.

Categories: Program Level Learning and Evaluation

Tags: , , ,









Grantmaking for Community Impact Project: Summary of Findings in Seven Sites

This infographic is a review of the seven sites researched in NCRP’s Grantmaking for Community Impact Project. It highlights the billions in public benefits secured by 110 organizations in 13 states.

Categories: Program Level Learning and Evaluation

Tags: , , ,









Guidance on Evaluation Reports to the Robert Wood Johnson Foundation: A Checklist for Evaluators

The Robert Wood Johnson Foundation (RWJF) is a long-time proponent of evaluation as a means to inquire systematically into the effects and impacts of its grantmaking programs. Evaluation reports to RWJF should generally include the elements listed in this checklist to be considered satisfactory.

Categories: Program Level Learning and Evaluation

Tags: ,









Is the Policy Win All? A Framework for Effective Social-Justice Advocacy

This article offers a theory-of-change framework to help those engaged in social-justice advocacy to reflect on whether social-justice values are being retained in the process. A reproductive rights effort in South Africa provides an example of how social justice values can be lost in the advocacy process. The failure to sustain work on the ground pointed to the need to maintain a base of support even after a policy victory. Strategies must be revisited as social and political contexts change. One of the critical social-justice values that supports the establishment and maintenance of alliances is collaboration, which must continue to Read more

Categories: Program Level Learning and Evaluation

Tags: ,









Making Measures Work For You: Outcomes and Evaluation

This guide looks at tensions that drive the debate about outcomes measurement, as well as common questions about its potential risks and rewards.

Categories: Program Level Learning and Evaluation

Tags: , , , ,









Mapping Change: Using a Theory of Change to Guide Planning and Evaluation

This brief guide explains why grantmakers use theories of change to guide their questioning, unearth assumptions that underlie their work, establish common language, and develop strong action plans. Contributors to the guide also describe how a theory of change sets the stage for evaluation by clarifying goals, strategies, and milestones.

Categories: Program Level Learning and Evaluation

Tags: , , ,









Measuring the Impacts of Advocacy and Community Organizing: Application of a Methodology and Initial Findings

The increasing emphasis by funders on strategic grantmaking and measurable outcomes may be a disincentive to support policy and advocacy work, because of the perception that outcomes can be difficult to assess. A tool for measuring impact can reduce the barriers to funding advocacy and policy work. The tool draws upon the literature on evaluating advocacy and organizing, social capital building efforts, and return on investment approaches to evaluation. The tool was applied in two sites, where funders found it useful to understand advocacy impacts and learn how advocacy can enhance their grantmaking goals.

Categories: Program Level Learning and Evaluation

Tags: , , ,









Partnership With Government: An Exit Strategy for Philanthropies?

Atlantic Philanthropies’ exit strategy involved a formal partnership arrangement with the Northern Ireland Assembly. This article draws on qualitative data gathered through interviews with key stakeholders — the funder, government officials, and NGOs — and considers the consequences of this approach for sustaining and mainstreaming policies and practices. It also offers both specific and general lessons on partnering with government as an exit strategy.

Categories: Program Level Learning and Evaluation

Tags: , , , , ,









Pathfinder Funder Edition: A Practical Guide to Advocacy Evaluation

Pathfinder is a practical guide to the advocacy evaluation process. This edition guides funders through the advocacy evaluation process from start to finish. Editions for advocates and evaluators are also available. Drawn from Innovation Network’s research and consulting experience, Pathfinder encourages the adoption of a “learning-focused evaluation” approach, which prioritizes using knowledge for improvement.

Categories: Program Level Learning and Evaluation

Tags: , ,









Philanthropy Lessons: The Many Forms of Evaluation

Evaluating impact can involve diverse data points, from tangible measures such as people served and dollars spent to less concrete aspects including lessons learned and challenges faced. Effective philanthropists embrace evaluation, understanding its dynamic and complex nature.

Categories: Program Level Learning and Evaluation

Tags: , ,









The Grantmaker’s Role in Theory of Change

Innovation Network developed these three introductory evaluation documents as part of Building Nonprofit Capacity to Evaluate, Learn, and Grow Impact, a workshop we presented in partnership with Grantmakers for Effective Organizations’ Scaling What Works initiative.

Categories: Program Level Learning and Evaluation

Tags: , ,









The Logic Model Guidebook: Better Strategies for Great Results (Second Ed.): Book Review

The second edition of The Logic Model Guidebook: Better Strategies for Great Results is a straightforward guide, with excellent and varied examples, that achieves its purpose of giving readers a “basic understanding of how to create and use logic models” (p. xii). As enthusiastic champions of logic models, the authors adhere to the assumption that articulating precise and detailed logic models will lead to better results.

Categories: Program Level Learning and Evaluation

Tags: ,









The Real-Time Evaluation Memo: A Tool for Enabling Evaluative Thinking and Learning in Foundations and Nonprofits

Real-time evaluation memos provide data-based feedback in a timely manner to inform decision making. Memos must be concise and include both data and expert synthesis and interpretation. The foundation must have a learning culture if the memos are to most useful; there must be time to reflect on the content and implications. The balance between data quality and timeliness must be managed and will be dependent on the topic. While useful for program management, these memos do not provide the kind of summative information that board members and other stakeholders may require.

Categories: Program Level Learning and Evaluation

Tags: , ,









The Robert Wood Johnson Foundation’s Approach to Evaluation

The Robert Wood Johnson Foundation has placed a high priority on program evaluation since its inception as a national philanthropy in 1972. It has developed a four-tiered system of evaluation that ranges from the evaluation of individual grants and clusters of grants to the qualitative assessments found in The Robert Wood Johnson Foundation Anthology series.

Categories: Program Level Learning and Evaluation

Tags: , , , ,









Tips for Evaluating Advocacy A Checklist for Grantmaking Organizations

This checklist will keep you on track for effective evaluation of advocacy.

Categories: Program Level Learning and Evaluation

Tags: ,









Using Community-Based Participatory Evaluation (CBPE) Methods as a Tool to Sustain a Community Health Coalition

Participatory evaluation has set the standard for cooperation between program evaluators and stakeholders. Coalition evaluation, however, calls for more extensive collaboration with the community at large. Integrating principles of community based participatory research and the Substance Abuse and Mental Health Services Administration’s Strategic Prevention Framework, which guides much coalition work, into coalition evaluation has proved useful to foster community affiliations and support reciprocal relationship building. The resulting evaluation method, named community based participatory evaluation (CBPE), takes time, money, and skilled personnel but can lead to more accurate results and coalition sustainability. The CBPE method has proved essential in sustaining two Read more

Categories: Program Level Learning and Evaluation

Tags: , , ,