Utilization or Impact?
Recently published 2022 year end appeals have talked so much about impact. We provided after-school services to 10,000 students! We engaged 500 individuals in mental health services! While we can be thankful that these organizations answered the call for the people who reached out to them, these numbers do not speak to impact. They do not speak to how these individuals’ behavior, circumstances, or lives changed as a result of participating in these programs. They simply quantify utilization of those services.
Still other appeals got a bit closer to measuring impact. We placed 1,000 people in jobs! We provided 5,000 people with a food pantry package! These organizations made some impact on the participants’ finances and hunger, respectively, but how much? Were those placed in jobs still working 6 or 12 months later? Were they placed making a livable wage? Did they get a raise to a livable wage during the reporting period? Did those receiving a food package then secure adequate food stamps or new employment?
More and more nonprofits open every year, in large part due to a founder’s belief they can effectuate life improvement better than those already in existence. This means there is more and more competition for philanthropic dollars, and this means impact matters more and more. Those organizations who are truly making impact must receive the lion’s share of philanthropic dollars. We must stop “bandaiding” social ills and instead make progress toward truly eliminating them or at least make measurable improvements.
Despite this clear need for measuring impact, however, nonprofits are hesitant. Some may be worried about working themselves out of business. Others may struggle with how to collect data in a way that informs program design to greatest impact. Finally, as these year end appeals indicate, many do not know what impact is, let alone how to measure it.
Programs must start by developing theories of change. How do participants wish to have their lives changed by engaging in one program or another, and what does an organization believe it is doing that changes participants’ lives in this way? Once a theory is established, an organization must identify an objective, preferably standardized (i.e., tested by others for validity and reliability) way of measuring this theory.
So, for example, perhaps the theory of change in my first example above is that parents enroll their children in an after-school program to ensure homework is completed, which in turn ensures that students pass their classes or increase their test scores on standardized tests. Based on this theory, the organization might collect data on homework completion and then grades or test scores in order to compare their data to those of students who do not participate in the program or to district-wide test score performance. Can you imagine receiving a year end appeal from an organization that can tout that it not only served 10,000 students in its after-school programs, but that 80% of those participants test at or above grade level on standardized reading and math tests compared to only 50% of students across the district? Oh wow, now we’re talking impact! These kids are performing better than their peers and have greater potential for future academic success as a result of their participation in this organization’s after-school programs. Similarly, what if the workforce development program cited above could indicate the percentage of job seekers placed in livable wage jobs who were still in those jobs 12 months later, thereby impacting their bank accounts by some drastic amount of money that would have been otherwise unattainable? Or, those who secure food pantry packages also received food stamps via their participation in this pantry program and only needed to access the pantry once per month versus weekly as a result. You hear the impact?
The needs of every community are ever-increasing. Serving the masses is one thing, making a difference, the literal tag line of so many organizations, is something else entirely.