Is it ok to measure outputs?
If you work in the social change sector, I’m sure you’re familiar with rhetoric that measuring outputs is bad, insufficient and distracting. In fact you may have seen a list of outputs be instantly dismissed in favour of a much more suitable list of outcomes and intended impact. After all, focusing on outputs is a sure way to incentivise the building of widgets instead of delivery of impact.
But is this true? Are there any contexts in which measuring outputs might actually make sense?
I think so.
Here is my TL-DR: measuring outputs is appropriate when an organisation meets two conditions:
The organisation delivers, what I call, a linear intervention; a specific product or service to address an immediate need; and
The organisation has a probable causal link between their direct intervention and the broader (societal) outcomes they hope to achieve.
The big question is, how do you know whether the second condition is true?
I think, RCTs and other control designs typically answered this question. But as momentum for ‘systems change’ has taken hold, we have been left longing for suitable, rigorous evaluation approaches. An approach that takes into account multi-faceted causal pathways and the actions of many organisations at once.
I’m yet to reach a satisfying conclusion for how to do this. In fact, I think there may be systemic problems with a systems approach (pardon the pun). But, given that I am frequently approached by other funders, peers, and friends who similarly seek to test how to validate a systems approach, I believe this is an area worthy of exploration and patience.
Recently I attended a workshop by evaluator Ralph Renger on Systems Evaluation Theory, and I liked his meaty response to this exploration.
Renger’s Systems Evaluation Theory outlines three main steps:
Define the system
Evaluate system interdependencies (efficiency)
Identify system emergence (effectiveness)
Before we get into the guts of the evaluation approach, let’s define a few crucial terms. Renger offered a neat definition of systems thinking:
Systems thinking is about identifying and trying to understand how connections, inter-dependencies, and causal mechanisms work.
He also offered this definition of a system:
A system is an integrated whole whose essential properties emerge from the interdependence between its parts.
In other words, when we talk about “systems change”, we are talking about attempting to create new value (an emergent property) by tinkering with the relationships and linkages (interdependencies) between the existing players, institutions and conditions (parts) that make up that system.
To know whether the system is in fact evolving or shifting, we must first be clear on what we consider the system to be (step 1 in the model). Who is in and who is out? Without a clear boundary, the “system” will become a nebulas, vague concept and, critically, measurement will be impossible.
Next, we need to examine whether the linkages have changed as a result of our intervention (step two). Before and after interviews, observations, process-flow mapping (and many more) tools can be used here. This should reveal whether the “system” (AKA those people and institutions) are interacting in new ways now that the intervention has been introduced.
Finally, we would progress to identifying whether some new value has been created (step three). This “value” must have been defined up front as the change we are seeking to create in the system. This is the heart of our intentions. If our theory of change was/is correct, then we expect that because of the new efficiencies in interdependencies (step two), something has emerged or been developed.
Let me provide a practical example of this using the homelessness sector.
Our first step would be to define the system. In this we would identify the critical organisations, institutions, conditions that influence homelessness. We would choose a scope for our work, which may be a particular target audience within a geography. With the critical organisations around a table, we agree on the emergent property that we care about — perhaps stability or a sense of belonging and safety.
Our next step would be to map the connections and relationships between those organisations. This process is detailed and time-consuming — it’s important to have the most clear image you can, as only then can you identify the pain points.
With all of the organisations together again, we would agree on a series of changes to those existing connections to try. This is essentially our best guess of what needs to shift to create the conditions that will lead to our goal (stability). We might think of these things as levers or pathways for systems change.
At this point, we are crafting a strategy. It is only once we have a clear strategy of what we need to do that we should begin to discuss measurement. You can’t measure whether your approach to winning is working, if you never articulated how to win.
As we move toward measurement, we would agree on a series of indicators or observable factors that demonstrate whether the interdependencies have in fact shifted. If our strategy or theory of change is true, we should begin to see those changes unfold as we had predicted. If they start to look different, then we would update our theory of change to reflect this new knowledge.
Over time, we should begin to see signs of the emergent property. The only way to do this is to define it clearly up front — the clearer and more precise we can be, the better. Again, if we don’t see this emerge, then we would seek to iterate our strategy once more, slowly pivoting until we begin to see evidence that we’re on the right track.
This approach suggests that no single organisation can shift the system. Indeed this sentiment is echoed widely. And it makes sense — if one organisation could, then it implies that organisation had power over many stakeholders and how they interact… unlikely.
Yet, when it comes to measuring systems change, we often ask each organisation to show evidence of the broad, macro-changes we hope to achieve. In other words, we hold single organisations accountable for achieving system change.
I’m yet to be confident in what is the most rigorous approach to systems evaluation, but I see a critical point in Renger’s work that is often missed. If we choose to adopt a systems approach, where multiple players are contributing to the higher-order goal, then we should hold the alliance/collaborative responsible for systems change. The individual organisations should be held responsible for their direct intervention.
AKA
Individual orgs accountable for things like the efficient delivery of mental health supports to people living on the streets, or timely connection of clients with emergency housing providers, or suitable re-working of government assistance portals.
Alliance accountable for faster, better, more effective linkages between each other. And, critically, then the emergence of stability.
You may notice the first list looks strikingly similar to what many would call outputs. Provided the alliance can demonstrate probable, causal links between the interventions and changes in system relationships, then this is it — this is when measuring outputs makes sense.
Maybe our continued use of the phrase “system change” has led us a little astray. In an effort to appreciate the complexity, we’ve failed to measure anything. Let’s strip it back to first principles to identify the specific change that an entity is delivering and measure that. If we choose to adopt a collaborative approach, then the onus lies with the collective “we” to measure whether the presumed causal links are actually true.
