5 challenges of evaluating place-based change

How can we measure long-term change in an area? Lily O’Flynn shares tools and advice for evaluating place-based change.

There is growing interest in moving away from individual interventions to solve problems that aren’t joined up and towards place-based change.

What is place-based change? At Renaisi, we define it as:

“Place-based change is a long-term approach to identifying, understanding and addressing social issues using a recognised area to draw in and lock in resources, expertise, and experience. It also values everyone’s contribution and perspective, not just the traditional wielders of power in civil society.“

Essentially, it’s about creating real and lasting change in a place by working long-term alongside all the people associated with that place.  

As long-term impact is central to the idea, obviously being able to track changes over time is an important part of this approach. But we know there are challenges to doing this.

At Stronger Things 2022, I hosted a session asking people to identify those challenges. Here I look at five they came up with and how you can overcome them.

Attribution is difficult in messy, interconnected systems

Place-based programmes are typically made up of multiple interventions and partners who engage over a long period of time.

This makes attribution to one thing impossible, unlike in a traditional evaluation where you would attribute a change to the money or effectiveness of an intervention.

So instead, I think about contribution. To do this you must understand the roles of the different actors (individuals, groups, organisations and institutions) and the unique value that each brings.

For example, the ability or resources to make financial decisions, trusted relationships with sections of the community, or lived experience of the issue.

Once that value is established, I find contribution analysis a useful tool to understand how each contributes to the initiative’s success.

Place-based projects aren’t linear, outcomes can change

Place based programmes are iterative, experimental and transformational by nature, which makes them hard to predict.

Traditionally a framework setting out what you want to achieve and how success can be measured would precede any work.

But for a place-based programme, too much projection means important findings and unexpected outcomes might be missed because they don’t fit categories that were carved out at the beginning of the programme.

Instead, I typically use theory of change models that focus on understanding progress towards change and ‘enablers of change’ as evidence of that progress.

Case Study

Take this example: A successful place-based programme will has a high-level vision for change such as ‘fewer people fall into homelessness’.

That might be the same as a single intervention programme. But what differentiates place-based programmes is the mechanisms or ‘enablers’ used to create long-term sustained change to ways of working across the system or place (using multiple partners).

The outcome is important to measure, but multiple potential enablers and interventions mean it cannot be attributed to just one thing.

In this case the enablers of change might include:

• a homeless shelter
• local authority
• GP surgery
• the Job Centre

all working together to provide support, and help people navigate the system.

In order to understand the impact, it is just as critical to understand what and who enabled the impact to be achieved.

Place-based projects are long term so results are slow to emerge

By a year or so into a programme, funders of it are understandably eager to find out what the impact of their investment has been. The trouble is it is very unlikely that the outcome or ‘high level vision’ will be realised by this stage.

Another challenge is that the evaluation and its findings must stay current. That is hard when most place-based programmes are funded for about 5-10 years, and policies and funder fashions change.

Over the past 20 years place-based projects have taken lots of different guises: regeneration, devolution, community-led, and systemic. 

The reason for listing these things is that for an evaluation to have the necessary rigour, it must stand the test of time.

As above, understanding the ‘enablers of change’ allows us to evidence signs of change throughout.

It also lets us understand what’s working and what needs to change. The Australian Government’s toolkit for evaluating place-based delivery approaches provides an excellent guide for creating robust evaluations around long-term programmes.

Two more challenges were shared, which aren’t unique to place-based evaluations, but are relevant to anyone that is involved in place-based work.       

Capacity and resource of practitioners to devote time to evaluation

Identifying all the actors in a place, developing a theory of change, measuring enablers of change, as well as progress towards change – all sounds very resource intensive!

But the key to carrying out a good place-based evaluation is to be proportionate. Do not try and capture everything. 

Instead, be realistic about what things you can influence and think about how you will work with partners to capture learning and indicators. Sharing responsibility makes the whole task feel less resource intensive.

That said, there should always be one person who pulls it all together. Our friends at Clear Horizon have a place-based evaluation course, which can help anyone who wants to develop the skills to do that. 

Achieving meaningful engagement with all stakeholders (especially beneficiaries)

Most important of all is valuing everyone’s contribution and perspective, not just the usual suspects.

That means your evaluation must build in meaningful engagement from all the people who live in, work in, or use the place. This will help you to thoroughly understand and demonstrate change, as well as make both the research and communication of findings more culturally relevant and sensitive.

But how do you ensure that engagement isn’t tokenistic? Approaches I’ve used are training up ‘community researchers’ to survey their peers, and informal channels that people already use, like Facebook, to sense check findings or ask for input.

I also like the platform Vocal Eyes because it allows the community to rate ideas and have transparent conversations to support participatory democracy, budgeting and crowdsourcing.

Conclusion

It’s true that evaluating place-based projects is difficult.

But by designing a flexible, inclusive way of measuring change over time, you’ll not only learn more about your work – you will build stronger roots and connections with the very people you are working with and for to create that change.

Lily O’Flynn is Senior Project Manager for Place Based Evaluations at Renaisi. Follow her on Twitter @LilyOFlynn

Renaisi is a social enterprise that helps people and places to thrive. Follow them on Twitter @Renaisi

Photo by Tom Rumble on Unsplash

Join our mailing list