10 Apr 2014
Jo Hamilton on why monitoring and evaluation matters for Transition
What role does measuring and evaluating your impacts have to play for Transition initiatives? How important is it, and how straightforward is it in a group that is already busy “doing stuff”? Jo Hamilton is a researcher at Oxford University’s Environmental Change Institute whose research focuses on those very questions. She is currently developing a project called Monitoring and Evaluation for Sustainable Communities (MESC) to develop and trial a range of tools to enable groups to self monitor and evaluate their work. She’s still recruiting groups and is running 3 workshops in April and May for groups who’d like more skills and insights on how to do this (more below).
The project idea emerged from meetings with the Transition Research Network, and is a collaboration between the University, Transition Network, and Low Carbon Communities Network. We started by asking Jo why it matters that Transition initiatives should do monitoring and evalution:
“Used well, Monitoring and Evaluation (M&E)** can be part of toolkit for helping Transition Initiatives assess and make the changes that they want to achieve.
My prior experience of being involved in, supporting, and more recently research with community groups, has demonstrated the power of reflecting on what has been achieved, learning from what has worked, what hasn’t, and what unexpected outcomes there have been. Whilst analysing comments from feedback forms after community events has sometimes felt like the last thing I’ve wanted to do, it has always been helpful: to guide future activities, to communicate what we’ve achieved in the event, and to help us see what other changes need to take place. Positive comments can give a much needed energy boost, whilst critical or negative comments can be the starting point of another conversation and provide useful feedback.
Why does having an evidence based for your impact matter?
Let’s face it, we’re not going to get ‘good feedback’ about the impact of local action from the weather or climate, so we need to see what feedback we can get from the people we’re working with, and the local environment we’re working in.
On a wider scale, having an evidence base is crucial to demonstrate what Transition initiatives have achieved, and to provide weight to argue for investment in local action, or policies that can enable local action to scale up. At present the evidence base is small, but growing. In addition to the evidence generated by groups themselves, in recent years there have been many academic research projects, masters and doctoral dissertations, which demonstrate impact. You can access many of these through the Transition Research Network.
What makes a useful indicator? What is worth measuring and what isn’t?
Indicators are specific pieces of information that you collect, so that you can track the changes you’re aiming for. Whilst it is useful to measure the number of people who are reached by or involved in group activities, the changes, or outcomes, that you contribute to are the key things to measure. These could include whether somebody chooses to eco-renovate their home, switch transport modes to more low carbon forms, or exert political influence. However, alongside indicators you also need to ask questions to understand why and how the changes occur and capture unexpected outcomes.
Is monitoring and evaluation something that groups should be looking to do from Day One, or can it be something they pick up later, and if so when?
‘Start where you are’ is the key phrase here, as groups get initiated in different ways and have different motivations. Planning M&E is similar to project planning, so integrating M&E into any form of planning is most helpful at the beginning of a project, although it can also be done at any stage. Simply examining the assumptions that underpin the activities you want to carry out, and the changes that could be expected is really useful. Whatever stage a group is at, M&E can help you learn more about what works, what isn’t working, and what could be done. We’ve compiled a step by step guide which you can download here.
How do you see the balance between getting on and doing stuff and measuring it? Is there a danger that measuring things can take away the energy that gives you anything worth measuring in the first place? There’s the balance?
It can be a tricky balance to strike, and many groups haven’t done M&E precisely because the focus has been on the doing. However, I liken M&E tools to penknives: they’re multifunctional tools, which fit in your pocket, and you know how to use them. Some penknives are nice and simple, whilst some look like they might be too cumbersome and complicated, thus are unlikely to be carried around and used. M&E is a bit like that. The process of M&E can be multifunctional, the trick is to select the tools you need, carry them round with you and integrate them with what you are doing anyway.
However, from experience and from research, I know that reflecting on what you have achieved over the past year, or reading positive feedback from an event, can be a real energy boost. Doing this with other groups can help get a wider perspective on the impact of your work, share valuable learning, and identify areas for collaboration on issues which are beyond the capacity of one group alone. It can be a fine tuning mechanism, to help your group set achievable goals.
Some groups (for example Low Carbon West Oxford) who developed a system for M&E from the beginning, have been able to demonstrate their impact to the local authority and funders, which has led to further collaborations and enabled them to replicate and scale up some of their projects.
A lot of measuring can be incorporated into other activities that you’d be doing anyhow – when you’re asking for people’s contacts for emails, ask a couple of questions too. At some events, simple feedback can be provided through engaging activities such as writing thoughts and feedback on post-its.
What are some of the principles that underpin good and worthwhile evaluation?
Following on from the previous question, it’s good to set some guiding principles for your M&E, and to ensure that you have the resources to do it well. Guiding principles could include making sure that your M&E is focused and feasible, whether it’s useful for, and usable by the group.
You might need to generate evidence for potential funders, or to leverage more support for your work from the local authority. In the MESC project we’ve been selecting indicators and devising resources that will hopefully enable groups to compare themselves to others, and which can be aggregated so that there’s a more comprehensive view of what is happening at a national level.
What sorts of things might a Transition initiative want to measure?
It depends what the focus of the TI is, or where the energy is for M&E. You might want to measure the carbon reduction achieved from participants in your activities, how your events are helping local residents in fuel poverty access grants and other services, or how your farmers market is influencing residents’ shopping patterns and food sourcing.
Who are they doing this for? Themselves? Local government? Academics?
M&E can provide useful information for the group and wider movement itself, in helping you to answer the question ‘so, what has your group actually achieved?’. This can help the group feel proud of what they’ve achieved, and help plan future activities. Local and national government always want figures of what Transition Initiatives and other community energy groups have achieved, and being able to provide some of those figures can help justify funding and provide evidence for policy making (such as the recent Community Energy Strategy).
Can groups do this alone or do they need to do it in partnership with other organisations?
We’re currently trialling resources and tools to find out what groups can M&E alone, and what support they need to do more. More in depth M&E could involve partnering with other organisations, such as Universities, or through the Transition Research Network.
How have you developed your resources?
The step by step guide to M&E and tools are based on the teams’ research knowledge and practical experience, and draw on a range of existing resources and research.
We got initial feedback on the step by step guide and some of the tools at two workshops that took place in June 2013, and we’ve developed and adapted the tools.
Lastly, you are running three free workshops for Transition initiatives who want to find out more about this. Can you tell us more about those?
Thanks, perfect plug to the workshops, which we’ll be running in three locations.
The free workshops will give you an introduction to planning your M&E, and a chance to trial a range of resources. The workshops are part of the MESC project, so participants can receive follow up tailored support to help you monitor and assess impact.
- Better understand what works and what doesn’t;
- Generate data that will help you to create better reports for funders and other stakeholders;
- Get a chance to trial a range of resources that will enable your group to self-monitor and evaluate your activities;
- Inform your next steps in whatever project or initiative you are working in;
- Respond to those queries of ‘so what has your group actually achieved?’
Workshop Dates and Locations, all 10am – 5pm (pick one)
Sat April 12th – Oxford at School of Geography and Environment
Sat April 26th – Manchester at Anthony Burgess Foundation
Sat May 10th – London, Lumen URC (nr Euston station)
Advance booking is essential, and priority will be given to groups who would like to participate in the MESC project to trial the resources. For further information please email kersty.hobson@ouce.ox.ac.uk or see the project website.
** Monitoring is the collection and analysis of information about a project or programme, undertaken while the project is ongoing. Evaluation is the periodic, retrospective assessment of a project or programme.