Integrating “Evaluation” and Learning
In order to avoid merely reacting to events, as advocacy organisations are sometimes prone to do, you should use a process of continuous evaluation. Evaluating your work will provide you with basic information that will help you learn from all aspects of a project – learn to avoid making the same mistakes again, and learn to understand and replicate your successes.
Many organisations feel that they don’t have the resources or the capacity to conduct in- depth evaluation, but evaluating your work and its results is the only way to improve your effectiveness, and therefore it should always be a priority. Evaluation should be built into your plans and your funding expectations from the start. The good news is that evaluation activities can be easy and low-cost, and you don’t necessarily need outside expertise to conduct them properly! Steps one and two below will probably take the most concentrated time, maybe half a day’s work at the outset of the project. After that, you should think of evaluation as an embedded activity that you can take responsibility for throughout the life of the project.
Look for key moments on your project’s time-line: where is there a natural pause in which you can analyse what has just happened? What are the important events or milestones
that should trigger feedback? And remember: evaluating and understanding where your project has failed, and which tasks were not accomplished successfully, will provide you with excellent opportunities for reflection and learning, and improve your subsequent work.
Review your goals
Your theory of change and your strategic plan will help you establish exactly what you
are evaluating. How are you expecting your technology project to help you achieve your strategic goals? What is the principal outcome you envisage for your tech project, and what are the other effects you’d like it to have? How will your project help bring about progress toward the changes you are advocating?
Thinking about these things should lead you to tough and specific evaluation questions;
for example: “Did the text messages we sent succeed in stimulating our constituents to take action?”, “Is our intended audience actually able to use our data portal?”, “In what precise way is this work contributing to greater government or corporate accountability?”
Be alert to the possibility that your goals may change as your project and your evaluation
of its results evolve. Take note of the moments in your project’s life-span when you learn something that changes your outlook, and what exactly triggers that change. Pay attention to the context (political, socio-economic, cultural) around your project, and how that changes as it progresses. Which changes in the surrounding context are due to your work? Which are the result of factors that are beyond your control? You can learn a great deal, even from the things that you cannot control.
Determine a baseline
Before you start implementing your project, make sure you know how your issue stands now, in the status quo before your project begins. This knowledge will provide you with a starting point, or baseline, from which you can track progress. Establishing this might require a series of interviews to determine how people feel about the accountability of their government,
or a measurement of how many people are currently using the relevant data, for example. Finding out how things stand before your project is instigated will allow you to track your progress from this baseline.
Once you’ve worked out the baseline of the current state of the issue that your project seeks to resolve or address, you should think about what the potential positive outcomes of the project might be. What will your tech project allow you to do that you wouldn’t be able to do otherwise? How will it bring value for users and for other people or groups? What will it allow them to do that they couldn’t do otherwise? Answering these questions will help you to think about the results that you hope to achieve, and how to measure them. You should also think about any potentially negative or unintended consequences that your project might trigger. Answering these questions will allow you to identify any risks and how to track them.
Metrics are the methods you use to conduct your measurements of your project’s effects. Think about how you will measure your success. It may be a question of analysing the number of new users your project reaches, say, or you might be hoping that your project will bring about changes in people’s behaviour. In the latter case the indicators of success would not be expressible in numbers only. For example, “We want our data to be used by investigative journalists. What impact will they have? How exactly will their use of these data help reduce corruption and increase accountability?” The answers to these questions may not be quantifiable, but it is important to be able to observe changes.
Nevertheless, don’t be afraid of numbers. When you’re thinking about establishing quantitative goals, remember that you want the numbers you’re aiming for to be realistic, not aspirational – this is for you and not for a funder! Think clearly about your organisation’s capacities, and try not to over-reach yourselves. While the hard-to-quantify metrics of behaviour are great, making yourself back them up with numbers for the results that CAN be quantified will force you to set parameters that you can use later to judge your success. You will probably start by using metrics that measure the unquantifiable aspects of your work, but the trick is then to put the numbers in to support them and add another dimension.
What are the measurable aspects of the project? Can you turn those aspects into numbers? What would be the quantifiable minimum effects that you could expect from your project? For example, if you’re hoping for a hard-to quantify change in media awareness, you might decide that it would be realistic to expect at least a 10% increase in news stories by investigative journalists using your data as a source. When your project arrives at one of the natural pauses in its life-span, whether you’ve reached the minimum quantitative goal you’d set yourselves or not, it’s important to understand what factors contributed to your success or failure.
Create an evaluation plan
Decide ahead of time when you will conduct evaluations as your project progresses. Create criteria that will help you trigger reflection and metric measurement as and when they occur. Are there events or activities on your project’s timeline after which you know you will be able to pause, debrief and capture key measurements?
Rather than making evaluation a major task that you undertake all at once at the end of your project, think of all of your evaluation work as something easy and necessary that
you undertake fairly regularly throughout the implementation of the project. Look for
key milestones in your project’s progress – moments when it would be useful to debrief yourselves and ask, “How did that go? What happened, exactly? What was unexpected, what worked as we’d foreseen? What could we do to improve the process?”
Think about who else should be asked to provide feedback, outside your own group. If your tech project is intended for use by people beyond your organisation, think about which
of the intended users should be invited to feed back, which would be willing to do so, and what provisions would make it easiest for them to give you the information that would be most useful.
Implement your tech project (and your evaluation plan)
Make sure you integrate the things you learn from your evaluation activities into your project as it progresses.
Store your evaluation findings securely but accessibly. Use a spread-sheet or other document that you will be able to access for future reference, particularly when you’re planning future projects or making reports to your funders. And make sure you are routinely backing up those documents!
Most importantly, use your evaluations to learn how to be more effective!
Having a light-touch evaluation plan will help you fine-tune your technology projects as you implement them. This kind of evaluation process will not depend on having additional funding or hiring an outside evaluator. However, you may need to undertake more robust evaluations, where you are having to prove the effectiveness of your project or organisation to external entities. If this is the case, you should definitely consider hiring a consultant to provide an objective outlook on your project and its evaluation, and you should also be prepared to raise separate funds to cover the costs of this.