For those of you who have successfully designed your Theory of Change (ToC), then congratulations, you have taken a vital step towards being able to successfully measure the impact of your project. For those of you who have not yet designed your ToC yet, we strongly advise you to design your ToC first, using our previous article as a guide. However, for those of you who do not yet have the capability to fully design a ToC, you can still use this article as a guide in making surveys for collecting impact data, as long as you have already identified your outcomes and outputs.
The next step after identifying your outcomes and outputs through your ToC, is answering the question of ‘how to collect the necessary data to show impact?’ The most basic form of impact data collection is a simple survey to the intended stakeholders.
Why do we design surveys for impact measurement?
Surveys for impact measurement are important to understand by how much your outcome has been fulfilled, and how much output has been created by the project. The data collected through surveys can enable you to understand that you have created X amount of immediate change towards X amount of beneficiaries.
However, the use of surveys is not only limited to understanding how much outcome and output have been created but also to understand other levels in the ToC, such as input. For example, you can use a survey to also understand how many volunteering hours each volunteer has spent on the project.
In this article, we mainly explain how to use surveys to collect outcome and output data. Below are the steps to create your survey:
- Step 1: Determine the outcome to measure
The first step in creating surveys for impact measurement is to determine which outcome or output you would like to measure. From your ToC, list down which are the outcomes and outputs you would like to collect data for. The reason that we collect data for output as well, not only outcome, is because output shows the throughput level of the project; which is the amount of progress the project has managed to make so far.
Note that you might be measuring the same data for both output and outcome, and if that is the case, it is perfectly fine. After you have determined which outcome and output you want to measure, the next step is to define the parameters for each output and outcome.
For the purpose of this article we will take an example of a food security project in Indonesia, which aims to also help farmers in rural villages sell more of their products by creating a value chain. We will use the outcome “farmers are able to sell more of their products with the help of the project” to be measured through our survey.
- Step 2: Define the parameters of your outcome and output
Parameters are the specification and scoping of the terms that you have used in each outcome and output. For example if you use the word ‘poor people’ in your outcome/output, you would have to define ‘what constitutes somebody to be included as poor people?’, or ‘is it all poor people in general or only poor people living in a specific area?’ and other questions that would further specify and scope the terms you are talking about. You can also use existing research or data as a reference to define your parameters. For instance you could use the national poverty rate to define ‘poor people’.
Usually you would only have to define the parameters for your stakeholders that would benefit directly from your project. However, if you have terms in your outcome/output that are not yet clear for the average person at first glance, you would have to define the parameters for those terms as well.
In the food security project example, you want to measure the outcome “farmers are able to sell more of their products with the help of the project”. Since for this outcome, the stakeholders are the farmers, you would want to set a parameter for the farmers. For this example, the parameter would then be ‘village farmers around West Java island who are able to sell their food products via the project’.
- Step 3: Determine the data points
After determining the parameters, the next step for impact measurement is to have a clear picture of what kind of data you are collecting, which can show the degree to which your outcome has been fulfilled. Those data that you want to collect are called data points. To determine your data points, you can ask yourself the question “what are the necessary data needed to show that the outcome has been achieved?”. Your answer to this question would then need to be specified into 1) type of data and 2) unit of measure for each data point.
To make this more clear, let’s go back to the example of the food security project. We need to answer the question, “what are the necessary data needed to show that the farmers indeed are able to sell more of their products with the help of the project?”. The answer to that question is:
- Each farmer’s ability to sell per time period—before the project and after the project.
The data to define ‘ability to sell’ would be then specified into average revenue received per month. This would then translate into 2 data points:
- Average sales revenue per month before the project (Unit of Measure: USD)
- Average sales revenue per month after the project(Unit of Measure: USD)
Hence, by acquiring these 2 data points, you will then be able to show how much increase in sales the farmers have experienced with the help of your project, or whether there was an increase in the first place. In addition to these 2 data points, you would also like to know how much farmers were able to increase their sales with the help of your project.
You might be wondering that if there was an increase in sales to the farmers, how can we know that it is truly due to your project, not due to other factors? for example government subsidies, or maybe another project that aims to help the farmers? The answer to this is by calculating counterfactuals, or simply ‘what would have otherwise happened in absence of your project activities’. However, as we are mainly focusing on how to create surveys for impact measurement, we will not be applying the concept of counterfactuals in this article. You can check out this article to read more about counterfactuals and the 5 dimensions of impact.
In determining your data points, you would also need to consider which data is accessible to you, and that is possible to obtain from your intended stakeholders. In this, you would have to consider the background of your stakeholders, e.g. literacy, education, socio-economic status. For example, you would not be able to ask about the average growth in sales per month to farmers in rural villages who mostly only went through primary education. You need to tailor your data points and later on your survey questions to your intended stakeholders.
- Step 4: Define the success indicators
Now you have your data points. Let’s say you got your outcome data and you look at all the data you have acquired. Then what? there needs to be a step where you define, from all the data you have acquired, how your project has succeeded in achieving your outcome. What is a success for you defined as? There is one note though, that determining the success indicator only applies to outcome data; as output data is only used to show how much progress the project has made.
In the example of the food security project, we already have the 3 data points. How can we say that we have succeeded in achieving the outcome? You would probably think about how much increase in sales has the farmers experienced, or maybe you want to have a specific number of how many farmers you would want to have their sales increased. The definition of success of your outcome is entirely up to you to decide; it depends on how much impact you want to bring to your stakeholders. In this example, the organizers of the project decided that they want to aim to increase the sales of at least 75% of the farmers that were helped by the project, with a minimal increase of 30%.
A useful tip in defining success is that you can also define success over a period of time. For example, in addition to the 30% increase, you could also say that you want to see a 2% increase in sales each month since the project started. Meaning, for the first month since the project began, the farmers witnessed an increase in sales by30%, then the next month by 32%, then the next month by 34%, and so on. This kind of success indicator can also be used for a project that aims for sustainability towards a certain stakeholder, for example for the stakeholders to be empowered. In this case, the key success is in how the stakeholders can be empowered to grow and increase the positive impact for themselves over time.
- Step 5: Determine how to collect your data
Once you are set with all the data points and success indicators, it’s time to go a bit into the technical, operational details of how you would actually collect the data. Here there are 5 questions to be answered:
- Who will collect the data?
- From whom will the data be collected?
- When will the data be collected?
- Frequency of data collection?
- How will the data be collected?
In answering these questions you would consider the tangible resources that your project has (e.g. technology, manpower to conduct surveys, logistical resources, etc.), as well as the intangibles (e.g. technological literacy). In answering the question of ‘who collects the data’ for example, would you consider to have a dedicated team of experts to collect the data, or would you prefer to have the actors (people who actually make the impact happen on the ground; e.g. volunteers) to do the job? While the former would require you to pay for additional costs (e.g. expertise, transportation, and other operational costs to conduct a dedicated impact survey), the latter would be very easy to do using the Artemis impact measurement app. This free app allows your actors to collect data in a simple and user-friendly way, while it also allows you as the project organizer to extract the data to see how much outcome and output has been produced by the project.
In the food security project example, the volunteers are the ones who will be collecting the data to the farmers every week, using the Artemis Impact Measurement App.
Step 6: Formulate the questions
If you have made it this far, congratulations! This is the last step of creating surveys for impact measurement, which is creating the survey questions. Your questions would be based on the data points you have made, as you will want to acquire the data points through these questions. As has been mentioned before, your questions should be tailored to your stakeholders. Are they people who will understand tech terms, for example? What is their educational, cultural background? All these intangibles need to be put into consideration when formulating the questions. Your questions should be simple and easy to understand at first glance, and preferably you should have one question for acquiring one data only, not more, so as to not overcomplicate things for your stakeholders.
If we go back to the food security project example, the interview questions would be pretty straightforward based on the data points. The project started to impact farmers since the first week of August, so this would be used as a reference point in asking the questions.
- Demographic questions (Name, age, village)
- Before August, can you tell us, based on your recent memory, how much supplies did you sell per month?
- Can you tell us how much income did you receive through those sales per month?
- For the past week, can you tell us how many supplies have you sold?
- Can you tell us how much income did you receive through those sales?
In this example, the project organizer understood that data regarding sales were not being well documented by the farmers each month, so the most effective way is to ask them directly and frequently regarding how much they have received through their sales over the past week, which is something that they can remember and tell.
Creating surveys to measure your outcomes and outputs can be done by going through the six steps, from determining which outcome or output you want to measure until formulating the survey questions themselves. It is a process that places great emphasis on understanding the resources available to you as well as the background of your intended stakeholders. Below is a table that summarizes all six steps, along with the key questions you should ask to conduct each step and the general tips.
Determine the outcome to measure
Which output or outcome from my ToC that I would like to measure?
Define the parameters of your outcome
Who are the stakeholders? What are the terms in my outcome that need further specification and scoping?
Determine the data points
What are the necessary data needed to show that the outcome has been achieved?
Define the success indicators
How would I define success in achieving my outcome?
Determine how to collect your data
Formulate the questions
How can I ask questions that can help me obtain the data points?
If you have any questions feel free to check out our website or get in touch with us via email: firstname.lastname@example.org
Read more about Impact measurement below :