Important Concepts to Understand About Measurement

Important Concepts to Understand About Measurement

Measurement entails ‘how’ you will establish the performance of your indicators. In other words, the extent to which your programme has achieved its outcomes. It answers questions around the impact of your intervention and whether or not your programme has caused or contributed to change, if it has been achieved. 

Measurement varies in difficulty. Some indicators relate to your outputs, the products and services produced through your programme representing evidence of programme activities. These will mostly involve systematic counting of products and services delivered and ideally you should develop tools and data collection methodologies yourself. They are related to the specific operations and management of your programme/organisation.  For outcomes with indicators that are more complex to measure, it might be best to consult with a researcher that has experience using different types of measures and research methodologies. As a start, you could post your question on this website as we have a network of evaluators that are supporting this community. 

Remember that measurement is empirical – this means it is knowledge that is derived from observation or experimentation and is not based on theory or logic or individual thinking. Keep in mind that there is a difference between people’s perceptions and information based on facts which is objectively verifiable information.  For example, you might come up with different answers when you ask people if they think they have learned something and then test them on that knowledge or new skillset.   

For evaluation of social programmes you will generally try to measure (please watch the video for some practical examples):

Indicators that are linked to an outcome that talks about ESTABLISHING OR DOING SOMETHING NEW. As there is no comparison involved, you should be able to measure this in a single instance.

Non experimental research

Can illustrate that something took place and describe it. This includes some cases where its occurrence is associated with the occurrence of something else, like your programme activities.  Although it might allow you to say that two or more things happened concurrently, it cannot say that one caused the other one. If you are making that conclusion you are guilty of some logical inference at best and wishful thinking at worst!

Indicators that are linked to an outcome that talks about ACHIEVING A MINIMUM STANDARD. As you are not comparing over time, but against the standard, you should be able to measure this in a single instance.

Indicators that are linked to an outcome that talks about INCREASE/IMPROVE/DECLINE OVER TIME. If you speak about an increase/improvement you will need to compare your indicator against two points in time. This means that your measurement will entail some kind of pre- and post-testing, or knowledge about the baseline conditions.  

Quasi-experimental research:

Pre- and post-testing and comparison to the baseline involves comparing the participants against themselves before and after the programme. If you are testing for objective factors and not only for the perceptions of people (for example, you are actually testing the knowledge of participants and not only asking them whether they think they have learned something), you should be able to effectively say if change has occurred or not. If you can be 100% sure that the programme was the only factor influencing the changes that you measure over time, this could mean that your programme is the cause of the change.   

Indicators that are linked to an outcome that talks about PROGRAMME EFFICIENCY AND ATTRIBUTION. You measure the impact or efficiency of your programme by establishing what the status of things would have been if your programme had never been implemented. Researchers call this the 'counterfactual'.  However, ‘how things would have been’ is clearly impossible to measure directly; it can only be inferred. One way of inferring this is by comparing the outcomes of those who participated in the program against those who did not participate.  

Experimental and/or quasi-experimental research: 

Apart from pre-and-post testing, there is a variety of quasi-experimental research design that could be used to compare groups of people. The difference between quasi-experimental research and experimental research is that in the former, groups are pre-assigned based on some characteristic or quality.  For example, in most cases of post hoc (after the fact) evaluation the ‘research’ group will inevitably be the group that has participated in the programme with their characteristics targeted by the programme. The challenge will be to find a ‘control’ group of people who did not participate, but would closely resemble the participants if they had not received the programme.

With true experimental research the assignment of groups are completely random, a bit like the lottery winner selection if everyone only had one ticket. The programme is then implemented only in one group and comparison of the two groups is done when appropriate.  Although randomised evaluations are the ‘gold standard’ in terms of evaluation, we know that many organisations don’t start with impact evaluation in mind, but rather with a passion to assist people who need help. Quasi-experimental designs are dependent on more assumptions than true experimental design. For example, that non-participants are identical to participants and are equally likely to enter the programme before it starts. If these assumptions hold true, this type of research gives us the correct answers to our evaluation questions.

There are ways to do quasi-experimental research that will reduce the risks of biased answers.  Have a look at this useful chart by JPAL which explains the different designs and the assumptions that underlie them.    

References: M&E Blog; JPAL; Exploring Research – Neil Salkind