MEDIMSOLA PROJECT: DEVELOPING A TOOL FOR THE MEASUREMENT OF SOCIAL AND LABOR IMPACT

Luis Miguel Morales
Senior Consultant Red2Red

 

Introduction and Context

The evaluation of public policies has undergone a significant development in recent decades, both at a national and international level. In addition to its considerable range which is both quantitative and qualitative, a greater and greater social and political visibility has been produced, in which organisations of the third sector have played an important role when it comes to integrating this evaluation as a tool for the improvement of their own projects, and to offer information on the execution and social impact of the activities that they carry out.

In this context, the European Union, through Regulation No. 1304/2013, establishes a series of directed actions, not only to determine if the projects being jointly financed by the European Social Fund (hereinafter, ESF) are being carried out or not, but also to know the social and labour impact of these projects. One of the fundamental changes in EU funding of the period from 2014 to 2020 in relation to previous periods has been the special paying of attention to the specific situation of the people after their involvement in the different projects, having designed to this end a series of short and medium-term indicators.

For this period of funding, the European Commission has also developed an “ESF Monitoring and Evaluation Guidance� [1] which establishes two possible approaches to evaluate the impact of the jointly financed initiatives. On the one hand, the theory-based impact evaluation and on the other hand, counterfactual impact evaluation. The first of these tries to reconstruct the intervention logic, step by step, identifying the mechanisms of change and the working of the programme, providing qualitative information of impact. The second implies having a treatment group, which receives the programme to evaluate, and another comparison or control group, with similar characteristics to the treatment group and which has not received the programme to evaluate, so it identifies and estimates what would have happened in the absence of the programme, and therefore attribute to the programme the effect that it may be producing.

Within the counterfactual impact evaluation are two approaches which allow for the establishment of the control or comparison group: the experimental and quasi-experimental designs. The first designs consist of the random assignment of people to the treatment group and to the control group, which means that some people will go through the programme and others will not, based on chance. The second designs, however, consist of the application of certain techniques which allow for the creation of the counterfactual group without chance necessarily intervening in the assignment of people to one group or the other.

The case presented on these pages is that of a counterfactual impact evaluation with a quasi-experimental design, in which the situation of the people taking part in the programme before the intervention has been used as a counterfactual scenario. This has been carried out this way because it is considered that, in the absence of an addiction treatment programme, it is rather unlikely that people would reintegrate socially or in terms of work.

Object of Evaluation: INSOLA Project

Before explaining the approach and tasks of the MEDIMSOLA Project, a brief explanation must be given of what the INSOLA Project (whose initials translate as Socio-Labour Insertion), as well as the way in which it the project is structured, given that it is the object on which the MEDIMSOLA project is being developed.

The INSOLA project has as its objective to improve the socio-labour insertion of people in situation or at risk of social exclusion, through an integral approach to addiction problems, establishing for this three different operations or services:

• Operation 1. Service of initial orientation, where behaviour expected in the first steps of a user are taken into account.
• Operation 2. Service of training, which includes actions directed at the competency-based training and development of skills for life (prior to employment), as well as actions directed at socio-family inclusion.
• Operation 3. Service of Specialised Orientation, which includes activities directed at competency-based training and the development of skills for (working) life, considering actions of work intermediation or the providing of labour insertion in protected environments, such as insertion companies.

The execution of the INSOLA project, jointly financed by the ESF, included the collection of a group of common indicators, which are gathered at three different moments of the intervention: firstly, the productivity indicators when the user joins the operation; secondly, the so-called immediate result indicators, collected up to 4 weeks after the end of the operation; and, lastly, the long-term results indicators, collected 6 months after the operation has ended.

As a result of the development of the MEDIMSOLA project different products are being developed which will contribute to the objective of the project to construct a systematic and integrated system of quality indicators which allow for the making available of rigorous and sufficient information to determine the impact that the INSOLA project is having on the people taking part in the project.

MEDIMSOLA Project

Taking this context into account, the development of a parallel project is considered, which, being based on the indicators defined by the ESF allows for the development of a systematic methodology to calculate the socio-labour impact of the people periodically taking part in the project. The objective is to improve the quality, effectiveness, efficiency and the impact of the operations that are executed through calculating the social impact.

This project is structured in three dimensions which are worked with in parallel for the development of this methodology:

• Analytical dimension: definition and development of the methodological tools of impact evaluation which allow for the periodic carrying out of social impact analysis of the INSOLA project, learning from this and highlighting the importance of ESF support.
• Organisational dimension: interaction of specific internal process and the involvement of the different agents of the organisation.
• Technological dimension: fundamentally based on the development of specific applications to adapt the available monitoring systems which allow the necessary information to be transferred, managed and processed to work out the social and labour impact of the INSOLA Project.

During the development of this project a Tractor Group has also been set up, made up of three centres of the Proyecto Hombre Association (PH Alicante, PH Canary Islands and PH Granada), which take part in the INSOLA Project, and which are developing a role of pilot centres in the development of the tool.

As a result of the development of the MEDIMSOLA project different products are being developed which will contribute to the objective of the project to construct a systematic and integrated system of quality indicators which allow for the making available of rigorous and sufficient information to determine the impact that the INSOLA project is having on the people taking part in the project. The first two products are directly related to the development of the methodology which allows for calculating of the socio-labour impact of the people taking part, while the following three products are a result of the launch of methodology and its first application to reality as an impact evaluation tool:

• Development of tools for the carrying out of net impact evaluations. Three different tools are being developed: a) aimed at analysing the sociodemographic characteristics of the people taking part, b) to carry out monographic analysis on characteristics of interest of the people taking part, and c) tool linked to the development of impact analysis. Likewise, a template type has been developed which may be useful for all the Proyecto Hombre Association centres.
• Action plan for the implementation of the system in the whole organisation. A first draft has already been made but this will be finished once the tool has been tested with the execution of the net impact evaluation report of 2019.
• Net impact evaluation report 2018. Submitted in October 2018, it was worked on with the information from the second half of 2017 and with the three centres of the Tractor Group.

Collectively in the three operations, it was noted that 83% of people taking part who finished the operation had achieved some type of improvement in the 4 weeks after finishing the operation. Furthermore, the employment rate increased from an initial 14% to 50% upon finishing the operations, meaning an increase in employment rate of 36 percentage points.

• Net impact evaluation report 2019. Pending execution between July and September of this year, and it will include information from the whole of 2018. It will be a repetition of that carried out in 2018 and will serve to validate the tools being generated in the project.
• Evaluation report of Social Return on Investment (SROI). Pending execution this year, and its submission is planned for 2019.

References

Gertler, Paul J., Sebastián Martínez, Patrick Premand, Laura B. Rawlings and Christel M. J. Vermeersch. 2017. Impact Evaluation in Practice, Second edition. Washington, DC: Inter-American Development Bank and World Bank. doi:10.1596/978-1-4648-0888-3. Attribution license: Creative Commons CC BY 3.0 IGO

Labeaga, J., & Muñoz, C. (2013). LA EVALUACIÓN DE POL�TICAS PÚBLICAS EN ESPAÑA: APRENDIZAJE Y PR�CTICA INSTITUCIONAL. Revista de Evaluación de Programas y Políticas Públicas, 0(1), 31-53. doi: https://doi.org/10.5944/reppp.1.2013.10778

Regulation (EU) No. 1304/2013 of the European Parliament and of the Council of 17 December 2013 on the European Social Fund and repealing Council Regulation (EC) No 1081/2006

Rogers, Patricia (2014). Overview of Impact Evaluation: Methodological Briefs – Impact Evaluation No. 1, Methodological Briefs no. 1,

[1] https://ec.europa.eu/sfc/en/system/files/ged/ESF%20monitoring%20and%20evaluation%20guidance.pdf