Scoping study on Impact Evaluation of ARM in low- and middle-income countries

Scoping study on impact evaluation of financial agricultural risk management (ARM) instruments in low- and middle-income countries The scoping study had five specific objectives: (i) to deliver a practical and communicable tool to map the existing evidence; (ii) to identify priority areas in need of evidence; (iii) to understand the opportunities and challenge of conducting impact evaluation in this field; (iv) to assess the quality of the selected literature and (v) to understand how, why and in what context impact occurs/fails to occur. The client, 3ie, used the results of this scoping study to orient its grant window on ARM impact evaluations. Methodological approach A set of five tools was used, in order to take into account the points of views of a large diversity of stakeholders and agencies active in the field: (i) a theory of change that first helped to identify the relevant outcomes; (ii) an Evidence Gap Map (EGM) which displays the selected papers according to intervention type and studied outcomes ; (iii) an online survey addressed to relevant stakeholders in the field, researchers as non-researchers world-wide; (iv) semi-structured interviews with a dozen of key stakeholders, researchers, policy advisers and practitioners; and (v) a quality assessment and literature review which summarize the robust findings of the existing evidence. (i) The purpose of the TOC is to think sequentially about the causal chain linking intervention to impact and explicitly mention the underlying assumptions at each step. Theory as well as experience from case studies and practitioners in the field have been used to construct the TOC. (ii) The initial systematic search strategy identified 10 203 references on the topic. After multiple screening phases (duplicates, scope, inclusion- exclusion criteria and availability), 92 papers were included in the EGM. All papers were read and classified into the EGM according to the type of intervention and outcomes analysed. This was followed by a quality assessment of the selected papers, aiming at identifying 6 sources of potential biases based on specific criteria (agreed upon with the client) and collecting relevant statistical information allowing to conduct a meta-analysis. A main selection criterion was that the study design allowed results to be attributed to the intervention through a plausible counterfactual analysis. Preferably this is done by using either econometric models, application of multivariate analysis or by using rigorous methods (experimental, quasi-experimental, difference-in-difference, regression discontinuity designs, Propensity Score Matching (PSM), instrumental variables (IV) and multivariate regressions using fixed effects. Finally, the analysis of the trends coming out from the evidence gap map was performed using Stata software. (iii) The online survey was designed on SurveyMonkey. It was kept short (10-15 min). An invitation was sent to more 400 email address, with 69 responses. (iv) The interviews were conducted following a semi-structured interview guide designed by ADE. The 12 respondents were implementers as well as researchers, all from key institutions of the field. The interviews lasted 40 min on average. Missions 1. All tools were closely discussed with the client and we spent a week in India, where 3ie’s main office is based, to facilitate discussions. 2. A mission in Kenya was organized to present the key findings of the study to an audience of researchers and implementers (matchmaking workshop in Nairobi) and to get the feedback from the participants.


Project Details
Country WORLDWIDE
Start date 2015.12.01
End date 2016.05.31
Client 3ie
Sector EVALUATION
Sub-sector Evaluation
Ref. ADE A542