TOR:
Consultant for Project Final Evaluation Indonesian Disaster Management
and Logistics Readiness. Sector: Risk Management Policy and Practice
1. Description of Project to be Evaluated
1.1 Project background
Indonesian
Red Cross (Palang Merah Indonesia – PMI), supported by American Red Cross, has
implemented a project since April 2016 that aims to provide assistance to
drought affected populations and improved PMI disaster response capacity and
improved coordination among key stakeholders during short-term El Niño related
drought, and related and longer term disaster management situations. This
project actually covers 2 sectors, that is WASH and Risk Management Policy and
Practice. Funded by USAID/OFDA, this project covers two sectors that are Water,
Sanitation, and Hygiene (WASH), and Risk Management Policy and Practice. The
first sector was implemented under emergency response scheme that was concluded
in December 2016, while the second sector is ongoing until the project closes
by the end of 2018. To achieve the goal, the project targets two outcomes that
each of them lies under each project’s sector. The first outcome is targeted
communities adversely affected by severe drought conditions receive adequate clean
water and hygiene promotion (for WASH sector); and the second outcome is PMI
Regional and Sub-Regional hub strengthening capacity and management by
developing SOPs, contingency plan, and improving personnel’s response skills
and knowledge (for Risk Management Policy and Practice sector). The project
would to conduct final evaluation for each sector. This Term of Reference of
Evaluation exclusively focuses on evaluating the Risk Management Policy and
Practice sector.
American Red Cross has been
partnering with Indonesian Red Cross (Palang Merah Indonesia – PMI) for
strengthening emergency response system through improvement in national
logistics system, logistics related policy development, a dedicated funding
(mini DREF) for replenishment stock in emergency, and personnel capacity
development. This system strengthening is actually the project has been
supporting through the Risk Management Policy and Practice sector. PMI is the
largest humanitarian organization in Indonesia and in the Asia Pacific region.
Serving as an auxiliary to Government of Indonesia, PMI works in close
coordination with Disaster Management Agency at national (Badan National
Penanggulangan Bencana-BNPB) and provincial level (Badan Penanggulangan Bencana
Daerah - BPBD). PMI has operating branches at Provincial level (34 Provinces)
and District level (459 Districts) with its volunteers distributed nationwide.
PMI has 2014-2019 Strategic and Operation Plan that includes increasing its
services for timely, appropriate, and well-coordinated emergency response.. To
achieve this objective, PMI has established a regional approach for improved
emergency response system in the context of Indonesia. Considering that the
country has more than 17,000 islands with diverse disaster risks, PMI divided its
response system into six regional areas and four sub-regional areas: 1) Sumatra
region (Padang and Aceh), 2) Kalimantan region (South Kalimantan), 3) West
Java, 4) Central Java, 5) East Java-Bali-NTT-NTB region (East Java and Bali),
and 6) Sulawesi-Maluku-Papua region (South Sulawesi, North Sulawesi and Papua).
PMI has established a large warehouse at each region for preposition stock and
WASH national scale infrastructure in Bandung, West Java, for emergency
response.
The project has put large
investment on personnel technical skills through six specialized trainings that
are assessment, distribution, shelter, WASH, logistics, and command center
using the regional approach. Within a year of implementation, the project has
trained 1,062 PMI staff and personnel from the six regions and resulted skilled
903 personnel. Of 212 personnel trained in 2017, on average, 93% PMI personnel retained
the knowledge from 3 specialized trainings (94% for Assessment, 93% for Distribution, 94%
for Shelter, and 94% for WASH). For the training quality assurance, PMI has
drafted Training’s Monitoring & Evaluation guide, standard curriculum for
four specialized trainings (assessment, distribution, shelter, and WASH), and
guide for logistics and command center exercises. Apart from this capacity
development, the project continues working on developing PMI logistics standard
operation procedures for the six regions and hazard focus contingency plan (haze
and drought) within the regional approach. PMI also has disbursed the mini DREF
to replenish stock out relief goods during Lombok earthquake response. Since
the project is going to close by the end of 2018, there is a need to implement
Final Evaluation to seek how the project has brought benefits to the strengthened
response system with the case of the Lombok response.
1.2
Project location
Since the project focuses on
strengthening the regional approach, the project operates in the six regions
(Sumatra, Kalimantan, West Java, East Java, Central Java, and
Sulawesi-Maluku-Papua). The project works across various divisions at PMI Head
Quarter that are Facility & Infrastructure Bureau, Disaster Management,
Training, and Volunteer Divisions. Based on capacity development, the number of
total people reached by the project is 1,062 that are PMI staff and personnel
from the six regions.
1.3
Project stakeholders
1.
USAID/OFDA: Donor
2.
American Red Cross: Supporting Partner (Operating Country,
AMEE Region, Head Quarter)
3.
Indonesian Red Cross: Implementer (at three levels: Branch,
Chapter, and Head Quarter)
1.4
Previous evaluation activities
Throughout
implementation of the project, PMI and American Red Cross performance
monitoring (Project Quarterly Report and Indicator Tracking Table) and Project
Mid-term Evaluation Report in 2018.
2. Evaluation Overview
2.1
Objectives of final evaluation
The Final Evaluation
objectives are to:
1. Measure
the benefits of project’s Risk Management Policy and Practice sector during
emergency response.
2. Determine
project’s effectiveness.
3. Document
key lessons learned and best practices.
2.2 Evaluation criteria and questions
The main evaluation question is: How did PMI’s capacity development and
policy strengthening improve its services for emergency responses?
In order to answer the main
evaluation question and meet the objectives of evaluation, evaluation criteria
with main evaluation questions and supported sub-questions are summarized as
followed:
|
Evaluation
criteria
|
Main
evaluation question
|
Supporting
question
|
|
Quality
Effectiveness
Sustainability
Lessons
learnt
|
1. To what extent trainings
increased PMI quality of services during emergency response in Lombok and/or
other relevant responses?
2. To what extent the
project has strengthened existing system for emergency response?
3. How well project
activities planned and implemented?
4. To what extent the
training model and strengthened policies sustain after the project close?
5. What are lessons that
can be learned by the project?
|
1.1 To what extent training alumnae applied
knowledge and skills they learned from the trainings into Lombok earthquake
response and/or other relevant responses? How far their responses adhered to
existing service standards? What factors contributed to different quality of services
among training alumnae during Lombok and/or other relevant responses (if
any)? Which specialized trainings (specialization) that have contributed the
most?
1.2 To what extent the existing training’s
monitoring and evaluation guide and tools relevant and applicable in practice
(Lombok and/or other relevant responses)?
2.1 To what extent the project has strengthened
the existing policies (SOPs, emergency plan, etc.) for emergency response?
What are key factors that influence the policy strengthening?
2.2 To what extent the strengthened services’
standard operational procedures (SOPs) and emergency plan have contributed to
quality of services during Lombok and/or other relevant responses?
3.1 How well the planned project activities
have been implemented? To what extent the project activities contribute to
the project objective’ achievement? What are external factors contributing to
the project implementation and achievement?
3.2 How well PMI and/or other relevant
stakeholders have been engaged and involved?
4.1 How far the training model/ approach have
been incorporated into PMI operation plan and adapted by other projects?
4.2 How far the strengthened policies have been
adapted/ bought by other parties?
5.1 What are lessons learnt on the capacity
development results from the Lombok and/or other relevant responses?
5.2 What are the lessons learnt on the
strengthened policies in practice (Lombok and/or other relevant responses)?
5.3 What are recommendations for prospective
PMI capacity development and emergency response system strengthening in the
future?
|
2.2 Evaluation methodology
The evaluation will measure
the benefits of the training results and strengthened PMI response system using
Lombok and/other relevant responses. Therefore, the evaluation preferably applies
qualitative methodology of which evaluators will use the existing standard
operational procedures, guides, and tools at Lombok as the case study. Also, the
selected consultant team are expected to develop other data collection tools
that enable them to answer all evaluation questions based on consultation with
PMI and American Red Cross. In the proposal, the consultant candidates should
be able to justify the proposed evaluation methodology with suggested
measurement strategy and its benefits for the evaluation results, analysis
plan, and strategy to reduce possible bias.
2.3 Main audience of evaluation
1. USAID/OFDA:
Donor
2. Indonesian
Red Cross: Implementer (at three levels: Branch, Chapter, and Head Quarter)
3. American
Red Cross: Supporting Partner (Operating Country, AMEE Region, and Head
Quarter)
2.4 Evaluation coverage
The evaluation will cover the Risk Management Policy and
Practice project sector that strengthens the national emergency response system
in the six regions. To understand how the project has benefitted the strengthened
response system and services, the evaluation will use Lombok earthquake and/or
other relevant responses as the case study(ies).
3. Scope of Work and Evaluation Design
3.1 Scope of work
- The selected consultant/firm will be responsible to ensure that the project final evaluation of Indonesian Disaster Management and Logistics Readiness: Emergency Water Supply and Strengthening Regional Logistics Capacity Project meets standard evaluation and research. The scope of work must include:
- Provide a complete inception report (project final evaluation proposal and tools). The inception report will be submitted once the candidate is selected. The candidates are expected to submit the draft project final evaluation proposal during administrative recruitment stage. The inception report will be developed under consultation with PMI and American Red Cross. The inception report must include: evaluation purpose, scope of work, evaluation question, key respondent, evaluation tools, detail work and budget plan, analysis plan, deliverable, additional information, and annex (es) for all tools developed and agreed.
- Develop evaluation tools, including interview/discussion guides and other supporting data collection tools in accordance with the agreed measurement strategy and key respondents.
- Lead data collection and coordinate with PMI throughout the evaluation process. Selection of key respondents and field visit to Lombok will be proceeded in close coordination with PMI and American Red Cross Monitoring & Evaluation division.
- Lead lessons learnt workshop with PMI. The consultant must lead lessons learnt workshop as part of data collection process. The consultant must ensure that the key findings are discussed and verified by the Red Cross team. Please refer to utilization focused evaluation as implementation guide.
- Perform data management, quality assurance, and data analysis. The consultant must be able to ensure the quality of data collected (both primary and secondary data) and perform data verification and triangulation for internal validity. The consultant is expected to update weekly progress based on the agreed evaluation work plan..
- Write and present the project final evaluation. The consultant must present accepted Project Final Evaluation Report (limited to key findings, best practices, and lessons learnt to be presented) to PMI and American Red Cross as the main audience of evaluation.
- Write a short success story of Lombok and/or other relevant responses. The consultant must produce the success story, no later than 4 pages, that is based on the evaluation results.
3.2 Logistic and administrative support
The consultant is expected
to use her/his own computer. Approved administrative and logistic costs will be
reimbursed by the Red Cross Team. The
consultant will be able to work remotely, after approval of the inception report.
3.3 Reporting relationship
The consultancy service
agreement would be facilitated by the American Red Cross through IFRC. The
consultant will report to Madiatri Silalahi, Monitoring & Evaluation
Division from American Red Cross, and Hanna Kusumastuti as the Project
Coordinator from PMI Disaster Management.
3.4 Ethical guidelines
It
is expected that the evaluation will adhere to ethical guidelines as outlined
in the American Evaluation Association’s Guiding Principles for Evaluators.
Below is the summary of the ethical guideline (taken from www.eval.org/Publications/
i.
Informed
Consent: All participants are expected to provide informed consent
following standard and pre-agreed upon consent protocols.
ii. Systematic Inquiry: Evaluators
conduct systematic, data-based inquiries.
iii.Competence: Evaluators provide
competent performance to stakeholders.
iv. Integrity/Honesty:
Evaluators display honest and integrity in their own behavior, and attempt to
ensure the honesty and integrity of the entire evaluation process.
v. Respect for People:
Evaluators respect the security, dignity, and self-worth of respondents,
program participants, clients, and other evaluation stakeholders. It is
expected that the evaluators will obtain the informed consent of participants
to ensure that they can decide in a conscious, deliberate way they want to participate.
vi. Responsibilities for General and Public Welfare:
Evaluators articulate and take into account the diversity of general and public
interests and values that may be related to the evaluation..
3.6 Future use of data
All collected data will be
the sole property for the American Red Cross. The consultant may not use the
data for their own research purposes, nor license the data to be used by
others, without the written consent of the American Red Cross.
4. Expected Deliverables and Duration of Consultancy
4.1 Expected activities and deliverables
i. Inception Report (Final proposal and tools) prior data
collection
ii. Conduct initial briefing and document review
iii. Finalized Evaluation tools
iv. Perform data collection
v. Implement data management, quality control, and analysis
vi. Provide updates of weekly/biweekly evaluation progress
vii.
Lead lessons learnt
viii.
Submit draft report
ix.
Submit signed inform consent (using American Red Cross form)
and photos taken throughout Evaluation process
x.
Provide all data collected used for data analysis and
reporting (sealed transcript and records, analysis table/working paper,
etc)
xi.
Submit Final Report (in English and Bahasa Indonesia). The
Final Report should include contents as follow:
-Cover with title
identifying Performance Evaluation. For the cover format, please refer to https://www.usaid.gov/sites/
- An abstract of not more than 250 words briefly describing
what was evaluated, evaluation questions, methods, and key findings or
conclusions
- Glossary and Acronyms
- Table of contents
- List of tables
- List of graphs
- Executive summary 2-5 pages in length that summarizes key
points (purpose and background, evaluation questions, methods, findings, and
conclusions)
- Introduction covering purpose of, audience for, and
anticipated use(s) of the evaluation, the evaluation questions, evaluation
method(s) for data collection and analysis including actions to reduce
potential bias, limitation of the evaluation method(d)
- Project Background that briefly provides information of
country and/or sector context, specific problem opportunity the intervention
addresses; and the development hypothesis, theory of change, or simply how the
intervention addresses the problem
- Key evaluation findings
- Lessons learnt
- Conclusions
- Recommendations
- Reference
- Annex including TOR, Inception Report, Evaluation Tools, Any
“statements of difference” regarding significant unresolved differences of
opinion by funders, implementers, and/or members of the evaluation team, Signed
disclosures of conflicts of interest from evaluation team members, and Summary
information about evaluation team members, including qualifications,
experience, and role on the team.
- A short success story of Lombok or other operation responses
4.2
Duration of consultancy
The consultancy will be
provided within 45 working days, starting from 8 October to 10 December
2018.
5. Required Qualification
Required qualifications for the consultant team
is:
- Five years of experiences conducting and leading qualitative researches/evaluations, 3 of which should be with international organizations.
- Excellent knowledge and skills in qualitative methodology. Be able to show experiences in designing evaluation, managing, and analyzing data.
- Experiences in leading evaluation particularly for utilization focused evaluation and evaluation of system strengthening projects.
- Experiences in story writing as part of Evaluation products shown by sample of stories from previous evaluations.
- Demonstrated expertise in capacity development, emergency response, and logistics as determined by working experience and would be plus with supporting educational background.
- Demonstrated high quality report (in English and Indonesian language) as shown by sample of Evaluation reports.
- Previous evaluation experiences with PMI and American Red Cross are desirable.
6. Application Materials and Procedures
6.1 Application materials
The consultant candidates are expected to submit the proposed Project Final Evaluation Proposal with table of content as followed:
1.
Consultant
profile
This section includes summary of the Evaluation team,
explaining all members’ education background and relevant working and
evaluation experiences. Details of each members’ background etc. must be summarized
in CV as part of Annex.
2.
Evaluation
purpose
Stating the evaluation purpose that refers to the TOR.
3.
Scope
of work
Stating the scope of work that is cited from the TOR.
4.
Evaluation
methodology
Completed this section with a table comprised of Evaluation
criteria, main questions, supporting questions, proposed methodology and
measurement strategy. Provide narrative explanations for the table, including
justification for the proposed measurement strategy and suggested selection
criteria for key respondents. Please look at the Sub-section 2.2 “Evaluation
Methodology” for further explanations required on the Evaluation proposal.
5.
Data
management and Quality control
The consultant team must explain strategy to ensure data
quality, data security, respondents confidentiality, and proper data
management.
6.
Analysis
plan
The consultant team is expected to explain its data analysis
plan, software used, and working duration. The consultant must be able to
explain strategy to avoid potential bias from the proposed evaluation
methodology and measurement strategy.
7.
Evaluation
timeframe
Complete this section with proposed work plan. Provide a
table comprised of description of Evaluation activities, estimate days of
implementation, and evaluation team member responsible for each planned
activity.
8.
Evaluation
budget plan
Provide a table that shows planned evaluation activities,
estimate days of implementation, responsible evaluation team member, and unit
cost.
9.
Reference
Provide at least 3 references with details as follow:
-
Name of reference
-
Position and organization
-
Mobile phone number and email address
-
Previous consultancy service
10..
Annex
The Annex should be consisted of:
- CV
of each Evaluation team members
- Sample
of previous Evaluation report in English comprised of: Executive Summary (1 page), Analysis results and key findings (4
pages), Recommendation (1 page), Conclusion (1 page), and a sample of stories.
6.2 Application procedures
Please submit your proposal
application in English not later than 11 November 2018 to recruitment@amredcross.org
with title Policy Sector Project Final Evaluation Application in
the subject line and label your CV, proposal, and sample of Evaluation report
with your name and document type in the file names. Only short-listed candidates will be notified
and contacted for an interview. Applications received after the deadline or
incomplete applications will not be considered.
No comments:
Post a Comment