German Institute of Development Evaluation (DEval)

In 2012, DEval was created to carry out independent evaluations of Germany’s development interventions, providing the foundations for informed policymaking. It was set up by the Federal Ministry for Economic Cooperation and Development (BMZ) to provide objective advice, and its reports have covered German development investments in countries such as Afghanistan, Myanmar and Rwanda.

The challenge

In 2008, a two-year system review was commissioned to analyse the evaluation system in Germany.  Twenty German development cooperation agencies were studied. “This ‘system review’ provided recommendations for reforming the evaluation system as a whole. One of the main recommendations was to ‘establish an independent evaluation agency which advises and supports the BMZ on system development and the development cooperation organisations in the development of instrument and in improving quality’.” It appeared that there was significant duplication across the function in the federal German government and that a new organisation was required.

The initiative

In 2012, Deutsches Evaluierungsinstitut der Entwicklungszusammenarbeit [German Institute for Development Evaluation] (DEval) was founded to provide independent evaluations of the performance of German international development. “The overriding goal of DEval is to independently evaluate the performance of German development cooperation interventions.” [1]

The institute “will have 38 permanent employees and will be based in Bonn, Germany's main location for orga­ni­sations working in in­ter­na­ti­o­nal de­vel­op­ment”. [2]

Its more specific objectives are to:

  • Support the Bundesministerium für wirtschaftliche Zusammenarbeit und Entwicklung [German Ministry for Economic Cooperation and Development] (BMZ) and development NGOs to design development projects using evidence.
  • To present project results transparently and to learn from evaluations.
  • Support the German Bundestag in scrutinising the work of the executive and provide needed evidence for policy initiatives.
  • Promote evaluation capacity development in partner countries engaged in development cooperation.
  • Build strategic partnerships with academic, research and other organisations.

Its main functions are to:

  • Conduct evaluations, performance reviews and impact analyses of German development cooperation activities. [3]
  • Examine and refine relevant methods and standards.
  • Prepare and disseminate the findings of evaluations and methodological research.
  • Run training programmes in Germany and abroad.
  • Engage in national and international cooperation networks.
  • Foster evaluation capacities in cooperation countries

The organisation operates as an external body which has permanent staff that evaluates independently the activities of the organisations implementing development policy in Germany. “A German cabinet decision of July 2010 defines the scope of the institute's evaluation remit as ‘all measures for which the Federal Ministry for Economic Cooperation and Development (BMZ) is responsible’.” [4]

The public impact

By looking at the bigger picture from an independent and external perspective, the institute has been able to improve evaluation methods and standards and thus raise the quality of such evaluations.

For example, it carried out an evaluation of the German development cooperation with Afghanistan. There were two studies:

  • A study analysing the longitudinal impact of development cooperation in Northeast Afghanistan since 2007.
  • A strategic portfolio review.

Both these studies were used as an empirical basis for developing the new Afghanistan Country Strategy 2014-2017.

To take another example, DEval was tasked with evaluating a development initiative in Myanmar on the resumption of German-Myanmar bilateral cooperation in 2012.  “DEval was asked by BMZ to incorporate evaluative thinking and evaluation expertise into the overall planning process of the German-Myanmar Programme on Sustainable Economic Development right from the beginning with the aim to create the preconditions for an impact evaluation on programme level at the end of implementation.” [5]

What did and didn't work

All cases in our Public Impact Observatory have been evaluated for performance against the elements of our Public Impact Fundamentals.

Legitimacy

Public Confidence Good

Overall, DEval is perceived positively by the public and considered to be a competent organisation, although some of the institute’s operations have been criticised. “It was found that internal and external perceptions are essentially positive and coincide to a large extent  ... Staff members at DEval are perceived as professional and skilled. The degree to which staff networked and the extent of their experience received a rather low rating from external observers.” [7]

DEval’s overall approach its role in international development was applauded. “The institute's philosophy is seen in a positive light by all respondents, who used adjectives such as creative, modern, quality conscious and progressive. DEval's attitude is considered to be honest, inclusive, benevolent and objective.”

Stakeholder Engagement Fair

In its objective of scrutinising government work, the relevant government body 'took note' of five evaluations. This shows a lukewarm form of engagement from the government and a certain degree of trust in the evaluation. The first managing director, Helmut Asche, felt that BMZ was not interested in its evaluations and actually hampered its work. He was subsequently dismissed (see Alignment below).

“Whether and to what extent an evaluation is made use of and serves the management of programmes and processes was found to depend inter alia on whether a demand for the evaluation had existed and whether the responsible divisions had closely followed the evaluation process, in order to ensure that tailored information was provided for BMZ's use.” [6]

For the evaluations the organisation conducts, it engages the development cooperation agencies in implementing the changes.

In four of its evaluations the relevant organisations were involved in conceptual and instrumental changes. In this case, the stakeholders were the government agencies and NGOs for whom DEval performs the evaluations.

Political Commitment Good

DEval was instituted as part of the structural reforms in the development policy under the development ministry. Since DEval is an institution of the German federal government and has received its mandate from the Federal German Ministry for Economic Cooperation and Development (BMZ), it has solid foundations as well as a commitment to its independence.

Policy

Clear Objectives Fair

DEval's objectives were defined at implementation and have broadly stayed the same, that is, to evaluate the performance of German development cooperation interventions from an independent standpoint.

However, based on a recent evaluations they have decided to reevaluate and reconsider their objectives in light of their overall vision. [8] This indicates a certain lack of clarity. “Some objectives need to be formulated more succinctly or need a sharper thematic focus.

The findings also indicate, especially with regard to long-term objectives, that DEval needs to discuss how far its objectives are integral to DEval's identity and vision … One aspect of the basic alignment of DEval's portfolio is the question of training. This was raised in the interviews with the civil society organisations, where respondents clearly expressed a need for training to upgrade their evaluation capacity … In developing its long-term plans, DEval will decide the extent to which it can address this need.”

Evidence Good

The BMZ initiated the first ever systematic study of evaluations in the field of German development cooperation in 1997/1998. There was a second study in 2007/2008. The German Development agency studied existing agencies within the country to evaluate and formulate the definition and founding principles of a new institute. The critical analysis drew from all relevant evidence and thoroughly studied existing agencies, though only within Germany.

Feasibility Good

The creation of DEval was a government led initiative, demonstrating it is politically feasible. The institute is financially feasible as it receives funding from institutional grants agreed by the German Bundestag and paid by the BMZ’s departmental budget. This amounted to 5.09 million euros in 2014. [9]

Action

Management Good

The original head of DEval was the research scientist, Helmut Asche. He was dismissed in 2014 and replaced by Prof Dr Jörg Faust.

DEval has an Advisory Board, which Dr Faust reports to. The Board is composed of representatives from organisations such as the German Bundestag, the academic and research community, partner countries and NGOs. The Advisory Board advises the institute on mission-related activities, the design of the overall evaluation programme and the planning and implementation of evaluations.

There are five divisions within DEval: three evaluation teams, a methodology centre, and administrative services.

Measurement Strong

DEval undertakes a baseline survey of various dimensions of its own objectives, to serve as a benchmark for assessing the impact of its work in the years to come. The survey  looks at two areas:

  • DEval's work in conducting strategic evaluations.
  • Methodology research - DEval's investigating methods, which are measured by its Competence Centre for Evaluation Methodology.

The evaluation and measurement of external development projects forms one of its core skills, and it is able to apply these skills to reflect on its own activities.

Alignment Fair

There does not seem to be a defined follow-up process by the development bodies, which affects the impact of DEval's evaluations:

  • The sharing of responsibilities between the BMZ's Evaluation Division and DEval is not clear.
  • The implementation of evaluations by the agencies rarely involves persons from the project regions.
  • Majority evaluations were reflected upon by the development agencies, but only given serious attention if it directly affected them.

Although DEval is an independent institute, its managing director is chosen by the BMZ Minister. Ashe was appointed in 2012 by the then Minister, Dirk Niebel. He was dismissed in 2014 by Niebel’s successor, Gerd Müller, whom Asche accused of hindering his work. There was considerable frustration within DEval “over the chronic shortage of staff, the workload and especially about the dispute between the former DEval managing director, Helmut Asche, and the BMZ Minister”. [10]