Systems thinking for evaluation design

Video transcript, summary, and concept map

In March, 2015, Bob Williams, the main author of Wicked Solutions and several other books on systems thinking, gave a 2-day workshop “Wicked Solutions: A Systems Approach to Complex Problems” on the use of the systems approach in evaluation design at the Research Institute for Humanity and Nature (RIHN) in Kyoto, Japan. RIHN is a development organization that conducts practical transdisciplinary studies of development problems and their solution. Some of the researchers participating in the workshop were involved in peatland management research on Bali and Sulawesi, Indonesia. Peatland is a major part of coastal wetland geomorphology around the world (4 million km2). The catch (22) is that peat grows naturally, but the process is reversed when the land is drained for agriculture. Peat covers half of the Netherlands, where the peatlands – which are often already below sea level – subside to ever lower levels, while sea levels are rising. A small part of the 2-day workshop was taped on video, edited, and posted on Youtube in two 30-minute parts youtu.be/lFcWhGE7moQ and youtu.be/5RRHpXl2hrw. They are not so much about peat but rather about the principles of systemic design. The transcription and slides have been combined in a single pdf. This post contains the concept map and summary. Have fun.

The video’s      … are in two parts: (1) introduction to systems thinking and the concepts; and (2) core concepts & the deep dive (that is, mostly of “Wicked Solutions”). In the concept map these two parts are clearly separated by a dotted line, yet the two parts are closely related. The introductory part provides the background for the second part. And the second part explains in adequate detail what the first part is mostly about.

Introductory part       Systems thinking is an interdisciplinary field of science and practice. As an academic interdiscipline it borrows from and contributes to almost all ‘straight’ disciplines including biology, philosophy, engineering, economics, cognition (i.e. psychology), computing, and management science. Systems thinking is special in that it integrates elements of ontology and epistemology. Sometimes it is more on the ontological side (e.g. systems analysis), sometimes more on the epistemological side (e.g. critical systems), sometimes it’s in the middle (management sciences, information systems). As a result of this enormous variety there is obviously no such thing as THE systems theory, there is just the “systems field” with a lot of ramifications in theory and practice that do not always same to have much in common with each other. There are a number of concepts that pop up quite often, e.g. planning, uncertainty, complexity, learning, (problematic) situations, and change.

The problem       Most people agree that systems thinking is a good antidote to the multitude of more narrow approaches to human reality. The trouble is that the systems field is so wide that hardly anybody knows where to start. Learning how to apply systems thinking in a practical way is itself is complex problem. People who need to learn systems thinking are usually experts in a field of their own (e.g. ecologist, soil scientist, mechanization expert etc.), each dealing with a subsystem of a larger whole. Below is an illustration of how an agricultural system is an integration of a natural ecosystem and a social system with about 20 subsystems (and counting, because I don’t see the peatland system, the irrigation and drainage system, the agricultural research system, the farmer communication system, the agricultural extension system etc.). My educational background is in tropical agriculture (BSc) and irrigation engineering (MSc), so I get really fired up about this. Below picture is from a set of slides from Ray Ison (here).

The solution       The problem of “allowing evaluation to become more systemic without having to adopt, accommodate and learn specific systems methods” was “solved” in 2004 by bringing a group of (about 16?) systems and evaluation experts together in Berkeley to explore “how to promote the use of systems ideas in monitoring and evaluation, not in terms of specific systems methodologies, but based on a set of principles. In other words, allowing evaluators to modify their existing practice rather than learning entirely new practice.” (Williams, 2016) After three days they came out of their room with three basic concepts: inter-relationships, perspectives, and boundaries.

Publications      From there Bob Williams was involved in a number of publications: (1) Systems Concepts in Evaluation (with Imam, 2007), with a first account of the three concepts (p. 6, only inter-relationships is termed “entangled systems”); (2) Systems Concepts in Action (with Hummelbrunner, 2011), in which 19 systems methods are described in terms of the three basic concepts; (3) Wicked Solutions (with Van ‘t Hof, 2014/2016), which provides a clear-cut and generally applicable methodology for applying the three basic concepts without any previous knowledge of systems thinking whatsoever; and (4) Using Systems Concepts in Evaluation Design (2016), which specifically applies the Wicked Solutions approach to evaluation (also available in Spanish). There is also a recent USAID discussion note on systemic monitoring (2014). The story doesn’t end there. The Systems in Evaluation TIG (Topical Interest Group of the American Evaluation Association) published a final draft of Principles for Effective Use of Systems Thinking in Evaluation in September 2018, which adds “dynamics” as key concept. This may not be necessary if we think of inter-relationships as a spatiotemporal concept, as has always been the case with C. West Churchman (1968, 1971, 1979). In 2014, Bob received the American Evaluation Association (AEA) Lazarsfeld Theory Award for his contribution to systems approaches in evaluation.

Wicked Solutions       … is by far the most practical of all these. In the bottom half of the concept map above its essence is explained. Each of the concepts could be considered a step in the process of designing an intervention, research program, or evaluation. Each step uses a set of tools and ideas to make it work: (1) rich picturing is used to map inter-relationships, which helps identify systemic structures, patterns, processes and dynamics, but also stakeholder values, norms, resources, conflicts, aspirations, goals, and motivations; (2) stakeholder analysis and stake analysis is used to identify possible framings (or “images”) of the problematic situation and possible purposes of the design. Since we are dealing with human, soft or social systems (which includes practically all systems, even the ‘hard’ ones), they can only be properly understood by discussing their purpose and meaning to the people involved; (3) critical heuristics is used to discuss boundaries. It helps us answer questions raised in the previous two steps by “revealing , exploring, and challenging boundary judgments associated with a situation of interest.” (Williams & Imam, 2007).

Participants        During the workshop the RIHN participants apply the Wicked Solutions methodology to their own research design in Indonesia. At the end four of them give comments. What they say is highly relevant. They consider the systems approach particularly useful in multi-stakeholder situations so as to enable them to make a more comprehensive design or project plan. Another issue is that the approach enables better stakeholder communication. The main problem is that it is difficult to use insights from the system approach to redesign an ongoing plan. Let me add a few comments to this: (1) the approach makes it less likely to ignore key stakeholders; (2) it is more likely that all stakeholders are heard and that they hear each other; (3) the key to application of the approach is to use it at as early possible a time. This will work much better if there is time to conduct project pilots or evaluation tests; and finally (4) there will always be learning as the activity unfolds. Learning will be better if the initial plan is more comprehensive, i.e. more systemic.

Final notes      In the workshop Bob uses cases from Latin America, Africa and Asia to illustrate the practical value of Wicked Solutions. They all happen to be in the field of agricultural development: (1) irrigated rice and malaria control in Peru; (2) irrigation development and pump procurement in Mali; and (3) peatland management in Indonesia. The Mali case is the detailed, worked-out case used in Wicked Solutions to ensure that starting systems learners can directly apply the methodology. Sometimes I use the words “solution” and “problem”. These terms must be understood in a systemic way, which means that most of the time there is not one, easily identifiable “problem”, nor is there one identifiable “solution.” For more information about that (i.e. wicked problems and the like), go here. Bob’s website is here.

This post has been reblogged by The Systems Community of Inquiry at https://stream.syscoi.com, the global network of systems thinkers, scientists and practitioners (Thanks!).

 

Advertisements

About csl4d

https://www.linkedin.com/in/sjonvanthof/
This entry was posted in General. Bookmark the permalink.

One Response to Systems thinking for evaluation design

  1. antlerboy - Benjamin P Taylor says:

    Reblogged this on Systems Community of Inquiry and commented:
    From Sjon van ‘t Hof’s always brilliant blog – and Bob Williams is always someone to look out for.

    Transcript, summary, and concept map
    During March 25-26, 2015, Bob Williams, the main author of Wicked Solutions and several other books on systems thinking, gave a workshop “Wicked Solutions: A Systems Approach to Complex Problems” on the use of the systems approach in evaluation design at the Research Institute for Humanity and Nature (RIHN) in Kyoto, Japan. RIHN is a development organization that conducts practical transdisciplinary studies of development problems and their solution. Some of the researchers participating in the workshop were involved in peatland management research on Bali and Sulawesi, Indonesia. Peatland is a major part of coastal wetland geomorphology around the world (4 million km2). The catch (22) is that peat grows naturally, but the process is reversed when the land is drained for agriculture. Peat covers half of the Netherlands, where the peatlands – which are often already below sea level – subside to ever lower levels, while sea levels are rising. A small part of the 2-day workshop was taped on video, edited, and posted on Youtube in two 30-minute parts youtu.be/lFcWhGE7moQ and youtu.be/5RRHpXl2hrw. They are not about peat but about the principles of systemic design. The transcription and slides have been combined in a single pdf. This post contains a concept map and summary. Have fun.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s