Does it work? 5 illustrations on attribution

Does it work?

It’s an easy question to answer when talking about the blender you picked up at a neighbor’s garage sale.

The question’s just a bit tougher to answer when asked by a funding agency trying to save the world through a comprehensive intervention.

“Does it Work?” is the kind of question that fuels the field of evaluation.  Everyone involved in an intervention wants to know that they accomplished something and who wants to fund something that doesn’t work?

So how do you approach the discussion?

Below you will find 5 cartoons I’ve designed that speak to the issue of attribution.  These are discussion tools, not answers.

 

About the illustrations

This post was inspired by Kylie Hutchinson, who gave me the idea and provided me with resources I could use to dive into the issue. The primary resource I used for this post was [PDF] Addressing the Question of Attribution in Evaluation by the Evaluation Unit of Canada’s The International Development Research Centre.

A few notes:

  • If you like the post, write a comment and let me know.
  • Share it with colleagues. Seeing people sharing my cartoons inspires me to create more cartoons.
  • If you think I’m missing critical pieces to the overall discussion, let me know in the comments.
  • Please feel free to use my cartoons in presentations, training materials, etc.

Separating from the crowd

One reason attribution is so hard is that there are often so many additional factors at play.

Impact assessment

Impact at the micro level

Did my contribution accomplish anything?

Micro level impact

 

Modeling a comprehensive intervention

The more comprehensive the intervention, the harder it is to gauge its contribution.  It’s also much harder to explain and model.

Modeling a comprehensive initiative

Simple and complex

I tried to get at the difference between simple systems and complex systems.  I’m ok with this cartoon but it still doesn’t feel quite right.  What analogies do you use?  Let me know in the comments.

Simple hose complex river

 

Doing the best with uncertainty

I asked Kylie if there was an expert who has influenced the way she approaches this discussion. Her answer, “Yes, John Mayne.” The following quote comes from page 16 of the  discussion paper [PDF] Addressing Attribution Through Contribution Analysis: Using Performance Measures Sensibly.

John Mayne

Additional illustrations

What’s missing from this post?  Let me know in the comments and I might just add more cartoons.

 

Additions

Susan Kistler was kind enough to offer some feedback on the simple/complex cartoon.  Here is a new version based on her comments.

 

According to Matt Keene two trees and three whales will be saved at the 2013 EEN Pacific Forum.  Since Matt didn’t specify in his post which whales, I thought I would help out.  Print it out, circle your choice, then deliver to Matt.

 

15 Comments

  1. leslieaj on April 4, 2013 at 9:48 am

    clysy EvaluationMaven I’m also a big fan of John Mayne on contribution vs attribution. Thanks for the cartoons. #outcomemapping #eval…



  2. clysy on April 4, 2013 at 9:50 am

    leslieaj thanks for the compliment 🙂



  3. susanjkistler on April 4, 2013 at 10:25 am

    Thanks @clysy ! @EvaluationMaven regularly inspires me as well. Love the cartoons. I tried to think a bit more about the simple/complex metaphor. How about watering with a hose versus watering with rain? The latter is relatively uncontrolled, less predictable, waters everywhere (not just where you want it), takes an entire system to replenish, etc. Not sure this does it. Am still thinking.



    • cplysy on April 4, 2013 at 10:33 am

      @susanjkistler  @clysy  @EvaluationMaven Thanks Susan, I like the whole rain thing better than the river I came up with.  A river is more complex than a hose but it’s not as complex as rain.



  4. AnnKEmery on April 4, 2013 at 10:39 am

    Once again, spot on! Great work Chris, and thanks to Kylie for providing inspiration behind this post.



    • cplysy on April 4, 2013 at 10:40 am

      @AnnKEmery thanks Ann 🙂



  5. Siobhan_Eval on April 4, 2013 at 10:54 am

    AnnKEmery omg I love these! http://t.co/ijGxJgZ1RV I’m totally using these to explain attribution issues to clients #eval



  6. AnnKEmery on April 4, 2013 at 4:23 pm

    david_henderson clysy Welcome to my office. http://t.co/ECpA8oY8fC



  7. SheilaBRobinson on April 4, 2013 at 7:08 pm

    Great post Chris! This is a topic I spend a great deal of time on when I teach my graduate Program Evaluation course. I love the blender analogy and the cartoons and will be sharing these with my next batch of students in May! Most of my work (and most of my students’ work) is in PK-12 education and the question “does it work?” (and its 1st cousin “is it effective?”)  is easier asked than answered.  My struggle often lies in explaining the challenges in assigning attribution to program effects to non-evaluators. My students get it, but only after they’ve been with me awhile! 🙂



    • cplysy on April 5, 2013 at 5:28 am

      @SheilaBRobinson thanks Sheila 🙂



    • cplysy on April 5, 2013 at 5:28 am

      @SheilaBRobinson thanks Sheila 🙂



  8. mattkeene on April 4, 2013 at 8:59 pm

    Now with all of the whale printing, we have to save another tree. And A-Hab-n’t you heard that its so difficult to save big and scary whales! What about a minke as big as your pinky?…As if I dont have enough to do. We’ll just aim for 2 whales next time so we are sure to succeed. Thanks chris.



    • cplysy on April 5, 2013 at 5:31 am

      @mattkeene that’s a good point Matt. Should I ask that, instead of printing out the sheet, people just call you and vote using the appropriate whale call?



  9. MariaGajewski on April 5, 2013 at 10:43 am

    Here’s a great example. I work with an organization attempting to reduce infant mortality. This graph http://theincidentaleconomist.com/wordpress/wp-content/uploads/2013/04/KindigFigure1.jpg shows that female mortality in the county they serve has improved dramatically in the past 20 years. How much of that improvement is due to this organization? I have no idea and no way to answer that question with any degree of reliability. Mortality is such a complex issue that it would take a chaos theorist to tease out all the confounding factors.



  10. susanjkistler on April 5, 2013 at 12:04 pm

    I want to save the narwhal.because they are so awesome.
     
    http://www.youtube.com/watch?v=ykwqXuMPsoc (guilty pleasure – the narwhal song)