Prior to webinar I asked Jane if she wanted a cartoon, she said yes, so of course I created a whole set.

About the Illustrations

The expertise is Jane’s with the cartoons being my contribution.  If you would like to learn more about causation check out Better Evaluation.

I have a few things going on this week so I’ll pretty much be an absentee blogger.  So forgive my lack of prompt RTs and comment responses.  If my awesome loyal followers could do me a favor and reply to any comments and RTs you notice, I would be much obliged.

Oh, one more thing, the best way to follow this blog is via my email list.  Each time I post (about weekly) I send a short email.  Also, I have an idea for a special giveaway just  for my email followers and, as soon as I get the chance, I will make it available.

A few notes:

• If you like the post, write a comment and let me know.
• Share it with colleagues. Seeing people sharing my cartoons inspires me to create more cartoons.
• What do you have to say about causation? Let us know in the comments.
• Please feel free to use my cartoons in presentations, training materials, etc.

Defining Causation

First things first, I asked Jane to give us a quick definition (FYI: you can tell this is her response because of the “u” in favorite)…

My favourite definition of causation is Scriven’s from his Evaluation Thesaurus, 1991.

“Causation: The relation between mosquitoes and mosquito bites. Easily understood by both parties but never satisfactorily defined by philosophers and scientists.”

I don’t have a formal definition I use myself, but usually explain it by saying you can’t do outcome or impact evaluation unless you know that those things you are documenting as “outcomes” and “impacts” actually “came out of” or “were impacted by” the program/policy/project/etc.

If you don’t have any evidence they did, then all you are documenting are coincidences. And what’s the point of that?

Correlation is not causation

Jane’s key point: You can’t actually do an impact or outcome evaluation without causal inference.

Over-complicating causality

Jane’s key point: We really do seem to be massively overcomplicating this space by thinking it’s harder than it is. We make causal inferences every day in our real lives and the way we do it is often quite sound. So let’s build on that, find ways to do it more systematically, transparently, rigorously, defensibly.

Real gold standard

Jane’s key point: Causal inference – like evaluation – is more about reasoning than methods; the real Gold Standard is sound causal reasoning backed by whatever evidence, whatever mix of methods you need to make the case for the audience you are speaking to (and using the evidence you can cost-effectively get your hands on).

Non-experimental and qualitative causality

Jane’s key point: Causal inference CAN in fact be done with non-experimental and even with purely (or heavily) qualitative methods. Counterintuitive for those raised in the traditional social sciences, but it really is true. [It also means you are not off the causation hook if you do qualitative evaluation!]

Never 100% certain

This specific point Jane didn’t send to me, but she has addressed it in the past (PDF) and I like posting at least five cartoons…

The causal link doesn’t have to be demonstrated to 100% certainty; we need to match the level of certainty to the decision making context (not throw in the methodological kitchen sink).

Additions

What else can you add to the discussion?

I asked Stephanie about some of the common mistakes she sees on a regular basis when working with slide decks and about the steps presenters should take to fix the mistakes.  Her answers sparked this post.

About the Illustrations

Stephanie gets all the credit for the mistakes and solutions, the cartoons are my contribution.

Before we get to the cartoons, I want to extend a big THANK YOU to everyone who endorsed me on LinkedIn following last week’s post. Your encouragement keeps me cartooning!

A few notes:

• If you like the post, write a comment and let me know.
• Share it with colleagues. Seeing people sharing my cartoons inspires me to create more cartoons.
• Are there any other common presentation mistakes that bug you? Let us know in the comments.
• Please feel free to use my cartoons in presentations, training materials, etc.

Presentation mistake: Slides crammed full of text.

Stephanie’s solution: Edit like a boss and save the real content for what comes out of your mouth. Limit slide text to keywords.

Presentation mistake: Slides with background textures

Stephanie’s solution: The safest bet is a solid background – either very light or very dark. Slide backgrounds are not wallpaper – they should be used to set a mood.

Presentation mistake: Using cliche images

Stephanie’s solution: Dig deeper to get beyond the cliche. Think about real life metaphors that might apply. Pull together a few people from around the coffee machine and ask them what visuals come to mind when you say a few keywords about your topic.

Presentation mistake: Images that conflict with the content

Stephanie’s solution: Unless you are intentionally trying to confuse people, match the mood of the image to the content of the presentation. If you can’t find anything that works, it’s better to use no image at all.

Presentation mistake: Highlighter-bright colors

Stephanie’s solution: The spirit of your favorite bright color can still live on – just add more gray to the mix so it is darker and less irritating to corneas. When you see the slider bar that lets you adjust the color, drag it down into the dark range.

Presentation mistake: Tons of references

Stephanie’s solution: Raise your right hand and repeat after me: I will cut the references from my slides and paste them into a handout.

Other common mistakes

What other presentation mistakes do you see on a regular basis?  Let us know in the comments and maybe you’ll spark a new cartoon.

Update 1: Presentation Mistake: Dataviz for the sake of Dataviz

This update based on Isaac Castillo’s comment below.

Isaac’s guidance:  If it takes more than 15 seconds for a person to look at your slide to understand your data viz message, then your message is not clear.  Simplify it, make it a handout, or chunk it out into multiple graphics (each one with a simple message).

Lack of data literacy is biggest barrier to #eval. Everyone in sector should know what a counterfactual is – yet many “evaluators” don’t.

— David Henderson (@david_henderson) April 23, 2013

That sounded like a challenge.  So I reached out to David to ask if he would like to collaborate on a cartoon post covering the subject; he graciously accepted.

About the Illustrations

The tips and expertise are courtesy of David.  The cartoons were developed by yours truly.

Could you do me a favor?  If you like the cartoons, could you connect with me on LinkedIn and endorse me for cartoons?

A few notes:

• If you like the post, write a comment and let me know.
• Share it with colleagues. Seeing people sharing my cartoons inspires me to create more cartoons.
• If you think I’m missing critical pieces to the overall discussion, let me know in the comments.
• Please feel free to use my cartoons in presentations, training materials, etc.

What is a Counterfactual?

Impact is the difference between the outcomes of an individual who participates in a program versus the outcomes for that same individual at the same point in time had they not participated in the program. Since the same person cannot simultaneously be enrolled and not enrolled in a given program at the same time, impact evaluations are concerned with estimating the missing counterfactual, which is an estimate of what would have happened to an individual had they not participated in a program.

In robust evaluations, counterfactuals are estimated by randomly assigning some people to a program treatment group, and others to a control group. For a brief discussion of impact evaluation and the role of counterfactuals in evaluation, see this page by The World Bank.

Developing a Counterfactual

Economy

Your program is likely not to blame for a sputtering economy, nor does it deserve credit for an economic upswing. The key to a good evaluation is to try to isolate the effect of your program, irrespective of external factors.

Imperfections

If you don’t acknowledge the imperfections in your data and try to estimate a counterfactual, your program officer will. You’re better off poking holes in your own data before any one else gets the chance.

Results Too Good to Be True

If your results sound too good to be true, you are more likely to have made a mistake in your evaluation than to have discovered you’re a genius.

Outliers

Outliers do not make for compelling client testimonials. Use your metrics to identify what the average experience in your program looks like, and get testimonials from people who fit this profile.

Comparison Group

100% of successful people succeed. Losers are a terrible comparison group for winners. Make sure your comparison group is identical to your treatment group, with the only difference being participation in your program.

Your Thoughts?

What thoughts do you have on the subject of counterfactuals?  Is there anything you would like to add to the discussion?  Let us know in the comments.

I really think you're going to like the posts but I'm not going to rush them out before they're ready. If you want to make sure you don't miss the posts, click here to sign up for my email updates. But to tide you over, here's a new compilation.

I was unable to make it to the 2013 Eastern Evaluation Research Society conference but several of my cartoons went in my place. From what I hear, my cartoons made an appearance in four different presentations at EERS 2013.

I love it when my cartoons take on new life outside of the blog. How many other forms of communication can look so good in power point slides, blog posts, newspapers, and hanging on a wall?

About the illustrations

All of these illustrations and cartoons have lived lives apart from my blog. While many are not new, most have never even made an appearance on freshspectrum.

A few notes:

• If you like the post, write a comment and let me know.
• Share it with colleagues. Seeing people sharing my cartoons inspires me to create more cartoons.
• Please feel free to use my cartoons in presentations, training materials, etc.

Dissertation Pit

Here's a cartoon I made for my colleague and friend Eden Segal. See her presenting in the tweet below.

@clysy makes an appearance at #eers13 twitter.com/jwasbes/status…— Jeff Wasbes (@jwasbes) April 16, 2013

Struggle between bad and good reports

This cartoon is from any early collaboration with the fabulous Ann Emery. Look below to see it being presented by one of my newer connections, Jamie Clearfield.

So glad to have finally 'met' @jamieclearfield yesterday at #eers13. Here she is, presenting w/ @clysy's cartoon: twitter.com/AnnKEmery/stat…— Ann K. Emery (@AnnKEmery) April 16, 2013

Evaluation Haiku

Here is another cartoon I drew for a Westat colleague/friend Jen Hamilton. The fantastic haiku is all Jen's.

Pie Distortion

The next two cartoons were commissioned by the awesome Stephanie Evergreen. (*Spoiler Alert* I may or may not be working on a new collaboration with the rockin ukulele player).

Power Point Hibernation mode…

Don't get too excited

This lucky cartoon is traveling with my fellow Durham Westatian (it's a word), Tamara Daley, to Spain for IMFAR 2013.

Not Rigorous

Not all of my cartoons live their life in presentations. The following couple were commissioned by the wonderful Kylie Hutchinson. At least one was destined for a training manual.

Market Evaluator

Be Unique

Those of you following me for a little while know I was cartooning a lot during last year's evaluation conference. Helen Salmon just plain stands out, which calls for a cartoon. This one now lives as Helen's twitter avatar.

Calling All World Savers

The fantastic Maria Gajewski reached out to me looking for a new profile image, here is what I came up with.

I meant midnight

I put this together when collecting Ignite presentations from session presenters. It ended up in the AEA newsletter.

Thank You Susan

This one I drew for Susan Kistler, another one that ended up in the AEA newsletter. Sitting on my iPad I have a string of cartoons drawn to thank those that inspire me daily, most of the time that's where they stay.

There are more

Starting soon I plan to release more cartoons than just the ones that show up in freshspectrum posts. I am constantly creating but not all the cartoons fit what I am trying to do on this site. If you want to see the extra cartoons join my email list, as I plan to release them exclusively via email.

Additions

I wasn't planning on adding any additional cartoons but after Sheila's sweet comment I had to add hers to this post.

Do you have a favorite definition? Is it the same definition used by your colleagues? How about your client’s definition?

Definitions can be fun topics of discussion, especially when the definitions we hold close don’t match the definitions of those around us.

About the illustrations

So I’m trying a discussion-focused approach to cartooning. I start by choosing an important but common topic of conversation. Then, with expert help, I research the topic to identify critical pieces of the overall discussion.

Once I understand some of the critical elements of the discussion, I cartoon. The cartoons are then posted on freshspectrum. This is not the end of the process. What I hope is that you, yes you, will also provide some input on missing elements.

A few notes:

• If you like the post, write a comment and let me know.
• Share it with colleagues. Seeing people sharing my cartoons inspires me to create more cartoons.
• If you think I’m missing critical pieces to the overall discussion, let me know in the comments.
• Please feel free to use my cartoons in presentations, training materials, etc.

Defining in Code

Sometimes it’s easier to define using metaphor, especially when you’re trying to paint a picture of every part of the field. Roger Miranda’s children’s book Eva the Evaluator gave me the inspiration for this cartoon.

Metaphorically speaking

Research and Evaluation

Jane Davidson wrote a nice post on Genuine Evaluation about the difference between evaluation and other pursuits.  A few lines stood out and inspired the following cartoon.

The fundamental difference is that evaluation asks and answers questions about the quality, value, and/or importance of things (design, implementation, outputs, outcomes, impacts, the project/program/policy/etc as a whole, and so on).

If we’re not doing that, we’re not actually doing evaluation.

Systematic Assessment

I reached out to colleagues to get their favorite definitions.  If you have one, let me know in the comments.  This definition was provided to me by David Bernstein.

“Program evaluation is the systematic assessment of program results and, to the extent feasible, systematic assessment of the extent to which the program caused those results. Evaluation includes ongoing monitoring of programs, as well as one shot studies of program processes or program impact. The approaches used are based on social science research methodologies and professional standards.”

You can find the definition here Newcomer, Hatry, and Wholey, 2010, p. xxxiii.  It inspired this cartoon.

Merit, Worth, and Value

Stephanie Evergreen was also kind enough to give me a favorite definition.  I’m going to go out on a limb and guess that Stephanie is not alone in liking this one by Michael Scriven.

@clysy Scriven: merit, worth, and significance

— Stephanie Evergreen (@evalu8r) April 9, 2013

Additional Cartoons

So what am I missing?  Do you have a definition that you love but don’t see reflected in the above illustrations.  Let me know, it may just lead to a new cartoon.

Update 1

Here is a new tweet inspired cartoon…

Hi @clysy love the metaphor one especially!’What is it we said we were going to do in that grant application?’

— Whitebarn Consulting (@SandraVelthuis) April 23, 2013

Update 2

Here’s a heavily cited evaluation definition by the late Carol Weiss.  Thanks for the prompting Sheila and David.

It’s an easy question to answer when talking about the blender you picked up at a neighbor’s garage sale.

The question’s just a bit tougher to answer when asked by a funding agency trying to save the world through a comprehensive intervention.

“Does it Work?” is the kind of question that fuels the field of evaluation.  Everyone involved in an intervention wants to know that they accomplished something and who wants to fund something that doesn’t work?

So how do you approach the discussion?

Below you will find 5 cartoons I’ve designed that speak to the issue of attribution.  These are discussion tools, not answers.

About the illustrations

This post was inspired by Kylie Hutchinson, who gave me the idea and provided me with resources I could use to dive into the issue. The primary resource I used for this post was [PDF] Addressing the Question of Attribution in Evaluation by the Evaluation Unit of Canada’s The International Development Research Centre.

A few notes:

• If you like the post, write a comment and let me know.
• Share it with colleagues. Seeing people sharing my cartoons inspires me to create more cartoons.
• If you think I’m missing critical pieces to the overall discussion, let me know in the comments.
• Please feel free to use my cartoons in presentations, training materials, etc.

Separating from the crowd

One reason attribution is so hard is that there are often so many additional factors at play.

Impact at the micro level

Did my contribution accomplish anything?

Modeling a comprehensive intervention

The more comprehensive the intervention, the harder it is to gauge its contribution.  It’s also much harder to explain and model.

Simple and complex

I tried to get at the difference between simple systems and complex systems.  I’m ok with this cartoon but it still doesn’t feel quite right.  What analogies do you use?  Let me know in the comments.

Doing the best with uncertainty

Additional illustrations

What’s missing from this post?  Let me know in the comments and I might just add more cartoons.

Additions

Susan Kistler was kind enough to offer some feedback on the simple/complex cartoon.  Here is a new version based on her comments.

According to Matt Keene two trees and three whales will be saved at the 2013 EEN Pacific Forum.  Since Matt didn’t specify in his post which whales, I thought I would help out.  Print it out, circle your choice, then deliver to Matt.

@evalcentral: 5 illustrations on attribution bit.ly/10dPrsV @clysy we’ll save 2 trees & 3 whales at #EEN13 ow.ly/jKduV…

— Matt Keene (@mwkeene) April 4, 2013

A better approach, in my opinion, is to write each post for a tiny audience. A post you know will be useful to someone but would not likely be publishable in any other format. It's the ability to get specific that makes the web such a potent publishing platform.