Evaluation

Summary: A logic model is an attempt to show how your program is supposed to work.

Have you ever gone for a walk or a drive without a particular destination in mind?

I think it’s one of my favorite things. Because there is absolutely no way to get lost. When you wander the only time you can get lost is when you try to find your way home.

On the other hand if you’re trying to solve some kind of big social problem, it’s probably not the time to wander.  You should have a little direction to go with your purpose.

Logic models are tools that lots of evaluators use to visually show how a program is supposed to work.  If we take this action, we should see this result.

A basic logic model just looks like a simple flow chart.

Reverse engineering a program’s logic.

You might think that most logic models get drawn when a program is being designed. It sure looks like a plan of action. But I don’t think that’s actually true, at least not always, and probably not even most of the time.

Many programs get their start because a person or group is trying to solve a problem. Somebody’s hungry, they give them food.  A family gets evicted, they find them somewhere to stay.  They don’t always have some grand plan for how their actions are going to lead to desired results at a large scale.

If you’re an evaluator being tasked with the job of judging how a program works, you really need to start by asking a very basic question, “how is this program supposed to work?”

And that’s what a logic model does, it gives us a picture of how.  It’s a tool we can build by reverse engineering a program’s logic, usually through collaboration with the program staff.

The basic elements of a logic model.

There are all different types of logic models, but they usually share some similar components.

Inputs > Activities > Outputs > Outcomes

Inputs: What goes in (money, people power, time, etc).

Activities: The stuff you do (actions)

Outputs: Counting the stuff we did or people we served.

Outcomes: The consequences of our actions.

If there is anything that people get hung up on it’s probably knowing the difference between an output and an outcome. 

Let’s say you’re teaching an after school program.  Ten kids show up each night.  That would be an output. Did they actually learn anything, or retain more of what they learned earlier in the day?  That would be an outcome.

Let’s practice with something silly

You can create a logic model out of all sorts of things you do on a daily basis.  So practice creating one with low stakes.

I have a very food motivated dog named Jaina.  If anyone visits the pantry to grab a snack, she hears it and comes running.  Her hope is that the person with the snack might be willing to share with her.

Input: Snack food in pantry (if no snack food, nobody gets snacks).

Activity: After hearing a cracker box opening, join the family member at the pantry door. Give them your most pathetic puppy dog look.

Output: Followed two family members today to the pantry.

Outcome: One of the two family members tossed a delicious cracker that was thoroughly enjoyed.

Learners, what questions do you have?  Evaluators, what am I missing?  

These guides are meant to be conversational and easy to read.  But that means I leave out a bunch of potentially useful stuff.  If you have any lingering questions, or suggestions as to what should be added to this guide, please leave a comment.

Summary: A formative evaluation is about improving your program, activity, or intervention.  A summative evaluation is about showing proof that it works.

Let’s say you run a non-profit food kitchen.  You see your job as feeding the hungry people in your community.  You have a pretty basic strategy that involves collecting food from the community and distributing that food through a storefront you rent in a strip mall.

Your local chamber of commerce decides to raise some money to help you.  With some of that money you decide to evaluate your work and see how you could improve your strategy.  

This is what we would call a formative evaluation.  

A little while later you stumbled on some potential grant funding from a local philanthropy.  This group wants an evaluation that shows whether or not your program works.  That the funds they give you are directly leading to fewer hungry people.

This is what we would call a summative evaluation.

A formative evaluation is about improving your program, activity, or intervention.  A summative evaluation is about showing proof that it works.

The soup metaphor and what formative and summative have in common.

When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative

Bob Stake

I like this metaphor because it’s simple and useful. It shows us the difference between formative and summative but it also gives us another important element.  

What do both of these examples have in common?

Soup.

I know it sounds kind of silly, but it’s important.  The cook is making soup. The customer is eating soup.  Soup is the “intervention.”

Both formative and summative evaluations generally require an intervention (i.e. something specific to evaluate).  An intervention is a potential solution to a problem.

Back to our food kitchen example.  Let’s say you didn’t have a strategy.  You want to do something for people in your community who might not have enough food to eat.  You don’t quite know what you’re going to do, you just know there is a problem.  

There are certainly evaluation methods that could help, but without a potential solution you’re not ready for either a summative or a formative evaluation. 

Quick Activity: Formative, Summative, or Something Else?

Let’s end with a quick activity to help you determine next steps.  I find that almost everything we do can be broken down into a few basic elements.  See if you can break down one of your own activities.

What is your problem, challenge, or opportunity?

In my food pantry example above, the problem could have been a lack of affordable food options in the community.  In the soup metaphor, the basic challenge was about delivering a tasty meal to a customer.  An opportunity could be that somebody wrote you a big check to help you do something, but they’re letting you decide what that something is.

When we develop programs, activities, and interventions it’s usually in response to something.  So what is that something for you?

What is your solution, activity, or intervention?

In order to show that what you’re doing is working or not, you have to be able to identify the thing that you’re doing (your “soup”).  If you just have a problem, and no solution, then you’re not ready for a formative or summative evaluation.

And if you’re in a testing mode, trying out a variety of different solutions, you’re probably better off with a different type of evaluation approach.

What are your desired outcomes?  

Actions have consequences, positive and negative. An activity, intervention, or solution is a type of action.

A consequence of an action is an outcome.  Ideally, this is the positive change we hope to bring about.  But remember, we could also make things worse.  

Pulling it all together.

Can you fill this out?

We are facing _____ (problem).  Our solution is to ________ (action).  Ultimately that should lead to _______ (outcome).

A formative evaluation will attempt to show how you can improve the action in order to better lead to the desired outcome.  This can help you make the stuff you do better.

A summative evaluation will attempt to show whether or not the action did lead to the desired outcome.  This will show that you have a working solution and help you go and get more money to do more of it.

Learners, what questions do you have?  Evaluators, what am I missing?  

These guides are meant to be conversational and easy to read.  But that means I leave out a bunch of potentially useful stuff.  If you have any lingering questions, or suggestions as to what should be added to this guide, please leave a comment.