Additional menu

FreshSpectrum

Chris Lysy's Evaluation Blog

Articles

Watching and waiting.

Leave a Comment

I’ve been a bit pessimistic lately. Or perhaps I’ve just been a bit a realistic, which can be far more depressing.

In the past several weeks I’ve written and deleted a few different LinkedIn posts. I know that many of you appreciate the honesty in some of my comics, even the ones that feel a bit dark. But there is a point where it’s just a bummer.

I’ll try to do better, just maybe not today.

I didn’t attend this year’s American Evaluation Association conference. I haven’t since before COVID, but I do miss it. The other day I was looking at the conference schedule online and comparing it to the last couple of years. I pulled the full schedule for 2023, 2024, and 2025 then stuck into AI to compare.

  • Unsurprisingly, the number of conference sessions are down…by a lot (a little over half the sessions). I’ll let those of you who were there comment on the actual attendance and feel.
  • The USAID & international funding cuts are definitely reflected. This also seems to mean less of the sessions that talk more about big experimental evaluation approaches.
  • There are lot fewer sessions on storytelling. But if you look at past years you’ll start to notice that a lot of the past storytelling sessions were also talking about equity and inclusion initiatives. You know, all the DEI stuff that this US administration hates.
  • With all the government cuts and so many layoffs at big social science research organizations, the government program related sessions also seemed down a bit.
  • One thing that has certainly grown, is the number of sessions about AI. Can’t say that’s surprising.

Note: I got all of this is just by looking at the schedules. It might not be the reality on the conference floor. I’m sure there are some quirks that might make this method not wholly reliable.

I wonder how much the field of evaluation is really being hit right now. A lot of the people who present at conferences and contribute to journals, work in universities, government agencies, and big social research firms. All of these have been hit pretty bad by funding cuts.

But I also know that there are a lot of people who do evaluation work who are not part of the evaluation association. Many have probably never heard of the association. State & local governments, health departments, schools, non-profits, community foundations, and business social responsibility departments all employ evaluators. I wonder how it is in all of those worlds.

The Report Design Multitask Problem

Leave a Comment

One of the biggest report design frustrations is something that’s not on most people’s radar.

I talk about this in my book The Reporting Revolution (it‘s only $5, you should buy it!), but I’ve also talked about the concept in workshops. Here is a clip from my Everyday Report Design workshop.

Most evaluators and researchers try to write, design, and illustrate their reports all at the same time – usually the night before it’s due. This multitasking approach is why your reports feel like a struggle and often don’t look as good as you’d hoped.

There’s a better way: Split the work into three parallel tracks – writing, design, and illustration.

In this video, I’ll show you how to:

  • Separate your writing from your design work
  • Use placeholder text (lorem ipsum) to design before you write
  • Get approval on layouts early, before investing time in content
  • Reduce the late-night stress of making reports “look pretty”

This simple process shift will save you time and create better-looking reports.

Prefer the transcript?

Transcript

The Cassandra Curse

Leave a Comment

I kick off my latest book with the story about Cassandra.

I feel like being an evaluator or researcher or data analyst puts you in this position where you learn things. It then becomes your responsibility to share what you’ve learned with others. But if we fail to share it effectively, what’s the difference between us and Cassandra.

Here’s the intro.

Girl meets a god who gives her a divine power to see the future. The girl then turns down the god’s romantic advances. The god, who is clearly a creepy jerk, feels scorned and curses the girl. The girl can still see the future (her gift) but nobody will believe her prophecies (her curse).

We researchers and evaluators also have a gift. We can’t see the future, but through our methods we can measure the past and monitor the present. Then we can use that knowledge to project the future.

And like Cassandra we also have a curse. There is no guarantee that anyone will hear our words or believe what we share.

But unlike Cassandra, our curse is at least partially self-inflicted. We use the web to share our work in much the same way as we did in the late 90s. And we write reports almost as if the last 20 years never happened.

I believe we need a change, a monumental change. Not necessarily in our tools but in our mindsets.

Because the status quo is a path towards irrelevance.

The beginning pages of The Reporting Revolution.

The book is only $5 on Kindle ($10 in print), it’s a quick read and is packed with a lot of my best ideas around reporting. It also has a bunch of cartoons and reads more like a blog than a textbook. I’m proud of it and want to share it as widely as possible.

Buying the book also gets you an invite to a free webinar next month. Check out my post from the other day to learn more.

The PDF Must Die

Leave a Comment

For the last ten years or so I’ve taken a pretty measured stance on PDFs. Saying that they’re okay in moderation, perhaps even useful.

That’s the stance I take in The Reporting Revolution. I share three reporting strategies and the first is a modern PDF strategy. I kept it in because the book is really about making a transition and I know that most of you are not really ready to give up on the PDF.

But let me be completely honest. I think the PDF must die because it’s holding us back.

Back in 2014, Michael Quinn Patton reached out to me about creating a set of comics he could use in the fourth edition of his Qualitative Research & Evaluation Methods textbook. He added a “rumination” before each chapter and wanted a comic to go alongside each.

Number 5 was “Convenience Sampling is NOT Purposeful Sampling.”

He talks about how in a past edition he listed convenience sampling at the very end of a table of sampling strategies as the approach with the poorest rationale and lowest credibility. But when reviewing research publications and reports he would see methods sections that would state convenience sampling as “Patton’s purposeful sampling approach #15.”

I think PDFs are a bad habit we need to break. Like smoking and fast food. A single cigarette will probably not kill you and neither will a single big mac. And while a PDF isn’t really a bad secondary reporting tool, it’s way overused.

But I’m going to do more than just make the statement, in the next few months I’m going to systematically show you the reasons why I think…THE PDF MUST DIE.

The Reporting Revolution 2nd Edition (NOW AVAILABLE!)

2 Comments

Now Available! The Reporting Revolution (2nd Edition)

What’s new?

  • Cheaper! $5 Kindle, $10 Print
  • Completely re-illustrated (less stock, more drawings & comics)
  • Revised strategy section (easier to follow)

Launch Special!

If you buy the book in November, you’ll get access to a free modern reporting jumpstart webinar on December 4 at 11AM Eastern. I’ll even record the thing, so even if you miss it you can still watch. All you have to do is buy the book, then send me an email to let me know.

Contact info here > freshspectrum.com/contact

The Reporting Three-Step Method

Summary: In this post I’ll show you a quick way to set up your own continuous reporting system.

I’ve been working on my follow-up to The Reporting Revolution. I think I did an okay job of setting up my basic argument, that it’s time to change how we report and move beyond the PDF.

I focused a lot on the why, and was less clear on the how. While I shared a handful of reporting strategies, I didn’t come out and say which I thought was best.

But the more I work with organizations on their reporting, the more convinced I become that continuous reporting is not one of the ways. IT IS THE WAY.

The Reporting Shift

Continuous reporting is what you do when you report as you go.

It can be as simple as it sounds. The biggest barriers are not from the technology required, but from habit, lack of training, and silos.

In most organizations, the people who evaluate the programs are different from the people who write the blog posts who are different from the people who design the website. So what would be easy for a little organization, one that has just a few people who wear many hats, can be incredibly complicated for a larger one.

But for the sake of this post, let’s pretend it is easy. That you, the evaluator, have the power to do the following three things.

[Read more…] about The Reporting Three-Step Method