Does evaluation have a resource problem?

This post ended up being really long, with lots of data. If don’t feel like reading it, here is the TL/DR version.

• Evaluation does indeed have a resource problem.
• Government funding changes have made it worse.
• I have a plan to do something about it.

I wrote a bit about watching so many of our peers losing their jobs back in February. Unfortunately that was not just a temporary thing. This has been a hard year for lots of evaluators and researchers. Especially for anyone who works in or with the US federal government. But I can’t help but wonder about the longer term implications that will result from this shock to the system.

Here is what I know.

I’ll let other experts talk more about federal and contractor jobs lost. And I know the American Evaluation Association has been doing research on the impact on the evaluation workforce. So I’ll focus on what I know best, evaluation on the web.

I started blogging before I started cartooning and before I started designing. It was 2009 and I was a data specialist in a non-profit just starting to learn about about evaluation. At that time blogging was in its hey day. Twitter was still young with 140 character limits and no images. Instagram had yet to launch.

It wasn’t long before influencers across all platforms started to move more into social media, podcasting, and video. And while blogging today is not irrelevant, it’s not the dominant form of personal online publishing that it once was.

And if you’re wondering what this has to do with anything, keep reading and I’ll show you.

The information behind the internet.

Google and ChatGPT are not magic. They are just tools that allow us to experience lots and lots and lots of information. But they require input. And the better the input, the better the output.

This is why early in my career I spent a lot of time trying to recruit new evaluation bloggers. My first evaluation conference presentation was a pitch for new evaluation bloggers (you can actually still watch it on YouTube). I even created a website (Eval Central) to help ensure that new bloggers had access to an initial audience so they wouldn’t get discouraged and stop blogging.

A few of these early evaluation bloggers are names you will certainly recognize (Stephanie Evergreen, Ann K Emery, Sheila Robinson, and many more). And even though many of us ending up blogging more about data visualization, reporting, and presentation design, evaluation was the starting point.

Of course, the fact that many of us did move away from evaluation in our posts does point to a central issue.

Our Internet Body of Work Problem.

Patricia Rogers launched Better Evaluation in October of 2012. This evaluation resource site was built around a central framework, it drew content from expert contributors, and created a lot of high quality evaluation resources.

Because of all that work, if you were to ask Google a question about evaluation, often you would find your way to a page on the Better Evaluation website. Just like if you were to look for an evaluation cartoon you would likely find your way here to Fresh Spectrum.

Like with academic literature, where the goal is to build upon a collective body of work, the internet is its own body of work. These two bodies of work do overlap but the fact that so many academic institutions and publishers paywall their publications, keeps that body of work mostly off the web.

What this means is that the importance of a resource library, like the one on Better Evaluation, should not be undervalued. And while there are many other resource libraries. Unfortunately for all of us, many of those resource libraries are just not good at publishing web friendly content. And even those that are, just don’t publish enough.

A simple analysis.

I’ve given you a lot of (informed) opinions thus far, but we’re data people. So let’s go deeper.

I put together a list of evaluation resource sites. Just to note, I tried to go pretty wide but I know I’m missing a good number of sites. If you notice any BIG evaluation resource websites missing from this list let me know in the comments.

This list includes university resource sites, some consultancies, international efforts, big association sites, and more. I also put my website in for context but generally avoided most personal blogs that write a lot about other topics.

There are 52 sites here.

• AEA365
• African Evaluation Association
• American Evaluation Association
• Australian Evaluation Society
• BetterEvaluation
• Brown School Evaluation Center
• Canadian Evaluation Society
• Center for Evaluation and Educational Effectiveness
• Center for Evaluation Innovation
• Center for Evaluation Research and Methodology
• Center for Evaluation, Policy, & Research
• Center for Program Evaluation and Quality Improvement
• CREA
• CRESST
• Data for Impact Project
• Equitable Evaluation
• European Evaluation Society
• Eval Academy
• Eval and Ink
• EvalCentral
• EvalCommunity
• EvalPartners
• Evaluation, Assessment, and Policy Connections
• EvalYouth
• Expanding the Bench
• FreshSpectrum
• German Institute for Development Evaluation
• Global Evaluation Initiative
• Harvard Family Research Project
• IDEA Data
• Impact Entrepreneur
• Indigenous Evaluation Network
• Innovation Network
• Institute for Assessment and Evaluation
• International Program for Development Evaluation Training
• MEASURE Evaluation
• MERL Tech
• Office of Assessment, Evaluation, and Research Services
• Participatory Evaluation Network
• Regional Educational Laboratories
• TCC Group
• The Evaluation Center
• The Evaluation Group
• The Social Research and Evaluation Center
• Tools for Development
• U.S. Government Evaluation Portal
• UK Government Evaluation
• UN Evaluation Office
• USAID Evaluation
• USAID Learning Lab
• Utilization-Focused Evaluation
• What Works Clearinghouse

To gather the data I used a tool called Uber Suggest which looks at search engine data.

I only looked at two numbers. Domain or Page Authority AND Estimated Traffic for the month of June.

SEO Authority (i.e. Domain/Page Authority): is a composite rating that takes account of backlinks (websites that link to that website) and search placement (where a website shows up when someone searches for a keyword on Google). It’s a proxy for how SEO people guess Google treats website authority.

Estimated Traffic: Estimated traffic is based on the appearance and placement of the website in specific keyword searches. It then estimates the number of people who likely clicked on that link based on placement and the number of people generally searching for that term.

Most evaluation sites receive fairly low amounts of search engine traffic.

Not all sites are designed for search. Some resource sites will rely pretty heavily on email newsletters or other means to connect with their readers.

But generally, when it comes to web searches, a lot of resource sites get only a little bit of traffic through search.

Only 10 of the 51 sites (not including my own) likely had more than 3K visits last month from search.

These websites include Better Evaluation, Eval Academy, Eval Community, the American Evaluation Association, the Canadian Evaluation Society, and Tools for Development.

4 out of those ten have either been shut down or are at risk of being shut down due to changes in federal budget priorities.

The USAID Learning Lab is completely gone. Visiting the site will get you an error message. Note: In order to get some traffic data I used February estimates (instead of June) for this analysis. This is the only exception I made.

MEASURE Evaluation and Data for Impact are both still live. They are run by the Carolina Population Center at the University of North Carolina Chapel Hill. BUT…it is clear on both websites that they are the products of USAID funding.

Data for Impact’s last blog post was in November of 2024. The last updates I found on MEASURE Evaluation were from 2021. It’s possible that the site is one of many legacy websites produced with funding from the US government but not in continued operation.

The What Works Clearinghouse website run by the US Department of Education’s Institute of Education Sciences last news article was published in December of 2024. And as of March, IES is down to 20 federal employees, from 175 at the end of last year.

How does a website that rarely gets updated, maintain an estimated 20K views a month from search?

If you’re still scratching your head at how search statistics work, this might help.

As you can see, MEASURE evaluation has been able to accumulate an amazing number of backlinks over the years. Again, this is the number of links you’ll find on other websites that lead back to Measure. Because of that, it has pretty high domain authority.

Compared to other sites, 20K visits sounds pretty good. But that’s actually a low point for this site. Back in December of 2023 it had an estimated 71K visits.

So what does this all mean to a regular evaluator searching for information on the web?

Let’s say you type into Google “Evaluation News.” Based on the SEO Keywords you are likely going to see a link from MEASURE evaluation come up first in the list of non-promoted, non-AI, links.

Measure is also going to be the top source for most people looking for “Data Quality Assessment Tools.”

As long as the website continues to be live, and as long as it’s still useful, these posts will likely stay high in web searches. That is unless another agency or individual writes a better article.

What happens when we completely lose a website?

As I mentioned before, USAID Learning Lab was taken completely offline. If you go to visit the website, this is what you’ll get.

Let’s go ahead and look it in Uber Suggest. Even though it’s gone, it still shows pretty high domain authority.

But if we look at the traffic you’ll see it goes down to nothing from a high of 38K visits in August of 2024 (just a year ago).

Now let’s look at the backlinks. According to UberSuggest it has 97,690 from 3,667 unique web domains.

But let’s dig down and look at those individual links. One of the first ones that comes up is a link on the Wikipedia Page for Learning Organization.

That page still has that link as being live, along with AgriLinks, DRGLinks, Edulinks, and ResilienceLinks. All of these links are now dead ends.

Let’s look at another. Here is a link from a Report to Congress by the State Department on Evaluation Quality, Cost, and other matters.

Here is what you’ll find in the text of that report.

USAID: USAID is recognized as a leader in evaluation among federal agencies. USAID’s evaluation policy (note: this page is also dead as it lives on the USAID website) sets specific guidelines for how to conduct high quality performance and impact evaluations. USAID has a publicly available evaluation toolkit which provides step-by-step resources to staff designing and managing evaluation to ensure that they are of high quality.

It sounds like a pretty cool evaluation toolkit, too bad the link is now dead.

You may suggest that we could go back and use the internet archive’s wayback machine to recover some of these missing posts and documents.

And while that’s true, unless action is taken by the holder of the domain/website, those missing posts and documents will never be connected to the appropriate backlinks.

The loss of that one website just made 97,690 links dead ends.

Is this an opportunity for evaluation bloggers?

Technically, yes. It is an opportunity to fill gaps in the body of work that is evaluation on the web.

Only one problem. We have fewer evaluation bloggers now. And many of the highest authority bloggers have moved into parallel fields.

On the American Evaluation Association website there is a page that lists 92 evaluation blogs. That sounds good, except that the page is not really updated…ever.

Only 34% of evaluation blogs remain active, with 17% posting monthly (highly active) and 16% posting quarterly (moderately active). Nearly half (45%) are inactive, having not posted in over two years. Additionally, 16% have changed domains or moved to different platforms, while 5% could not be located at all.

And of those 34% of evaluation blogs that remain active are several blogs that rarely ever talk about evaluation.

What about our existing resource sites, like Better Evaluation?

Before I get into this, let me note something. I think Better Evaluation is the field’s best designed resource site. The internet is constantly changing through big shifts (hello AI) and little ones (periodic Google search algorithm changes). In that way, big changes in traffic could have absolutely nothing to do with organizational changes.

In other words, this is not a criticism of Better Evaluation’s new management, just a little analysis of the available data.

Better Evaluation became part of the Global Evaluation Initiative back in November 2022. Patricia Rogers, original founder and CEO, stepped down in August of 2021 after a decade at the helm.

So what did that mean for one of the biggest resources in the evaluation world?

First, a reduction in content. You can see it in the blog archives. In the three full years after Patricia left, Better Evaluation posted 28 blogs. In the three full years before Patricia left, the site posted 50 blogs.

The Better Evaluation YouTube page shows us a pretty steep decline in videos posted. Better Evaluation lead a number of webinars and then posted those webinars to YouTube. But there hasn’t been a new webinar for 3 years.

The site was never a video first website, it was a resource website. And as you have seen from the data above, it is still one of the most popular. But even as a resource site, we also see a huge drop in traffic since 2023.

So where are we now as a field?

It’s true that certain parallel fields (such as data visualization) seem to have become more popular than program evaluation. But at the same time, evaluation is more popular on the web now than it has been over the last 20 years. To see some of those relevant changes, just head to Google trends.

In case I haven’t made it clear.

I believe the field of evaluation has a resource problem. There is still demand for good resources, but less supply actually showing up in web searches. And if this is important to us as a field, I don’t think we can rely totally on amateur bloggers and creators to fill the gap.

Relevance in the Digital Age.

I’ve found that most people who do evaluation do not consider themselves evaluators (I call them evaluation-doers).

They are the program managers, non-profiters, researchers, health workers, and all sorts of other people who do evaluation work. They often don’t have lots of evaluation experience or academic training in evaluation, and becoming an evaluator is not their goal. So we can’t rely on associations or universities for everyday support.

They learn evaluation like people learn anything else, they go to Google. They type in questions like, how to do an evaluation for a grant, what is a developmental evaluation, how do I get better survey response rates, or how do I plan a focus group?

The quality of the sites that come back from the search will have an almost direct impact on the quality of data work done by the searchers. Showing up in those searches is one way for our field to stay relevant in the digital age. And we’re losing ground.

My new plans for Eval Central.

Here we are, full circle.

I’ve decided to do something about the gap in evaluation content. Because I know that we evaluators DO have agency in what shows up in web searches and on YouTube.

I plan to rebuild Eval Central. This time as a Community of Practice and Capacity Building Hub.

The other thing I’m doing this time is seeking funding BEFORE I launch.

My plan for the Eval Central CoP is a scaling up of a model I’ve been developing with the CDC for over the past 5 years.

And if it gets funded, here is what you can expect.

• Open (FREE) to ALL evaluators across the globe.
• Premium (FREE) evaluation training courses.  
• Monthly webinars (FREE) & office hours (FREE).
• Designed for evaluation practitioners.
• Also designed for people who do evaluation work but don’t call themselves evaluators (evaluation-doers).
• Built on a modern community platform designed to support Peer to Peer connection.
• Led by me, Chris Lysy of frehspectrum.com

Think Rachel Ray, NOT Le Cordon Bleu.

Now I need a favor. I’ve built a proposal but it would be helpful if I could show that there is interest in this kind of effort. So I also built a Community Waitlist.

If you are interested in this FREE community. Click on the link and add your email.

And if you know anyone at a philanthropy, business, or other organization that might be willing to help sponsor this new community, can you put them in touch with me?

In addition to sponsoring something that will provide a LOT of evaluation community value. I also have some pretty valuable sponsor incentives to offer.

Just send me an email or schedule a time to chat via my calendar (https://calendly.com/clysy/30min).