So I’ve heard a lot of disdain for long reports over the years. They’re easy targets and get picked on often (even I take a shot every now and again).
But here’s a secret. I actually kind of like them (shhhhh, don’t tell anyone).
It’s not for the reading, I can barely even make it through a long blog post, but for the incredible opportunity that long reports provide for designers like myself.
Here, let me explain.
What is the goal?
The goal for a long evaluation report is usually to provide a systematic comprehensive review of the evaluation and its findings. This means lots of detail on questions, methods, evidence, findings, and recommendations.
The report needs to be strong enough to stand up to challenges. And in such an important and often politicized field, evaluations will face challenges. We are talking about value, merit and worth here, that’s always contentious stuff.
Who is the primary audience?
Most of the time there are two.
- The primary evaluation client. Who often require, sometimes by statute, lots of detail and context. Even when they pine for simple.
- The evaluation team. The comprehensive report writing process is a vetting and sensemaking process. It’s often where an evaluation comes together for an entire team.
The problem.
The problem is that far too many evaluation teams and clients take the final comprehensive report as being the thing for the public. Kind of like saying…
“Here you go large diverse audience, read this long boring report.”
But it’s a horrible experience for lots of those audiences. So horrible that many users will avoid almost anything with a “report” label.
Some of the audiences that won’t like it…
- People who use their smart phone to read things
- People who use their tablet to read things
- Casual report readers
- “Visual” people
- People with specific needs from the report
- People who are generally overwhelmed by the massively overwhelming nature of our overwhelming contemporary society
- Children
- Politicians (sure they won’t be happy without the comprehensive, but they likely won’t read)
- Nonprofit board members (see politicians)
- Bloggers
- Social mediaites
- Your family
- Your boss
- People who don’t like evaluation jargon
- People with short attention spans
- Some members of the evaluation team
- Many members of the client organization
- People who write long random lists
- Most other people
The opportunity
What if we’re looking at it the wrong way?
What if the final comprehensive report is just a step in the reporting process? And one that doesn’t come at the end.
Let’s look at what we have, which is 200 pages of heavily vetted public evaluation findings, data and evidence.
For someone like me, a user experience designer who has spent well over a decade as a researcher and evaluator, the final comprehensive report is a bumper crop. Ready to be harvested and transformed into high value products.
With just a little adaptation (and really, it only takes a little sometimes) we can turn the long boring comprehensive report into countless:
- Blog posts
- Infographics
- Dashboards
- Cartoons
- Social media images
- Social media posts
- Interactive reports
- Automated email sequences
- Explainer videos
- Animations
- Motion videos
- Activities
- ebooks
- Podcasts
- Webinars
- Tutorials
- Visual paper reports
- Posters
- Kid’s books
- And other stuff most evaluators are not used to creating.
Have an important report that very few people are reading?
Let’s talk about using my UX design process to turn it into something of value for all of your audiences.
If I haven’t convinced you, and if you still think we shouldn’t create these behemoth reports, let me know in the comments.
Great post, Chris! I’ve been thinking of more interactive ways to present evaluation findings. I’ve been dabbling into infographics as a starting point. Do you have any thoughts on replacing the standard executive summary with a one-page infographic inside of the report itself?
Thanks Nicole. Integrating visuals, like the infographic you mention, is a nice way to make a report more readable. And executive summary infographics are some of the most straightforward types of infographics you can create. It also gives you a visual piece that you can detach and present to different audiences as a stand-alone.
If the plan is to keep it exclusively in the full report, you’ll still scare away many audiences that will only look at the page count/file format and won’t bother to read.
I so much agree! Thanks for this post…. The usual suggested length for a report (20/30 pages) is completely useless. Too long not to be boring, too short to contain meaningful findings and the evidence needed to ground and make sense of the recommendations. So, in the end, all conventional reports look the same, because the narrative and conclusion in a “medium side report” lacks the needed details.
I had evolved the concept of a “mother report” that is a large systematized aggregated of findings. And I am trying to invite users to work out, together, the final products from it. In some cases I have been lucky to find people who wanted to engage and really appreciated the “sense making” that went into the mother report. And that derived interesting and diverse outputs from it. In other cases I was frustrated by people asking simply for a shorter version and for “recommendations”… You really need to know your audience. Some people are not accustomed to think and just want to have some recommendations they can quote!
A mother report, even if BIG, does not need to be read all. If it is well structured, people can just go through it and chose what to read and what to skip… But they have all evidence in one place. For these people who are really interested in the evaluation, and who care about improving, it has proved to be really useful.
Thanks Silva 🙂
I really like the term “Mother Report”.
My hope for the future is that the full comprehensive report will become a compilation of lots of little reports, presented in many different formats, and delivered throughout the evaluation process. Integrating intended evaluation user feedback loops throughout the entirety of the evaluation process and iterating the report based on feedback.
Most of the projects I work on are not there yet. So the knee jerk tends to be, “eep our report is too long, let’s make it a little one.”
Let’s hope so… the challenge is then to make people appreciate that analysis and reporting are much longer processes that they are used to… And that require more interactions. But the benefit is great, because evaluation can then seamlessly link to also to diverse things as training, communication, advocacy initiatives…
Exactly, it can become useful and appreciated. Instead of burdensome with limited value produced.
Only way to prove is through example with clear evidence on the benefits.
An this requires a lot of commitment… because producing a mother report and then facilitate or engage in the creation of additional products takes a lot of time. Because people do not believe if they not see, I ended up to do it anyway, beyond the paid work, to prove the point. Which is risky, because sometimes the mother report was not really valued. But every time you start afresh… even if you have something to show, people are so used to the old ways of doing things, that they do not understand how revolutionary is a process of creation of systematized evidence, rather than the production of some recommendations. Reporting time always end up being squeezed and the commitment of these commissioning the evaluation not always translates in the buy in of other stakeholders… Any idea about how to go about this is appreciated!
Here is how I would approach your mother report. Think of the mother report as a skeleton. Sketch out the outline at the beginning of a project. Then build it iteratively by integrating deliverables into every stage of the evaluation process.
Since deliverables are often necessary even at early stages, you don’t have to sell the mother report. It’s just a “value add” at this point.
Design each deliverable as if it is not not fully independent. Comprehensive is built slowly through time with every individual specific piece. But you can also go back to the growing aggregated mother report at anytime to see the evaluation from the top down.
YES! As a consumer I like reports so that I can find the detail on what I am interested in. I want the short report with references to where to find the detail in the long report.
I also find it useful as the evaluator for at least three reasons.Without the long report, I have to develop a sub-report when I am asked for details on some part of the findings. It is also useful when a year or two of five or ten down the road I am trying to remember what I did and what came of it. I find the long report even more useful when I have to continue what someone else did. A previous evaluator may have evaluated the first year and moved on. Now at the five year mark I am brought in and expected to build on what they did.
I also find it useful to organize what needs to go into the shorter reports. As I am writing the long report, I note highlights or important points for the short reports.
Great post Chris.
In addition to what everyone has said, I would add the third of my “Four Principles of Effective Reporting” which is layering.
Layering is the idea of allowing stakeholders to go as shallow or deep into the findings as THEY choose by producing several different report products, each appropriately tailored to the target audience. The trick to this technique is to make sure that the products are LINKED to one another, sort of like a trail of breadcrumbs.
So if you give an oral presentation, hand out a post-card with a link to where they can download a two-pager. At the bottom of the two-pager make sure there is a link to the final detailed report. On the website, give them different options such as an infographic, short video, or more detailed podcast. If they’re short on time, they’ll watch the video. If it peaks their curiosity, they might download the entire report.
Two of my favourite examples of layering are http://www.gao.gov/duplication/overview#t=0 and http://www.paintstewardshipprogram.com
Thanks Kylie,
Your insight in this arena is always valued and appreciated 🙂
As well as your insight on Demelorent Evaluations of Purple Monkey Kangaroos.
Wait for the infographic coming-out on them – fascinating reading!
I agree! For a long time my colleagues and I were trying to move away from long reports that get read very little. However, even if we do something simpler, we often still end up doing much of the work since you still need time to digest the data, slice it in different ways, dig deeper, synthesize, etc. As you discussed, more and more it seems best to embrace the long report, but very thoughtfully follow-up with tailored presentations, report briefs and handouts.
The key is making sure to build in time and budget for these things since often reports can be seen as the final deliverable near the end of a contract. That being said, I find that these additional materials can come together pretty quickly if everything has been fully worked out in your report (and if you are data viz-savvy or have some data viz-savvy friends/colleagues)!
Additionally, as few people actually read long reports and they are often primarily for documentation, it seems appropriate to be as efficient as possible with things like formatting. Standard report templates, chart and table templates (that still get customized), and automated numbering are pretty easy things that save a lot of time. I still find I spend a lot of time on data viz since it seems wrong not to and much of it gets carried over to the other materials. Wondering if you have any other thoughts about efficiency with the reporting process (product?) when it isn’t the true end product and isn’t likely to be widely disseminated?
The question to ask…is the client really on board with wider dissemination in that they would be willing to commit funds to make it happen? The evaluator perspective is often that everyone should know about this, but does the client care that much or do they just want/need the report?
If the client does care, then make the case that the comprehensive final report is not the final deliverable. In my own head, I’ve added a step to the evaluation process called the adaptation step. It comes after the final report but before dissemination.
Whether or not the client cares, if you do, I think it’s important to make reporting a habit throughout the process and not just something you do at the end.
So when coming up with your methods, go ahead and create an infographic shell or explainer video. Or when you’re analyzing your data, go ahead and create a simple interactive tableau dashboard with dummy data. These can be part of your own day-to-day evaluation planning and sesnsemaking process. Then when the report is ready and vetted, it’s much easier to complete all the extra pieces.
Hi Chris – I completely agree with your thoughts about the 200-pager! Most of mine are considerably shorter — but still too long for most audiences. I love the idea of creating a “bumper crop” that can be “harvested” to develop myriad other communication tools. Thanks for the fresh take! Sal
Thanks Sally 🙂
PREACH!!!
As someone still new to the field, it helps me grow as a professional to see other large reports, and to create my own. I liken it to accountability, similar to keeping a daily log of what I do. It’s a comprehensive reporting of all the activities and processes that went in to that nice little 2 page infographic handout. If something needs to be fact checked, or we want more information to explain X to a client, it is there, nice and tidy, indexed, collated, and bundled up. If printed, it’s a physical representation of my work (and time). I can account for what I did, and why. There is a lot of value in that!
Great post.
Thanks Courtney 🙂
Super interesting post, Chris. Thanks for writing it. I think the main thing that this points out is the importance of really considering audience and purpose before anything else. 200 page reports serve an important purpose (pleasing funders) but perhaps don’t serve the audience of readers well at all.
You talk here about breaking larger reports into more digestible chunks, which makes total sense. I’m curious, though, if you think that in the future we will move to digital reporting in ways that can make the types of things you are talking about (cartoons, videos, podcasts, etc) part of the report itself, and not simply add-ons that help disseminate the content within it.
I know this isn’t evaluation per se, but this example “report” on the “hidden digital divide” is really inspirational: http://m.scidev.net/index.cfm?originalUrl=global/icts/data-visualisation/digital-divide-data-interactive.html
I love how it incorporates photos, audio, video, and interactive data viz in one super compelling package. Do you have any other examples of creative reporting that you might be able to share?
My hope for the future report is that it will look more like a blog than a website. I view reporting like design, it’s an ongoing process and not a singular product.
The example you give is a pretty cool parallax mobile-responsive site. I really like it. While it definitely hits more audiences and checks more boxes than the traditional 200 pager I can think of a few audiences off the top of my head that it would miss.
A quick run through an accessibility checker shows that it’s missing a bunch of alt text for the visuals. You’ll miss on the people who still love print. Some people will get confused/annoyed by the parallax. There is no easy way to “follow” the report for updates. No visible feedback loop to adapt future report pieces based on audience needs. Won’t be tailored to different audience needs, or adapt based on segment.
All of this to basically say, no one report will ever be everything.
I’ll have more examples on the site soon 🙂
It’s great to read this post. I have dealt with a few evaluation ‘condensation’ tasks in South Africa. In SA – and the rest of Africa – evaluations are multiple due to the high number of donor funded projects, as well as having a (new-ish) National Department of Planning, Monitoring and Evaluation established. Key audiences are quite saturated with lengthy report reading… so we have found that the lengthy reports are not as useful to stakeholders anymore. As organizations compete for funding, demonstrating just the key points are critical for these info-overloaded decision makers. Here is a blog on some of the basics we help some African development organizations with: http://piktochart.com/blog/visual-content-social-change-africa/ It is basic tips which we have found super helpful to simplify and re-purpose evaluation reports
thanks Chris- totally agree that there are different purposes and audiences. Re-purposing one report can really get a lot of mileage from all of that hard work! I was fortunate to take Kyle’s class a few years back and never forget (and often preach) the concept of layering. in fact, it just came up with a very large international food company that has a huge report, but need to respond to scientific stakeholders (they need a 400 page report), NGOs, general public, managers, Board, etc. all asking the same question differently – and each needed a tailored format, using the same basic report content. it’s really about improving the usability of findings rather than strict accountability.