Principles of Audience Research and Evaluation

Screen Shot 2017-04-20 at 9.53.59 AM.png

Elee kicks things off for our evaluation workshop – April 19, 2017

I was so thrilled to co-present a workshop today at the National Council on Public History’s annual conference/meeting in Indianapolis along with VSA, Vice President for Professional Development, Elee Wood.

Our 4-hour workshop, entitled Principles of Audience Research and Evaluation in Public History: Purpose, Process and Implementation, covered background information and basic processes for doing evaluation.

We are hoping to develop a version of this presentation that can be shared at conferences that would be of interest to people seeking to learn more about how to study visitors and do evaluation in visitor-oriented settings. So stay tuned for opportunities to take the workshop at a conference coming to a town near you!

Screen Shot 2017-04-20 at 9.55.19 AM.png

Click here for a link to the presentation.


Upcoming visitor studies training opportunities

I’m really excited to announce a couple professional development opportunities that I’ve been helping to organize on behalf of the Visitor Studies Association, in my capacity as chair of the professional development Association.

The first is a webinar on April 27th featuring presenter, Beverly Serrell–the woman who literally wrote the book on Tracking and Timing as a method for studying visitors in exhibits. This is part of a series of professional development opportunities that VSA is co-hosting with ASTC:
Webinar_Timing-Graphs.jpg    Serrellheadshot copy
Registration link:

The other event is the 2017 VSA conference in Columbus Ohio.  We’ve got a lot of great Pre-conference workshops planned so be sure to check out all of these great PD opportunities as well:

Screen Shot 2017-04-18 at 10.57.46 AM.png

Toward a more perfect pie chart

In honor of Pi Day (3.14), I present this pie chart from a report that I did for PBS Kids back in 2013.  Rather than creating three different charts, I sought to visually represent three sets of key participant demographic data in one, visually appealing chart.

In addition to layering three pie chart rings together into one chart, I also toyed with the image fill feature as an alternative to a color fill as a way to convey information without having to add additional text to the chart.  It was fun (and challenging) to try to find images that could convey location information at-a-glance. I think the end result is also more visually appealing with those images included.

I’d be the first to say that this chart isn’t perfect, and I know there are folks who disapprove of pie charts in general, but I think this chart is able to communicate some basic information about our participants fairly clearly and concisely – and in a visually appealing way. Screen Shot 2017-05-16 at 11.17.14 AM

Election-themed babyshower

This week we celebrated the upcoming arrival of Alison’s baby with an election-themed baby shower. With a due-date near the US elections in November, this seemed like a fun way to forego traditional pastels for some good old red, white, and blue! Alison actually inspired this theme when she sent an election themed package to let us know that she was expecting. Congrats Alison – we can’t wait to meet baby Allen!

The Data Revolution

Screen Shot 2016-08-02 at 9.47.56 AM.png

This year’s theme for the Visitor Studies Conference–playing on the historic revolutions that took place in Boston, where this year’s conference was held–was “The Data Revolution.”  There were an impressive number of presentations and workshops that focused on data and a variety of approaches to data analysis.  As always, I’m happy to share out my conference notes, as well as link to my presentation (along with co-presenters Claire Quimby and Elee Wood) “How to Keep from Drowning in Data.”  Lastly, I hosted a dining discussion on the topic of “Cool New Tools, Tips, and Tricks for Evaluation.”

Below (an example from my notes): a scattergram that plots several data points from different exhibits based on Sweep Rate Index (SRI) and the Percentage of Diligent Visitors (%DV)


Less is the New “More”

Big is out – and tiny is in!


If you haven’t been hiding under a rock recently, there’s a good chance that you’ve heard of the tiny house movement. This trend toward small and ultra-small houses has been gaining momentum worldwide as more and more people have been seeking simple and affordable housing options.

One of the underlying reasons is that people have embraced the notion that having more stuff doesn’t necessarily make people happier.  Perhaps the same can be said of reports.

Over the course of my career as an evaluator, I’ve written many reports that have exceeded 100 pages – but rather than being the useful resources that we hope they will be for our clients, they often become glorified bookends. Maybe less is more where reports are concerned as well.

Can we go from this:


to this:



Okay, not literally tiny reports.  But seriously, I’ve been sensing a growing interest in and demand for tiny reports – i.e., reports that total less than 20 pages and take less than a half hour to read – Reports that give readers a good sense of the key findings without inundating them in every detail of an extensive study – Reports that are far more likely to be read, shared, and referenced than their lengthier counterparts.

I’m not sure that all clients/evaluation stakeholders are ready to embrace the notion that less is more – and that might be the first challenge: i.e., getting the primary audience to see how a shorter report might be worth a great deal more than a much longer/more comprehensive report.

An equally daunting challenge–once you’ve gained buy-in for the idea of producing a tiny report–is the fact that its not always easier to write less.  Presenting information in the most succinct/elegant way possible can actually be more time consuming and mentally taxing than mindlessly spewing out every single finding. Despite these challenges, however, I’d argue that the resulting use of a tiny report should be the driving factor in advocating for, and adopting, the move toward shorter reports in contrast to longer more burdensome reporting formats. After all, a great report is only truly great if it gets read and used – no matter its size.


Updated 5/31/17:

I loved this related post by Kylie Hutchinson on the AEA 365 blog – “The Demise of the Lengthy Report.” I especially appreciated the hamburger analogy and image – and it serves as a great reminder that its always important to check stakeholders’ appetites where reporting is concerned.


It depends…

“It depends” is a phrase that most evaluators know well.  In the work that we do, context and participant characteristics often matter a great deal.  In this recent EdSurge article by Patricia Gomes she explores concepts and caveats that relate to efficacy as discussed by Barbara Means and Jeremy Roschelle, co-directors of SRI’s Center for Technology in Learning:

Big takeaways:

  1. Tools don’t exist in isolation.  How they are used by educators matters a great deal. A great tool being implemented half-heartedly or incorrectly may not produce desired results.  Conversely, a poor tool being implemented in the right context by a skilled educator could produce results that wouldn’t be seen in other contexts.
  2. Educators often seek a one-sized-fits all approach, but like cars, maybe the best product varies based on the needs and preferences of different users.
  3. Measures matter too.  You’ve got to know what you are looking for and pick measures that can effectively identify desired outcomes.  (I would also add a note about the fact that it is also good to incorporate methodology that also allows you to uncover unintended outcomes – if we are too focused on very specific outcomes, we may miss other unanticipated outcomes–both positive or negative–that could also be important to understand).
  4. Some outcomes may take time to emerge. Big and fundamental changes in learners take time.
  5. For all these reasons: “It depends” is a valid and accurate response for people who wonder if certain ed tech tools work.