Measuring long-term impacts in the short-term, finding big impacts from even the smallest program, and other evaluation challenges

As a follow-up to last week’s VSA/ASTC “Back to School” Webinar, I’m going to be posting responses to participants’ questions.

The first question that I’m going to tackle, was posted by Gerrie Hall.  Gerrie asked: “how realistic is it to measure long term impact for museum visits that are one session?”

There are two very important points raised in this question: The first being related to timing – i.e., a desire from funders to know more about the long-term impact of school group experiences when we don’t always have a long-time to gather data and report our findings, and the second being related to the level-of treatment, i.e., being able to tease out the impact of a single, and often very short, educational experience.

To answer the first part of the question, I’ve generally found it to be a good policy to be honest and open with funders about what outcomes we are able to measure at any point in time but we can also go two steps farther 1) to explain our long-term data collection strategy and 2) to highlight evidence that speaks to the likelihood of longer-term impacts. By indicating what we are doing to keep looking for and measuring longer-term outcomes over a much longer period of time, our funders can come to see our long-term investment in the evaluation process and understand that we are committed to learning more about the types of impacts that happen over a longer period of time. We can also aim to identify findings immediately following a program experience that are linked with the likelihood of longer-term impacts (i.e., giving funders a brief glimpse into the likely future, based on what we know in the present). Informal science literature is a good place to look for some examples of things that can be measured immediately following a museum experience which have been shown, over time, to be linked to different types of knowledge, attitudinal and behavioral outcomes. Obviously, greater amounts of attention and engagement during the program are important pre-cursors to longer-term recall and deeper levels of sustained understanding. Some attitudinal outcomes immediately following an experience may also be linked to longer-term behavioral changes. There’s also an opportunity for local institutions to build their own sets of data that correlate short-term and longer-term outcomes – in other words, as you start to gather shorter-term and longer-term feedback about your educational programs, you will be in a great position to show what types of short term data correlate with longer-term outcomes.

Answering the second part of the question can be a little more challenging, but presumably, your funders are somewhat pre-inclined to believe in the potential of museum-based-educational programming because they’ve already provided you with some funding. This belief is likely to come from either an intuitive belief in the value of museum-based experiences and programming or familiarity with the growing body of literature on informal learning that shows the value of hands-on, interactive learning experiences, especially those that incorporate elements of free-choice learning. To have the best possible chance of showing big gains from what is arguably a very short experience (in the context of far more time-intensive efforts within formal educational settings), focus on the elements of your program that are most likely to have a significant effect – a good place to start is to identify the elements of your programming that are most distinct from the types of experiences that students are able to have in formal school settings (i.e., the types of things that allow you to isolate the impacts of museum-based learning experiences as opposed to other types of educational experiences). Do they get to see or touch real life animals? Do they get they interact with scientists or other types of experts? Do they get to create something or do other types of activities that may not be possible in a classroom setting? Its also important to look for clues in the types of things students say they like most about your program(s) – those are other areas that you should certainly be focusing your evaluative attention on both in the short-term and long-term.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s