Sometimes its good to see your shadow

A goal of many evaluations is to help programs or projects get an objective assessment of how things are going. Just like Punxsutawney Phil is tasked each year with coming out of his home to determine if his shadow is visible, project stakeholders must be open to periodically taking a step outside of their day-to-day activities to review how things are going–though admittedly (and thankfully) they have the advantage of evaluation data to reflect on, rather than having to rely on a large woodchuck.

Nobody wants Phil to see his shadow on Groundhog’s Day– we all want an early spring.  Likewise, most project stakeholders are far more eager to get news about all the things that are going well, rather than seeing their proverbial shadow…i.e., the “not-so-great news” which might indicate a need to make some revisions or spend a little more time working through challenges that are impeding progress on their path toward success. I’d argue, however, that without these moments for stakeholders to reflect on how things are going, they are often cheating themselves out of an opportunity to be the best they can be or to do the best they can do. As evaluators, its our role to periodically invite our clients to step out to see if they can see their shadow…or perhaps discover that “warmer days” are near on the horizon.

 

Advertisements

New Year, New Motto

There are times when the purpose of evaluation seems to get lost in the shuffle in a rush to get a program off the ground or a product out the door. In those cases, evaluation can feel like a necessary evil rather than an important step of the overall process.

I wanted to start the year off with a motto that could help me remember the importance of what I do, so that I can help clients (and potential clients) see the inherent value of evaluation as well.  In my quest for that motto, I stumbled across this oldie but goodie: “That which is measured improves” – rumored to have been said by Karl Pearson a statistician. Sometimes referred to as “Pearson’s Law” I think this quote sums up one of the important benefits of evaluation – i.e., its potential for making something even better.

In the absence of measurement, we arguably don’t know if there has been improvement. As the old “G.I. Joe” saying goes, “knowing is half the battle,” but, I like the implied assertion in Pearson’s Law that the act of measurement may foster and help to facilitate improvement.  That’s what I seek to do as an evaluator: help make everything the best it can be!

 

 

One of these things is not like the other

Perhaps it goes without saying that there is a subtle but important difference between checkboxes and radio buttons in online surveys. This is a perfect example (encountered this past week) and a good reminder that it is important to think through what kind of response you want, and to pick the appropriate response format.

 

Reflections on Evaluation 2017 Conference

Just wrapped up a great week in DC for Evaluation 2017.  I met up with many of my friends and colleagues from the visitor studies world; got to spend time with my fellow Rockman et al team member, Kristin Bass, and reconnect with former Rockman team member, Maryann Durland; met with some former and new clients for the first time face to face; gave a presentation on telepresent data collection and hosted a roundtable discussion on assessing the impacts of STEM media projects for youth; and even got to hang out with Michael Quinn Patton in the very last session that I attended…if there is such a thing as “evaluation rockstars,” Patton should definitely count among their ranks, and I’m certainly a big fan of his utilization-focused approach to evaluation. So, I’d say it was  a pretty good week!  

As always, I like to share my notes so that others can also have a chance to benefit from all the things I was able to hear and learn at this week’s conference. I also find conferences to be a great opportunity to hone my note taking skills.

Without further ado, here’s the link to my notes: https://cl.ly/1h263S1V3u19

Long-term Impacts of Early Childhood STEM Instruction

One of the projects I’m currently working on is a formative evaluation for a multi-year grant with PBS Kids to develop STEM-oriented programming for young children. Its not really possible to see longer-term impacts of instructional programming (esp. with a formative evaluation initiative – focused on quickly gathering feedback to help drive immediate changes/improvements to a product), so its exciting when other researchers are unable to uncover links. On that note, I’m excited to share a link to the following article which highlights recent finding that connect early STEM learning to outcomes observed later in a child’s life.

Education in ABCs and density – Research: preschool math performance predicts future K12 academic achievement by Jessica Ablamsky (April 25, 2017)

 

IRB deregulation on the horizon!

At long last, we can see some new Human Subjects guidelines/regulations on the horizon! An article entitled “Long-Sought Research Deregulation Is Upon Us. Don’t Squander the Moment,” recently appearing in The Chronicle of Higher Education, provides more info on the upcoming changes: https://shar.es/1F3hQF

In the meantime (i.e., til January 2018), if you are doing work with Human Subjects and/or have been asked to get an IRB review – this webinar may be of interest:

“Surviving the IRB Process” (part of a professional development partnership between VSA and ASTC)

The power of posters!

 

When I saw this article come through my feed this morning I said “yes, yes, and more yes!”:

Posters – They’re Not Just for Conferences Anymore!

Having written many a 50+ page report only to have it go un-read, and therefore ultimately unused, I’ve become a firm believer in the power of shorter-format reporting – including slidedeck reports, posters and infographic summaries. I agree with the following statement by Stephanie’s guest host, Kylie Hutchinson, “An effectively designed poster can be very ‘sticky’ compared to other forms of reporting. While a fifty-page report is sitting on a shelf somewhere collecting dust, a poster can hang around an organization’s lunch room or hallway for a long time, continuing to engage stakeholders and disseminate your key messages.”

However, don’t rush off thinking “great, I’ll save so much time if I don’t have to write a long report!” Creating effective short-format reports can often take just as much time as a longer report. The process of curation is time consuming, as are efforts to craft visual components that have an anesthetic appeal, but can also effectively communicate desired information. Deciding to go with a short format report likely won’t save you time/money, but definitely might be a better allocation of those resources if your stakeholders agree that it would be an effective way to communicate findings to them.

I’m happy with the way my AERA poster turned out, but sadly I’m not able to travel to San Antonio to present due to a scheduling conflict. Thankfully my colleague Julia has kindly agreed to fill in. You can stop by to see her and all the other great presenters in a poster panel about different types of observational methods in informal learning spaces– organized by Aaron Price–from 10:35a-12:05 on Saturday, April 29th in room 221 D (meeting room level) .

I used a template from: Makesigns.com to create this poster, in about an hour.

Here are a few other examples of “short reports” that we’ve created over the past few years. The first was designed to be a summary of online usage statistics for a program/product that we were evaluating.  The second is an example of a summary of findings from an online professional development session that we evaluated.