One of these things is not like the other

Perhaps it goes without saying that there is a subtle but important difference between checkboxes and radio buttons in online surveys. This is a perfect example (encountered this past week) and a good reminder that it is important to think through what kind of response you want, and to pick the appropriate response format.




Sew what?


Nothing like a little cross stitch to get the creative juices flowing! So first, a quick note about the image above. I accidentally hit the Pinterest button on the top of my browser window while I was reading my email this morning. Because there were no other images in the window that could be pinned (apparently), I got a generic cross stitch image with my email address and company name.  It was cute, so I thought I’d share it – but then I got to thinking, since I do quite a bit of cross stitching, that there are quite a few similarities between cross stitch and evaluation.

1. It helps to lay some groundwork for success upfront.  You’ve got to count squares on your canvas before you start sewing to make sure you aren’t going to run out of space. Eyeballing it gets you into trouble. I’ve learned this the hard way on a few cross stitch projects where I was trying to use up scraps of fabric and was certain I’d have enough room to fit everything in with an even border around all sides – and the finished product ended up being a bit un-centered. The same thing goes for evaluation: eyeballing a project can get you into trouble.  It’s worth it to take the time to get to know a project – the key players, the goals, and all the intended activities.  A logic model is a great way during the planning stages or early-on in a project to make sure all the pieces are going to fit in a purposeful fashion.

2. There are many ways to stitch.  Over the years I’ve tried a lot of different methods…individual x’s, completing a whole row of lower left to upper right hand-corner stitching (////) before completing the same row with stitches in the opposite direction (i.e., \\\\), adding long stitches across multiple rows and then going back to add the other side of each individual square’s x (though I think this is cheating).  When all’s said and done, the finished product often looks the same from the front, but its a whole different story from the back. I think there are three evaluative lessons to be learned from this example.

A) Sometimes it matters how the back-side looks. With some evaluation projects, clients are looking for quick answers to questions and they don’t much matter how the questions get answered so long as they get accurate feedback. In other cases, clients want more transparency and may appreciate opportunities to chime in about the process the evaluation team takes in going about seeking answers to the questions set forth for the project. Sometimes, the “how” of it matters just as much to a client as the finished product, and that’s okay.

B) Different methods may be better in different contexts. While I don’t have scientific proof of this claim, I would assert that some modes of stitching are quicker in certain contexts (e.g., big chunks of color vs. patterns that are more diffuse or complex). As evaluators, it’s important to consider the methods we are using and their fit with the specific data-collection context. Sometimes we need to think strategically about which evaluative method is most appropriate given a wide range of factors or constraints (e.g., time, cost, privacy issues, literacy or language barriers, cultural differences among participants, institutional barriers, and more). One of the important skills that we develop as evaluators is being able to identify and implement the methods that are best-suited to each individual situation.

C) Don’t get in a rut.  I often change things up in the middle of a sewing project just to break up the monotony of stitching the same way over and over again. At the onset of an evaluation project, there is definitely value to thinking outside of the box (i.e., the same set of evaluative methods that we rely on day in and day out), but there may also be value to adding in some variety within an evaluation project to keep things fresh, and to help ensure that we are seeing things from different viewpoints.

3) It’s okay not to be perfect. I’d be the first to admit that my cross stitch projects are rarely perfect. I miscount rows or misplace x’s all the time. Rarely is a mistake so major that I have to unstitch things to make a correction, so most times I just roll with it and make modifications to the pattern as necessary. Ultimately, I’ve come to think of those minor diversions from the pattern as my special fingerprint on a project. It’s a lot harder admitting that my evaluation projects are rarely perfect, but after a considerable amount of reflection, I think it’s fair to say that there is no such thing as a perfect project nor a perfect evaluation. Rarely does a five-year project adhere perfectly to every single detail of the plan set forth in the proposal – so the evaluation has to pivot and evolve. Even with great instruments, the resulting data can be messy. The true sign of a good evaluation, not unlike cross stitch, is the quality of the final product.

As evaluators, we are often charged with asking the question, “so what?” but next time (if you are so inclined) think of this post, and ask “sew what?” –  you might just find some creative inspiration to make your evaluation effort even more awesome.

Below: a few of my recent cross stitch projects

Money Talks!

In grad school I heard stories about the days when grad students would be put to work polishing quarters or ironing dollar bills to send along with surveys. I’d thought that day had long since come and gone, but low and behold, I received a survey in the mail this past week with a crisp dollar bill as a proactive “thank you” for my response to a survey about my car. That same day, my boss shared the following article:

Want people to answer your mobile survey? Pay Them! – by Jacob Korenblum on ICT Works

The article shares examples of several studies that showed that incentivized responses were not statistically significant from responses that were not incentivized (obviously an important consideration for the sake of validity) – and made the claim that properly incentivized surveys saw significantly higher response rates.

photo (9)

As a survey respondent, about a half hour in to taking the survey I’d been paid a dollar for, I started questioning whether that incentive had indeed been appropriate, but by that time I was starting to see the light at the end of the tunnel and decided to persevere in the name of science (or at least evaluation)! Would I have responded to the survey if I hadn’t been paid a buck? Probably – but I see every survey as an opportunity to learn a new trick or two. None-the-less, the crisp dollar bill definitely got my attention (and I was ever so glad that envelope hadn’t gone straight into the recycling bin).

Over the years – here are some tricks that I’ve learned with regards to properly incentivize survey responses:

1. If you are asking people for responses that might take a little longer (e.g., 30 minutes plus) and you have limited funds for incentivizing responses, it seems better to give people a chance to get a larger incentive. With an incentive budget of $250 and need to get as many people as possible to take a survey that will take 30-40 minutes, I’d rather offer a chance to be entered into a drawing for an iPod than offer $5 for each response (esp. if there are extensive administrative costs associated with getting each respondent their $5 incentive upon survey completion).

2. If you over-incentivize you may run the risk of getting more responses than you think you are going to get. This is another reason why I’d argue that it is better to go with a drawing for a single incentive (or a few incentives) rather than offering a set-price incentive to all participants. Some of my colleagues once offered a flashlight to survey respondents, but somehow, news of the survey linked onto a “freebies” list and they had far more responses than they had anticipated. That type of scenario could be good if all respondents were members of the intended target audience for the survey and there had been an ample supply of incentives – however, it could also be really bad if it produced a lot of unhelpful data and larger number of respondents expecting incentives than there were incentives to give (and/or a budget to provide those incentives). For this reason, it is essential to strike the right balance in terms of incentives that will succeed in getting the right types of respondents to respond, but not create an incentive for others to jump on board.

3. Look for ways to streamline the incentive delivery process. The actual incentive is only part of the cost of providing an incentive, there are also logistical tasks that can add to the overall cost of offering an incentive. Many online survey programs have options for automatically delivering electronic gift cards to respondents upon survey completion. Services like this can save a lot of time in comparison to having to manually check to see who completed a survey and individually email or mail out gift cards or other incentives. See #5 below for even more things that can automatically (and freely) be shared with participants upon survey completion.

4. Sometimes a simple “thank you” may suffice. In cases where incentive funds are limited or non-existent it may be sufficient to make a sincere appeal to people for their help and thoughtful responses or feedback about a program, product or event that they care about – even if you have nothing tangible to offer. In this scenario you can lead with an appeal that makes people feel good about giving their time for what they perceive to be a worthy cause – the incentive is the warm and fuzzy feeling that they get for doing a good deed.

5. Think outside the box! There may be free or very inexpensive things that you can offer to survey respondents – e.g., special content related to a favorite show (e.g., an audio “thank you” from one or more of the stars of that show) or other digital freebies, (e.g. a poster they can download and print). Here’s a more extensive post on the topic of creative survey incentives: The Dino Does It! Advantages of Creative Survey Incentives.

Tips for Play-Testing

The following article by Anna Jordan-Douglass – one of the great folks at The Jim Henson Company that I’ve had a chance to do play-testing for over the past few years – shares a wealth of great tips for ensuring games are fun and functional by testing them throughout the development process.

The Tests the Thing: Making Sure Your Product is Fun and Playable (In Games + Learning: )

Among Anna’s tips: Keep it simple – play testing sessions don’t have to be too long, or too formal, or include too many participants; you can find important things letting a few kids spend a few minutes playing a game somewhat informally.  Sometimes, its not what kids say, but rather what they do, that tells you what they really think about a game – if they don’t want to quit playing, its a good sign that you are on the right path to a fun game that other kids will enjoy as well.

As I’ve been cleaning through some files and reviewing reports for the “Where Fun and Learning Clicks!” project that I worked on toward the earlier part of this past decade, I’ve been reflecting on some of the other things I’ve learned over the course of a decade’s worth of play-testing apps and websites with youth. Where Fun and Learning Clicks! was a program led by the Corporation for Public Broadcasting to fund development and test the impact of five websites geared toward tweens. I hope to share a more formal recap about my efforts on that project at a later point, but in the mean-time, here are a few play-testing tips that I would throw into the mix:

1) Not every kid will like every game – and that’s okay! There are many different types of successful game genres for adults, and there’s a reason for that – not all adults like playing the same kinds of games, and the same goes for children. Some games might have greater general appeal, but it is reasonable to assume that there will be some variation in how much any game appeals to different children.

2) Never underestimate a child’s ability to innovate. Kids are incredible innovators when it comes to games.  Given time and free reign to do so, they can take an ordinary game to the next level by creating new ways of playing or competing with other players. Given two weeks of free play time, I once saw a group of kids transform a relatively simple flip-book style game wherein different outfits could be created, into an awesome variation on a collection game wherein they were competitively comparing the collections of different outfits they’d amassed.

3) To get good data, its important to make sure that a child is at ease and feels comfortable in the environment where the testing is done. We’ve had great success doing testing in homes and, when home testing is not an option, typically prefer to run testing sessions in community centers or libraries that children are familiar with, but when it is too tricky to work out the logistics associated with testing in multiple sites, or setting up sessions in a remote site, we do everything we can to make our play-testing room as kid-friendly as possible, with a low table, bright posters, and a few strategically placed stuffed animals that help to make the room more comfortable and inviting to young children.

4) What kids do is sometimes more important than what they say.  Sometimes a kid will tell you that he loves a game, but every other minute he might be looking or asking for other games he can play.  Another child may not be able to say much about what she likes about a game, but the fact that she doesn’t want to stop playing (as Anna mentions in her article), actually says quite a bit in and of itself.  There’s a great deal that can also be gleaned by watching for body language cues during play testing sessions – i.e., watching for things like signs of determination, distraction, or joy.

5) Because there’s so much value in being able to see what kids are doing – recorded video can be a super-valuable tool for facilitating more extensive review and analysis of play-testing sessions and/or helping clients see strengths and weaknesses of their games. When testing computer games, over the shoulder video (esp. coupled with screen capture + webcam captured video) is great – but things get a little trickier when kids are testing games or apps on handheld devices. When taping play-testing sessions on handheld devices, we’ve found that recording with a handheld iPod or digital video camera is a simple way to keep all the action in-frame, even as a child moves around.

The Dino Does It! Advantages of Creative Survey Incentives

There is a certain art to figuring out the right incentive to drive participation for any kind of evaluation activity. Do you offer something small to everyone who participates, or a chance to win one or a small number of more valuable incentives?  Offer too much and you might blow your budget; offer too little and you may not get enough participants. Further complicating matters is the fact that no pool of potential focus group participants or survey respondents is identical – what might work well for a group of high-school students in California may not cut the mustard with a group of informal science educators in St. Louis.  Not surprisingly, the true value of any incentive is in the eye of the beholder – and with that in mind I wanted to share a story about a very successful survey campaign that incentivized adult respondents with merely a chance to win a toy dinosaur.

As part of our evaluation effort for Radiolab <>  (the popular science podcast and radio show produced by WNYC), we created a survey to gather feedback from people who attended their live show, Radiolab Live: Apocalyptical, in venues throughout the United States. For lack of better and more scientific terms, the audience for Radiolab and its live show tend to be young, hip, and very creative — they have done everything from writing songs, making videos, and creating new logos, to helping name our mammalian ancestor. So, instead of offering a chance to win an iPod loaded with Radiolab podcasts — which, as a high-value prize, presented certain legal hurdles — we ultimately decided to scrap the idea of a prize with high monetary value, and return to the drawing board with the added challenge of finding an original incentive that had no real monetary value, but would still be seen as valuable by potential respondents.

It was Molly Webster, a member of Radiolab’s production team who stumbled upon the genius idea to offer a toy dinosaur, signed by Jad Abumrad and Robert Krulwich, the show’s two hosts, as the incentive.  In her own words:

I have one sitting on the shelf to the right of my desk, staring at some fossils I collected in the field while reporting the story, so when I dropped my head to the side in a sign of frustrated desperation, trying to think of what we could give away, my eyes settled on it. And ahoy! It became our prize.

What made the dino such a great incentive? First off, dinosaurs played prominently in the Apocalyptical program, so it was a prize that was likely to resonate with individuals who’d seen the show. Secondly, the dinosaur had a very minimal cash value, meeting the low-value preference. Thirdly, since it was small and unbreakable, it would be easy to ship to a winner anywhere in the US. But finally – and most importantly – I suspect it was the show hosts’ signatures that elevated it from a cool, albeit quirky, prize to something that Radiolab fans really wanted to win.

As an evaluator, I’m usually thrilled when I get a couple hundred survey responses for a program of this nature – so I was truly amazed when we got more than 6000 survey responses!  Did the dino make a difference?  Obviously we can’t know if the dino was a more popular incentive choice than an iPod would have been, but based on past experiences where we’ve offered a more conventional incentive and gotten far fewer responses, I feel that the creativity and uniqueness of the incentive might have been an advantage in this instance. It also didn’t hurt that Radiolab listeners, i.e. the folks who made up the majority of the Apocalyptical audience, are really passionate about the show and likely valued the opportunity to provide feedback to help make something they love even better. None-the-less, 81% of respondents opted in for a chance to win the dino – and one lucky winner received a plastic dinosaur signed by Jad and Robert in her mailbox.

My take-aways from this experience: 1) Don’t be afraid to think outside the box where incentives are concerned.  2) Keep your audience in mind and try to come up with incentives that they will perceive to be valuable – even if there’s little or no monetary value. 3) Involve clients in the incentive-brainstorming process – they know their programs and audiences better than anyone and might have a brilliant incentive idea that could drive thousands of responses for your next survey.Image

Do-it-yourself Data

With some evaluation projects, all you have to do is show up at an event and gather data. But what do you do when you need to gather formative data about activities before they are shared and implemented more broadly? When recently faced with that challenge, I decided to organize a series of “experimental outreach events” to test a set of educational activities designed around a television program and website that we are evaluating. We learned a great deal from seeing the activities being implemented by real informal science educators – and also learned a great deal from seeing participants of various ages doing the activities. While I’m generally of the opinion that evaluation should be as hands-off as possible from the events and activities being studied, this was a great opportunity to test out methods for collecting data in a situation that required my team to be more hands-on.

Event flyer:

Event website:

Event video: