We’ve had many sub-zero temperatures here in Indiana over the course of the past two weeks. Along with the foot of snow that fell this month its made for a pretty frigid February – and I know I’m in good company across the midwest this winter. In any case, seeing my car’s temperature readout at -6 this morning got me thinking about the fact that there’s a point where cold is cold, and colder doesn’t seem to be much more difficult to tolerate. Likewise, I feel the same way about the warm weather we get in the summer. Is 101 really that much warmer than 91 if both are sufficiently hot to make us feel uncomfortable?
Scaled responses on surveys seem a little like temperatures. I do an activity in some evaluation workshops that I lead where I ask participants to use different scales to pick the response that correlates to a specific response to a specific question. For example, given the question: “How interesting is today’s workshop?” and the hypothetical response “pretty interesting,” I ask respondents to indicate what number they’d pick on a 100 point scale where 0 is not interesting at all and 100 is extremely interesting. The size of this scale typically produces a wide range of potential responses anywhere from 60-90 (typically). I then ask participants to use a ten point scale (same question, same answer) and the spread drops down from a range of thirty different responses that could potentially mean the same thing to a range of 2-3. Lastly I present a 5-point scale where 1 is not at all interesting and 5 is extremely interesting and usually close to 100% of the participants agree that the correct response for thinking that the workshop is “pretty interesting” would be a 4.
There are times when having the precision that a much larger scale offers might be valuable – but ultimately, if there is little agreement on the part of respondents on what each point on that scale means, how helpful was having that much variety when it comes time to analyze the data? Five-point scales still have their flaws, especially in so far as different people might be pre-disposed to be more generous or more conservative in their ratings, however, there definitely seems to be more consensus on what the different response options mean. This morning I jokingly found myself wondering if we could get away with a five-point temperature scale as well: Really Cold!, Cold, Nice, Warm, and Hot! I don’t think there’d be any disagreement that today is a “really cold” day.