“It depends” is a phrase that most evaluators know well. In the work that we do, context and participant characteristics often matter a great deal. In this recent EdSurge article by Patricia Gomes she explores concepts and caveats that relate to efficacy as discussed by Barbara Means and Jeremy Roschelle, co-directors of SRI’s Center for Technology in Learning:
- Tools don’t exist in isolation. How they are used by educators matters a great deal. A great tool being implemented half-heartedly or incorrectly may not produce desired results. Conversely, a poor tool being implemented in the right context by a skilled educator could produce results that wouldn’t be seen in other contexts.
- Educators often seek a one-sized-fits all approach, but like cars, maybe the best product varies based on the needs and preferences of different users.
- Measures matter too. You’ve got to know what you are looking for and pick measures that can effectively identify desired outcomes. (I would also add a note about the fact that it is also good to incorporate methodology that also allows you to uncover unintended outcomes – if we are too focused on very specific outcomes, we may miss other unanticipated outcomes–both positive or negative–that could also be important to understand).
- Some outcomes may take time to emerge. Big and fundamental changes in learners take time.
- For all these reasons: “It depends” is a valid and accurate response for people who wonder if certain ed tech tools work.