I have just returned to work after spending two days at Shar-E-Fest 11. Hosted by Wintec, this year’s keynote speaker and National E-learning Symposium presenter is Professor Thomas C Reeves from the University of Georgia. Tom’s keynotes and workshops gave me plenty to think about and follow up, some of which I will unpack here. We are very much looking forward to hosting Tom and his wife, Trisha, at the Open Polytechnic tomorrow as part of the National E-learning Symposium.
The role of formative evaluation
One of Tom’s workshops look at the role of formative evaluation to improve the effectiveness of online teaching. We initially unpacked the difference between evaluation and assessment; loosely, we assess characteristics but we evaluate things. Tom challenged us to create a formative evaluation plan for a fictitious new course and decide what it would look like. “Plan?” we all mumbled, “for formative evaluation?” To be honest, it wasn’t really an approach many of us had given due consideration.
Tom gave an overview of four major paradigms in relation to formal evaluation within the context of course or programme design. Each is valid with its own benefits and advantages but equally has its own flaws and biases. Briefly:
- Experimental (quantitative) paradigm. Based on the notion that there are facts with an objective reality that exist regardless of our beliefs, eg hitting our head against a hard wall is going to hurt. The goal of experimental evaluation is to detect the causes in change in phenomena through quantitative analysis.
- Interpretive (qualitative) paradigm. Reality is socially constructed through collective definitions of phenomena, eg in the Inuit language, there are multiple words for different types of snow (although Wikipedia disputes this particular example as a popular urban legend). The goal of using this paradigm is to interpret phenomena from multiple perspectives.
- Postmodern (critical) paradigm. Reality is individually constructed based upon experience, eg culture, gender etc. The goal of this type of evaluation is to improve the status of under-privileged minorities, which sometimes leads to discussion around the hidden curriculum.
- Pragmatic (eclectic) paradigm. Reality is complex and many phenomena are chaotic and unpredictable. The goal of evaluation using this paradigm is to provide decisions makers with the information they need to make better decisions. Chaos theory, anyone?
What does this mean for us?
These paradigms are all very well and good, but how do they relate to us as educators? As we know, evaluation is about far more than simply ticking the right boxes. When asked to evaluate the programmes or courses we teach, it is important to consider who our key stakeholders are and what paradigm they are coming from as this will ultimately affect the type of evaluation carried out as well as any findings. This will help us plan the purpose, type and methods for evaluation and determine what kind of information we will gather, analyse and report. Tom shared some of his tools and resources to help frame evaluation plans and make sure we are finding out the most relevant information.