Learning analytics and the pull towards what is easy to measure
Every dashboard reports the quantities the system happens to collect, and those are rarely the quantities that matter.
A learning platform produces data because producing data is a by-product of running software. Logins, time on page, videos completed, questions attempted, click sequences. The dashboard then displays these, and because they are the numbers available, they become the numbers discussed, and eventually the numbers managed.
The substitution
What has happened in that sequence is a substitution. The institution cares about learning. Learning is difficult and expensive to measure. Engagement is trivial to measure, correlates loosely with learning, and arrives free. So engagement is measured, reported, and gradually treated as though it were the thing itself.
This is not peculiar to education software. It is a general property of measurement systems: the available proxy displaces the intended construct, and after a while the substitution becomes invisible because everyone has been looking at the same dashboard for two years.
Time on page is not attention. Videos completed is not understanding. Questions attempted is not learning. Each is a trace left behind by something that may or may not have happened (Campbell, 1979).
What follows once a proxy is managed
The second effect is worse than the first. Once a proxy is used to judge performance, behaviour adjusts to the proxy. (Strathern, 1997) Students learn that the platform records completion, so videos are played and left running. Teachers learn that the report shows activity, so activity is generated. The numbers improve. The thing they were standing in for does not.
This is a well-known dynamic and it is not a criticism of anyone's honesty. It is what happens automatically when a measure becomes a target, and the only defence is to know that it will happen and to design around it.
What analytics is genuinely good for
The case against over-reading is not a case against the data, which has real uses that the dashboard framing obscures.
- Finding who has stopped. A student who has not opened anything for three weeks is a fact worth knowing, and it is exactly the kind of fact that a busy institution misses. Detecting disengagement is the strongest and most defensible use.
- Finding where everyone struggles. If two hundred students all falter at the same item, that is information about the item, not about the students, and it is actionable.
- Finding the sequence that stalls. Where in a module do people stop and not return — a question about design, answerable from traces.
Each of those is a question about the system or about a clear behavioural fact. None requires the data to stand in for learning.
The questions to ask of any dashboard
What is this number a trace of, and what would I have to believe for it to indicate learning? What would improve this number without any learning occurring? Which important thing is not on this screen because it is hard to collect? And is anyone being judged on a number that they can influence directly?
The last one is the important one. A measure used for description survives contact with the people it describes. A measure used for judgement does not, and the institution loses the measure and the information at the same time.
References
Every work below was checked against a primary or catalogue record. Where a volume or page range could not be confirmed it is left out rather than guessed.
- Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67–90.
- Strathern, M. (1997). ‘Improving ratings’: Audit in the British University system. European Review, 5(3), 305–321.
