Big data is set to profoundly change the way we assess learning and our understanding of learning processes.
The promise of big data is to continuously capture individual data, contextual data, and relational data among learners.
This will mark the end of the distinction between learning time and assessment time. Learning and assessment will occur continuously, with personalized feedback based on the learner’s progress and level of autonomy.
But what lies beyond these promises? A team of researchers at Stanford University has examined the added value of this big data in assessment and presents a selection of practical use cases within three distinct online learning environments.
Three Case Studies
- The Open Learning Initiative (OLI) platform from Stanford and Carnegie Mellon Universities.
This environment offers an intelligent tutoring system that adapts to the learner’s individual progress throughout assessments—which are integrated into the learning paths on the platform.
The sample consists of a dataset of over one million individuals, but the unit of measurement is on a small scale: the pedagogical objective and the underlying competencies. Generally speaking, a course on OLI—and its data—is broken down into 30 to 50 learning objectives and implies between 100 and 1,000 skills to be acquired.
The initial learning model, developed by researchers, is compared in real time with the data provided by learners and adapts to it, to provide learning predictions for each learner and each skill.
- Three courses on the basics of programming: Code.org’s “Hour of Code,” a course on Coursera, and a Stanford in-person course titled“Introduction to Computer Science.”
Assessment here is based on solving open-ended problems—programming challenges—by working through intermediate steps. All these partial solutions are recorded as they are completed, to track the learners’ “progress path”…for one million people!
Data from the in-person course, for example, suggest two general behavioral trends: learners who show steady progress toward the final goal and those who tend to get stuck, for a relatively long time, at an intermediate step.
The problem-solving trajectories for the first exercise (rather than the score obtained) show a significant correlation with the results of subsequent assessments (mid-course).
Data from the “Hour of Code” program also show that problem-solving paths can significantly diverge from the “correct” methods recommended by teachers!
- Data from several MOOCs were used to analyze participant retention.
The researchers built a model to predict dropout rates based on several “sensitive” variables: the time spent watching videos during the first week, performance metrics (rate of incorrect answers on the first attempt, time elapsed between two attempts, etc.), and the proportion of videos and exercises skipped.
It’s easy to see the value of having a community of learners: Based on their model’s predictions, the researchers sent a survey to 9,400 people deemed “at risk” during the third week to study in greater depth the three main reasons for dropping out of a MOOC: procrastination, difficulty, and lack of time.
Through this series of examples, the researchers demonstrate how the use of big data for evaluation purposes can help us better understand the many facets of learning. The ambition is lofty, but there is a caveat: there is a significant difference (and gap) between measuring and using.
As the authors point out, a 2011 study by the National Research Council revealed the ineffectiveness of ten years of education policies based on assessment results in the United States.
Having complex, large-scale data does not necessarily mean interpreting it correctly, much less successfully using it. This remains a challenge for the years ahead.
Illustration: Laborant, Shutterstock
References
1. Stanford University Lytics Lab. “The Future of Data–Enriched Assessment” in *Research and Practice in Assessment*, vol. 9, 2014. http://www.rpajournal.com/dev/wp-content/uploads/2014/10/A1.pdf
2. National Research Council. Incentives and test-based accountability in education (2011). http://www.nap.edu/read/12521/chapter/1
(Links accessed on March 14, 2016)
See more articles by this author