CLARKSVILLE, Tenn. — The “Netflix for education” analogy has become somewhat of a cliché for edtech companies using student data to recommend anything from courses to textbooks. The pitch is simple: Why waste time choosing, or leave it to chance of whether a human advisor will understand your unique situation, when an algorithm can tell what you want based on your academic history?
That idea relies on a technology known as predictive analytics, a statistical model that analyzes past data to make estimations about some future event or trend. In higher ed, that data often includes student grades, test scores, attendance and, in some cases, even demographics.
Located about 50 miles north of Nashville, Austin Peay State University in Tennessee has a downhome and cozy feel, sprawling with red brick buildings and some offices that look more like cottages. Yet the campus has gained a techie reputation as one of the birthplaces for predictive analytics in education through a tool dubbed Degree Compass. Officials at the college, which has just over 10,000 undergraduate students, built the platform to recommend courses based on how well students did in previous ones.
At first, Degree Compass was met with fanfare and praise—and data to back it up. Overall student graduation rates rose from 31 percent in 2010 to 36 percent in 2014, for example. But today, the tool’s hype has died down, and campus officials and students say it hasn’t lived up to all of its promise. Graduation rates haven’t budged much since 2014, and retention recently dropped to below the point it was when the tool was introduced.
Now, a college that helped pioneer predictive analytics is finding the technology’s limitations.
Degree Compass was the brainchild of mathematician and former professor Tristan Denley. While working at Austin Peay, and later serving as the institution’s provost and vice president for academic affairs, Denley noticed a problem: students struggled to find the right courses or degree path to take. And without choosing a focus early, many would take courses they didn’t need, reinforcing achievement gaps and delays in college completion.
The situation occurred more frequently for first-generation students, and Denley attributed the issue to an “information problem.” Essentially students whose parents went to college were more likely to find the resources they need to choose degree paths, stick to them and discover career options related to their field of study.
So, with funding from the Bill and Melinda Gates Foundation, Denley built Degree Compass in 2011 as a way to alleviate that problem by suggesting what courses students should take. (Disclosure: The Gates Foundation supports projects at EdSurge.) The system links with the campus degree audit system to help steer students to courses that satisfy their degree requirements, and, using a student’s grade history, applies predictive modeling to estimate which course the student might perform best in. Students see their best options through a university website, which displays recommendations using a star-rating system similar to Netflix.
The program was intended not only to help students perform better in individual courses, but also to increase the likelihood that they complete their degree. Denley wrote in 2012: “The system makes its strongest recommendations for courses that are necessary for a student to graduate, that are core to the university curriculum and the student's major, and in which the student is expected to succeed academically.”
Degree Compass quickly expanded to not only suggest courses but also recommend majors, with a system called MyFuture that would point students towards degree pathways and career options that their course history suggests they might do well in.
At first, the system showed positive results. Early analysis of the tool looked at about 10,800 students at Austin Peay, and later, Denley and his team expanded Degree Compass to three other colleges in Tennessee, adding about 40,000 students to the experiment. The algorithm was successful nearly 90 percent of the time at predicting if a student would pass a particular course. And student grades across the institution showed steady improvement from 2010 to 2014, according to an analysis by Denley.
During that time, six-year graduation rate increased from 33 percent to 37.4 percent, the report said, and noted even greater gains for low–income students (from 25 percent to 31 percent) and African American students (from 28.7 percent to 33.8 percent).
News and media quickly latched on: “Students who follow its course recommendations increase their number of credit hours and gain better grades,” the Economist wrote in 2013. “These gains largely erase the achievement gap between whites and minorities at those particular southern colleges.”
Private companies also took notice. In January 2013, education software provider D2L purchased Degree Compass to offer as its own version of the product.
Shortly after the acquisition, other changes slowly began to happen. For starters, Denley, who largely oversaw the tool and the research behind it, left the university in the summer of 2013 to become the vice chancellor for academic affairs at the Tennessee Board of Regents. After additional administrative turnover, Denley’s passion project slowly dropped on a list of priorities for student-success and academic affairs officials at Austin Peay.
D2L made changes to the product as well—albeit not always for proactive reasons. Kenneth Chapman, vice president of market research and strategy at D2L, explains that the platform today does not show students their grade projections—to avoid discouragement or self-fulfilling prophecies.
The company also noticed that institutions weren’t as excited about a student-directed degree planning tool as they had expected. “We were picturing a student self-serve experience,” he said. But “for the large majority of institutions interested in Degree Compass, putting the recommendations in front of the student and leaving the student alone was not what they envisioned.”
There were technical challenges, too, specifically around getting Degree Compass to successfully integrate with a school’s course catalog and student information system. “It taught us the barriers and challenges institutions were having around how they get degree program requirements loaded into Degree Compass,” Chapman added, which is vital “to get historical data to make predictions accurately.”
The Center for Teaching and Learning at Austin Peay is a cottage-like building and home to the office of Loretta Griffy, a math professor and associate provost for student success at Austin Peay. With walls studded with handwritten Post-it notes and a desk framed by stacks of papers, it may not be obvious that she has largely overseen Degree Compass and MyFuture in Denley’s absence. But in fact, Griffy has been involved with the tech initiatives since their early days, and began working with Denley on the project while he was still at the campus.
Austin Peay continued to use its original version of Degree Compass even after D2L acquired the product—and works with that same version today. Now, Griffy said it hasn’t had a revolutionary effect on the campus in the long run.
“It is simply a tool that’s available for anyone to use,” said Griffy. “We have a decentralized faculty advising model, which means we have 385 academic advisors. They each have their own individual styles and how they interact with students and we have a variety of advising tools that they can use. This is just one of them.”
Griffy also said that the school no longer tracks how many students use Degree Compass or MyFuture—nor could they share any data on how many students have taken the recommendations.
That’s a stark contrast to the kind of reporting and tracking Denley had done. “We had survey data to see how widely students were using the technology, the ways in which advisors were actually incorporating that as part of the conversations they were using in advising,” he said. “The adoption was quite widespread at Austin Peay.”
Similarly, students have contrasting reviews. At Austin Peay’s campus coffee shop, Einstein’s Bros. Bagels, you might find students catching up for a caffeine boost between classes, or hunched over books prepping for an exam. In between meticulous glances back and forth from a laptop screen and hand-written notes, first-year student Sophia Potenza said she entered the college knowing she wanted to study nursing, and that she “didn't really use [Degree Compass].” Abbigayle Nepler, an undeclared freshman, only used the tool when it was required by one professor—but it didn’t sway her decision-making.
At the nearby school cafeteria, sophomore Montrell Harris paused from a late lunch to explain how he entered college less sure about his degree path, compared to students like Potenza. He said the tool helped him navigate course options: “Coming from high school, looking at all these classes will blow your mind.”
Either way, Chapman, of D2L, admits the tool wasn’t the company’s best bet. “Sometimes we really hit the mark and nail it early, and get it to the point where something is cultured in our organization by the time the market is ready for that,” he said. “I wouldn’t call Degree Compass one of the applications that we saw as hitting the mark.”
Up Next
Austin Peay has seen improvements in retention, going from 67 percent in 2010 to 71 percent in 2016. But the school dipped back down to 66 in 2017—almost 10 percentage points away from the school’s goal of 75 percent.
But Griffy claimed the tool wasn’t designed to improve graduation and retention rates. “The software is a tool to help students make decisions on the front-end. It is not a retention tool.”
Because of that, she said it’s hard to know to what extent—if any—that Degree Compass influenced either the rise or subsequent fall in retention. “Here is the difficult part,” she said. “None of this can be attributed to that tool because we have had probably 20 student-success initiatives going on, so which one do you attribute?”
What is clear, she said, is that the system is limited by a catch-22: In order to most accurately recommend courses and majors, the algorithm needs a student’s prior grade history. However, because the tool only pulls from college-level performance, students can’t gain much from it until they have a semester or two under their belt. By then, they might have already made up their mind up about their path, or missed the opportunity to get on-track for crowded and capped majors such as nursing.
Chapman said it’s an issue he has heard before. “That speaks to a conversation we have [with customers] whenever we stand up more complex and sophisticated tools,” he said. “Does it have enough data in a cold start to make any relevant predictions? Are the predictions themselves starting to bias the recommendations being made?”
Other ethical questions have emerged throughout Degree Compass’ history. Could suggestions systematically skew groups of students towards certain classes or fields of study, or push them towards easier classes rather than teaching students how to work through challenging academic material?
Another common concern about predictive analytics in education is that these systems have the potential to reinforce academic and racial stereotypes. Denley intentionally omitted demographic data from the algorithm in an attempt to mitigate that issue.
“We try to recommend to institutions to really soak in and understand the implications” of the data, Chapman said. “If predictions are not significant, then it won’t be an appropriate use case—and heaven forbid recommendations are made off of bad data.”
There are other reasons why the he thinks Degree Compass didn’t quite “hit the mark.” First, he said, the tool was tailored more towards supporting a college’s course catalog and student information system, which was “a little farther away from the teaching and learning focus of our company.” He also added that the company has noticed many of its customers are less often asking for pre-packaged tools and systems like Degree Compass, and requesting more support on how to make sense of the data and analytics already available and collected at the university.
At Austin Peay, issues over the effectiveness of Degree Compass have been overshadowed by larger problems the university is trying to solve. “We have lots of other challenges that are going on. We have a growing freshmen class, and freshmen typically need more services and attention,” she said. “We also ran out of residential housing, so that’s another big challenge to retention.”
Griffy describes the tool as “just one option” among a host of other student-success initiatives happening at the campus. For example, the school also uses an internally-made academic-alert system, where faculty can flag students showing early signs of struggle and have an academic-alert coordinator intervene before grades come in.
Even more popular than the predictive analytics tools though are low-tech paper degree-planning charts. Those, she added, “are more powerful for everybody,” and that “first-time freshmen like that better than really anything” when it comes to helping plan out courses and majors.
“But it’s not fancy,” she added. “You print it and walk around with it. It’s very simple.”
Denley now serves as the executive vice chancellor for academic affairs and chief academic officer for the University System of Georgia. There, and in his previous position, he has worked on developing other predictive analytics systems.
He isn’t shocked that efforts around Degree Compass have stagnated at Austin Peay. “There is a really important aspect of the use of analytic tools: They are only as good as the implementation,” he said. And while systems were put in place to keep Degree Compass running, it might not have been enough to keep the trajectory Denley initially hoped for.
“When there’s a transition of leadership, there needs to be some concerted efforts to hand over, not only the reins, but also the information and intellectual understanding as possible,” he said. “That happened when I left,” he added “but it doesn’t surprise me that [Degree Compass] fell out and got lost in the shuffle. Maybe that’s the way that things happen sometimes.”
As for D2L, Chapman said Degree Compass will remain on the shelf—for now. “I don’t expect we will see a large investment in taking it to market going forward, but I don’t know if the organization has decided what the next step is.”
