Can AI help bridge the divide between state assessment data and the public?

30 Jan Can AI help bridge the divide between state assessment data and the public?

By Dale Chu

A new AI tool debuted last December that complements the work we’ve been doing here at AssessmentHQ to monitor the reporting of state summative test results. Developed by Emily Oster, Zelma aims to make assessment data “more widely accessible and engaging for the general public.” USA Today and The 74 covered the release, with the latter reporting that Oster believes the tool will help “democratize” school performance data. Those are mighty lofty ambitions. Does Zelma live up to them?

First off, Zelma’s potential is clearly there. The tool’s user interface is simple and clean. Putting in an inquiry is as easy as the click of a button and—presto—a line or bar graph magically appears. When I played with it, I was able to, for instance, quickly grab ELA proficiency rates over time in Indiana vs. Minnesota. The biggest value add, however, might be the ability to quickly download clean, district-level data by subgroup—something that can be rather onerous if left to one’s own devices. I also liked being able to see recent and popular searches, which gave me a sense of how others were trying to use the new tool.

Now to Zelma’s limitations, which in fairness are largely a function of state education agencies and their slipshod management of data systems writ large. Oster herself notes, “The data is messy: formatted differently in every state, often difficult to access and hard to explore in depth.” Compounding this challenge is the inclination of states to switch their tests every few years, making it difficult if not impossible for a full longitudinal accounting of student performance (thank goodness for NAEP!). 

I asked Oster about this during an introductory webinar she held shortly after the tool’s release—a recording of which is supposed to but yet to be made available—after she credited Minnesota for having the largest unbroken trend line, going back to the late 1990s. In contrast, Indiana’s data only goes back to 2015-16, which I knew wasn’t right. Oster’s colleague chalked it up to uneven data among states—which goes to show that it doesn’t matter how many bells and whistles AI brings to the table, Zelma’s ceiling will always be the data underlying it. 

As if to underscore the point, the tool only has proficiency data, so nothing on growth—which Oster conceded was beyond the scope of the project, as well as Zelma’s capabilities. What’s more, while the tool is best for intrastate comparisons (e.g., district to district), it also allows users to do interstate comparisons—which is a big no no when it comes to state standardized exams. To be fair, Zelma provides caveats here, but the point could be lost among less saavy users.

As one education policy veteran noted, “It looks like they still have some bugs to work out.” But this is where AssessmentHQ can be a logical companion. Use Zelma to do state and district level analysis, and use AssessmentHQ to confirm the data is good—which is to say, in compliance with federal reporting requirements (e.g., student subgroup numbers and participation rates). With new energy and attention being brought to the effort, there’s reason to think that greater transparency could be on the horizon.