To better understand just how difficult it can be to collect the correct data and report it the right way, picture this: State education department officials from 40 states have been meeting twice a month since fall 2016 to iron out the intricacies involved in complying with a new federal mandate that requires them to publish how much federal, state and local money every school in the country receives.
They call themselves the FitWigs – short for the financial transparency working group – and their job isn't an enviable one: Figure out what pot of money is used to pay for which personnel and what programs at each individual school.
"It's sort of amazing to watch it play out," says Marguerite Roza, research professor and director of the Edunomics Lab at Georgetown University, who helped mobilize the group nearly two years ago knowing full well how difficult a reporting requirement it would be for school districts.
"They don't have a lot of capacity," she says. "They have a hard time keeping staff. The work that's put on their shoulders is big and huge."
The mandate is one of the last to go into effect as part of the Every Student Succeeds Act, the federal K-12 law that Congress passed in 2015. Among many other things, the law ramps up data reporting requirements for states, districts and schools. In addition to reporting individual school spending, it also requires, for example, that every school keep tabs on students who are chronically absent.
"They're not in panic mode right now," Roza says of states, adding that most of them already issued guidance and instructions to school districts and some have even tested their data collection system by having districts log into a portal and input it as if they were doing it for real.
"They've rolled up their sleeves and are marching forward," she said. "The biggest loose end that most of them don't have figured out yet is how they're going to display the data. That's the work they have left."
That being said, Roza fully acknowledges that the first year of reporting the spending data won't be perfect.
"There are places where the first years of data are messy, and that's just what its going to look like," she says.
To be sure, data discrepancies aren't solely the result of faulty data input. They can also be driven by incentive to look better, as is believed to have been the case earlier this year when an investigation revealed that 1 in 3 District of Columbia Public Schools students who graduated in 2017 shouldn't have received diplomas due to missing too many classes or improperly taking credit-recovery courses.
There's also the case of breathing life into a convenient statistic that fits in neatly to enforce a narrative but goes unchecked before it's used. That was the case last week when Secretary of Education Betsy DeVos during the first G-20 education summit in Chile repeated a claim that 85 percent of the jobs of 2030 haven't yet been invented.
"You throw all of this stuff into a blender and all of sudden I'm not so sure about the things I was sure about this morning," Pondiscio says. "I'm not the guy who says don't use data to inform your decision. And, not to say we shouldn't be trying to get better at this or ignore it, but it should make us more reflective. The incentive you have to make things look better than they really are, you know, this is all a giant study in conflicting interest and data confidence."
However they come about, data discrepancies can have real-world implications for researchers, practitioners and policymakers who make important decision about how to run school systems and ultimately for students who enroll in them.
The revelation that only a fraction of the students the District of Columbia reported as graduating should have been graduating, for example, rocked the education reform community, which for years heralded the District as the fastest-improving in the country and one that was reaping the benefits of more rigorous standards, new teacher evaluation and compensation models, and increased school choice. Indeed, many other urban districts looked to the District as a model for wholesale improvement.
Kowalski is quick to note that the country has only been in a "make-data-useful environment" for about a decade and that demand for data is higher than ever.
A new Data Quality Campaign poll, for example, shows teachers and parents understand the value of data and want to use it to make decisions on how best to educate students.
"I think our polling shows that there is widespread demand and belief that data is a critical strategy for decision-making in education," she says. "That level of belief and demand should drive the resources necessary to get quality data in the hands of those who need it most.