Thursday, December 12, 2013
By Prashant Mittal
PORTLAND - It wouldn't be unsafe to say that our education system in the United States is at least a couple of years behind systems in countries that are leading in education.
Schools that achieve high reading or math proficiency scores one year are also likely to have high three-year average reading or math scores. Maine’s school ranking formula counts both the one-year and the three-year scores, so schools with high scores get double the credit and schools with low scores get double the penalty.
Graphics by Prashant Mittal
Seven Maine high schools moved up in the state school rankings when the formula was altered to ensure that the schools weren’t penalized twice for the same criteria: once for having low one-year proficiency scores, and again for having low three-year average proficiency scores.
Our system needs a major jolt and revival to a new life. The new school grading in Maine is a step in the right direction, done with good intentions -- but, sadly, with a blindfold.
The formula for calculating recent high school grades is a classic example of mediocre mathematics.
INEXACT RANKINGS HAVE BIG IMPACT
The metric is so extremely inconsistent that when it's slightly modified, it causes dramatic movement of anywhere from 10 to 45 moves in the rank of a school, i.e., a school that is originally ranked 100 may actually come down to 55. This may change the school's grade from an F to a C or even a B.
The top 35 schools are rock solid and do not move much in ranking, no matter what variations are made to the metric.
The next 50 schools move three to nine places depending upon only slight changes in the metric, while the biggest shift takes place in the bottom 50 schools.
Incorrectly ranking the schools with inferior grades has a much bigger negative impact than ranking the superior schools correctly.
In the current system, where every school is considered the same irrespective of their size, it is likely that the resources propelled into the low-performing schools will be far from optimal. There is a higher likelihood of disproportionately spending funds and resources on schools that probably don't need it, throwing the schools that actually need support further down the drain.
The problems with the current grading system are multifold.
SAME TYPE OF DATA COUNTED TWICE
First, the same factors associated with higher grades are used multiple times in the algorithm, or calculation formula, to create a total score of excellence.
High schools were graded on two principal domains: proficiency and progress.
Proficiency was broken down into math and reading. Progress was broken down into four parts: math and reading progress, and four- and five- year graduation rates.
The problem lies in the intrinsic high correlation between these six criteria. Statistical modeling techniques warn against using a single piece of information more than once in an estimation.
For example, imagine a dataset of ages. If 10 is added to all those ages, then the new set of numbers will not give any additional information about the original data. The two will be perfectly correlated with each other. This phenomenon -- known as multicollinearity -- makes estimates sensitive to small changes in value. The current grading system is severely suffering from the problem of multicollinearity.
The concept of high correlations is shown in the scatterplot graphic above with the data that was used for grading.
Data shows that schools that achieve high math proficiency scores one year also have high three-year math scores. If a school has a high math proficiency rate, then it is extremely likely to have a three-year running average of high math proficiency as well. The same is true of the results for reading outcomes.
Surprisingly, these correlated criteria were used independently in the algorithm. A school with a lower math score is essentially punished doubly in this algorithm, and a school with a higher score is rewarded doubly. While analyzing the data, I found that minor modifications -- such as not punishing schools twice for the same criteria -- result in a noticeably different set of rankings. The table above offers examples of how schools moved in the rankings.
(Continued on page 2)