May 20, 2013

Off Campus: Lies, damned lies and (school ranking) statistics

A fair method of grading schools would compare similar institutions and use a consistent metric.

By Prashant Mittal

PORTLAND - It wouldn't be unsafe to say that our education system in the United States is at least a couple of years behind systems in countries that are leading in education.

click image to enlarge

Schools that achieve high reading or math proficiency scores one year are also likely to have high three-year average reading or math scores. Maine’s school ranking formula counts both the one-year and the three-year scores, so schools with high scores get double the credit and schools with low scores get double the penalty.

Graphics by Prashant Mittal

click image to enlarge

Seven Maine high schools moved up in the state school rankings when the formula was altered to ensure that the schools weren’t penalized twice for the same criteria: once for having low one-year proficiency scores, and again for having low three-year average proficiency scores.

Our system needs a major jolt and revival to a new life. The new school grading in Maine is a step in the right direction, done with good intentions -- but, sadly, with a blindfold.

The formula for calculating recent high school grades is a classic example of mediocre mathematics.


The metric is so extremely inconsistent that when it's slightly modified, it causes dramatic movement of anywhere from 10 to 45 moves in the rank of a school, i.e., a school that is originally ranked 100 may actually come down to 55. This may change the school's grade from an F to a C or even a B.

The top 35 schools are rock solid and do not move much in ranking, no matter what variations are made to the metric.

The next 50 schools move three to nine places depending upon only slight changes in the metric, while the biggest shift takes place in the bottom 50 schools.

Incorrectly ranking the schools with inferior grades has a much bigger negative impact than ranking the superior schools correctly.

In the current system, where every school is considered the same irrespective of their size, it is likely that the resources propelled into the low-performing schools will be far from optimal. There is a higher likelihood of disproportionately spending funds and resources on schools that probably don't need it, throwing the schools that actually need support further down the drain.

The problems with the current grading system are multifold.


First, the same factors associated with higher grades are used multiple times in the algorithm, or calculation formula, to create a total score of excellence.

High schools were graded on two principal domains: proficiency and progress.

Proficiency was broken down into math and reading. Progress was broken down into four parts: math and reading progress, and four- and five- year graduation rates.

The problem lies in the intrinsic high correlation between these six criteria. Statistical modeling techniques warn against using a single piece of information more than once in an estimation.

For example, imagine a dataset of ages. If 10 is added to all those ages, then the new set of numbers will not give any additional information about the original data. The two will be perfectly correlated with each other. This phenomenon -- known as multicollinearity -- makes estimates sensitive to small changes in value. The current grading system is severely suffering from the problem of multicollinearity.

The concept of high correlations is shown in the scatterplot graphic above with the data that was used for grading.

Data shows that schools that achieve high math proficiency scores one year also have high three-year math scores. If a school has a high math proficiency rate, then it is extremely likely to have a three-year running average of high math proficiency as well. The same is true of the results for reading outcomes.

Surprisingly, these correlated criteria were used independently in the algorithm. A school with a lower math score is essentially punished doubly in this algorithm, and a school with a higher score is rewarded doubly. While analyzing the data, I found that minor modifications -- such as not punishing schools twice for the same criteria -- result in a noticeably different set of rankings. The table above offers examples of how schools moved in the rankings.

(Continued on page 2)

Were you interviewed for this story? If so, please fill out our accuracy form

Send question/comment to the editors

Further Discussion

Here at we value our readers and are committed to growing our community by encouraging you to add to the discussion. To ensure conscientious dialogue we have implemented a strict no-bullying policy. To participate, you must follow our Terms of Use.

Questions about the article? Add them below and we’ll try to answer them or do a follow-up post as soon as we can. Technical problems? Email them to us with an exact description of the problem. Make sure to include:
  • Type of computer or mobile device your are using
  • Exact operating system and browser you are viewing the site on (TIP: You can easily determine your operating system here.)