A certain clinic advertises that the death rate of people between the ages of 45 and 65 is 14 out of 100, but the death rate of people in that age range who go to the clinic is one out of 100. This looks very good, until you look at how the statistics was gathered. The clinic says these statistics are for the last decade. This means there are no people who were followed for the entire age range from 45 to 65.
Even assuming the clinic finds out every time a patient dies, the way the patients are in the study guarantees bad statistics. The clinic is in an expensive area, with many high powered jobs. The man who retires at 62 due to bad health, moves to Florida and dies a year later, is not counted.
In addition, assume that a 64-year-old man starts coming to the clinic and he lives past 65. He counts as someone who beat the odds. True, but he was not picked out of all people who entered the 45 to 65 age range, he was picked out of all people who reached 64 who were sufficiently health conscious to go to the clinic. His chances of reaching 65 were much greater than a random 45 year old.
If a school system decides to spend money to ensure that as many students as possible pass a certain test, they are likely to spend more money on students where it makes a difference. The students that are slighted are the ones who would never pass the test, no matter what was done, as well as the students who would pass the test regardless of the money spent on them. Statistics on test passing would be a misleading indicator of the general quality of the school. A school that spent money to improve all students would not have as impressive results.
At least in the case of the clinic, the statistics may have been gathered in ignorance, but the hypothetical school system is trying to beat the system. At least it is an honest way of beating the system, honest in the sense that if everything is revealed to the public, it won’t lead to indictments. Recently, the Atlanta school system tried a less honest way of trying to beat the system, and those indictments are making headlines.
In view of the stakes, I suspect many schools are putting their money where it can do the most good. By that, I mean the most good for the school system, not for the students.