So Unstable It Boggles the Mind
Part Four of the NEPC Report on Dr. Chetty’s Research
Find out what's happening in Woodinvillewith free, real-time updates from Patch.
In his NEPC report on the use of Value-Added (VAM) scores to evaluate teachers (Blog postings -NEPC Parts 1, 2, 3), Moshe Adler contends that, “Value-added scores in this report [Chetty] and in general are unstable from year to year and from test to test, but the report [Chetty] ignores this instability.”
Find out what's happening in Woodinvillewith free, real-time updates from Patch.
According to Adler, those Value-Added (VAM) scores varied widely from year to year, which is why he describes the scores as “unstable.” The measure of variability/invariability, or stability/instability, is called “autocorrelation”. The highest value of autocorrelation is 1, which means there is no instability in the scores at all. Sources of scores that are predictable, and therefore stable, tend toward a value of 1, while sources of scores that are unpredictable, and therefore unstable, tend toward a value 0 (zero). The autocorrelation values in this [Chetty] study “ranged from .23 to .48.” Another report, from economists McCaffrey, Sass and Lockwood and using data from four large Florida counties, had autocorrelations that ranged from “.008 to .36”
Flip a coin 1000 times and you’ll probably come up with an autocorrelation of .5 for the next flip of the coin. Neither of these reports show results that reach the level of correlation or stability of a coin flip. And these are the results upon which Judge Treu’s decision is based.
A third study by Koedel and Betts used San Diego data. The “average results was that 13% of teachers who were at the bottom 20% of the VA scale in one year were at the top 20% the following year, and 29% of those at the bottom 20% were at the top 40% the following year.” This shows us just how far off the mark Value-Added (VAM) predictions can be.
It’s clear that Value-Added (VAM) scores are highly unstable. They are too unstable to tell us anything useful about predicting effective or ineffective teachers. Yet, those are the scores Dr. Chetty refers to in his testimony in the California teacher tenure trial, Vergara v. California. And Judge Treu has accepted Dr. Chetty’s testimony without reservation and made it the foundation of his decision.
Less reliable than a coin toss, but it may well become the basis for denying 97-99% of California teaches their due process and tenure rights. It boggles the mind.