Open Season on Reason

Standardized test season is open.  
As American school children gear up for the annual battery of examinations, school districts scramble to find computers and rejigger the daily schedule to accommodate the serial rounds of testing that will span 6 weeks or more.

With this also comes “get rid of the bad teachers” season.  In New York, which leads the way in adopting every new wrinkle in education reform, Governor Andrew Cuomo noted that only 38 % of New York State high-schoolers achieve “college readiness,” according to their standardized test scores, while 98.7 % of New York’s teachers are rated “effective.” “How can that be?” he mused. “Who are we kidding, my friends? The problem is clear and the solution is clear. We need real, accurate, fair teacher evaluations.”

This week, the Washington state Senate embraced this view, passing a bill mandating that student test scores constitute at least part of teacher evaluations.  Doing so, it is hoped, would allow Washington to retrieve its lost No Child Left Behind waiver and reclaim $40 million in federal money.

Unfortunately, the problem and solution are far from clear.  The reasoning implied by Cuomo and our Senate strains credulity, because it neglects the strange incentives already emerging from the implementation of this thinking.

As the emphasis in education shifts towards standards-meeting, teaching energy and focus likewise moves toward the level of the standards. Students who are comfortably above the standards needn’t be worried over or engaged educationally. Students far below the standards...well, they can likely get special services.

In such a climate, schools increasingly target the so-called bubble kids (those just below the passing mark) in order to get them up and over the top, into passing.  Accomplishing this makes the school’s pass rate go up, and the school is deemed more successful. High scoring kids scoring even higher means nothing under this incentive structure. Neither does fantastic improvement that falls just short of passing.

The scores-evaluations connection also indulges bad logic. Using test scores to demonstrate teacher quality is a claim badly constructed, as it does what’s called sampling on the dependent variable by using one measurement for both the cause and effect.  In other words, the hypothesized connection between teacher performance and student scores is tautological, and therefore reveals nothing. Here’s why.

The claim implied by Cuomo and our Senate is that Effective Teachers cause Passing Students, or Teachers cause Test Score changes. Seems clear enough...bad teachers generate lower test scores; good teachers generate higher test scores.  But notice that the outcome—Test Score—actually provides the measure of both the teacher and the student--one piece of data measures both the cause and effect.

To make this plausibly valid we must define measures of Teacher Impact—or “Good” and “Bad” Teacher—prior to and separate from Test Scores.  In other words, creating a fair and logically valid assessment of teachers’ impacts on students requires that we define and measure Good and Effective teaching prior to looking at student test scores. Unfortunately, this is not how the analysis proceeds, because it's far too easy to simply define Teacher Quality by Test Score results.

This kind of slack thinking allows all manner of strange things.  Take, for instance, the growing movement to deem Advanced Placement course participation (not AP test score performance) as an indicator of college readiness for a high school student.  Tacoma, among others, lauds itself for how these more rigorous courses motivate students to greater success. 

They are, for instance, only the second district in Washington to implement Academic Acceleration—automatically enrolling all students in advanced courses.  This may appear to meet the needs of the neglected high achieving students, but that’s not the stated intent of the program. 

Rather, this will better prepare formerly lower achieving students for college, and so on.  But here, meeting standard (passing at 3) doesn't seem to matter as much.  Of the nearly 1800 AP exams taken last year by Tacoma students, only 31% earned a passing score.  Two schools had pass rates below 15%. 

So, test scores are the Holy Grail in one case, irrelevant in the other.  Is this good educational policy, or merely ideology?