Teacher Accountability and Rating

I know I've beaten this horse before, but more and new data (isn't that what we're always wanting) have made the idea of rating and accountability even more vexing. Let me explain using some testing outcomes from my school, 8th grade reading. Bear in mind, the outcomes I'm summarizing are all from the same tests, 8th grade reading WASL/MSP, 2006-2009. Also bear in mind that I'm assuming that test score outcomes are going to be used as a proxy for teacher performance.

The OSPI Report Card shows that the following percentage of our 8th graders passed the test (that means they got a score of 400 or higher). The number in parentheses following is the 7th grade (previous year) pass rate for the same group. (I joined this school in the fall of 2006, so include the 2006 score only to match up with the next set of scores.)

OSPI Report Card

2006--70.1

2007--74.2 (65.9)

2008--72.1 (74.6)

2009--79.6 (73.2)

The Evergreen Freedom Foundation reports a Fraser Institute score on their web site.

Their score is determined by simply averaging all the test scores for our group. Scores of 1 or 2 are 'not meeting standard,' while 3 or 4 signify 'meeting standard.'

1--MSP score of 375 or less

2--score of 376-399

3--score of 400-418

4--419 and above

2006--3.1

2007--3.05

2008--3.01

2009--3.25

This means that the average score for all the 8th graders on the reading test was 3.1 out of 4 in 2006, and so on.

Note that the percentage passing rate (top) only measures the proportion of 3s and 4s out of all tests taken.

The average score adds up all scores and divides by the number of tests taken.

A couple of things are interesting here. One is that in the OSPI scoring system, there is no change in the outcome measured when a student moves from 1 to 2, or 3 to 4, or if a student moves the other direction. Moving above or below 400 is all that affects the 'score' (percentage). In the Fraser Institute scoring system, movements from 1 to 2 and 3 to 4 (or reverse) do affect the overall average.

Another is that while the percentage of 8th graders passing in 2007 was greater than 2006, their average went down. This must mean that the 2007 students achieved more 3s and fewer 4s than the 2006 8th graders, who had a lower pass rate, by a higher average score.

So, which metric should we use? The average score instrument is blunt. Since a student earning a 400 and another earning a 418 both get 3s, we cast into the same category what are really quite different outcomes. But the percentage passing calculation is even more blunt. For example, 375 is the cutoff between a 1 and 2. For the percentage passing calculation, this difference means nothing. For the average score calculation, it means a lot.

If we were to be rated based on the whole class average score, I will take it as a success to get a student from 374 to 376. That's a 1 moving to a 2. If we use the OSPI scoring system, I will be less motivated (incentivized, as they say in economics) to focus on any such movement. Of course, neither scoring system takes account of a student's movement from 376 to 399 (which is a substantial increase), or 400 to 418.

And we haven't even mentioned whether an 'improvement over prior grade' metric is worth considering as a measure of teacher performance. Of course, one effect of that would be to increase a feeling of competitiveness among teachers...probably not great for school climate.

I'm not trying to make any moral, emotional, financial or spiritual judgments with this. I would like to have a more clear-minded conversation about just what we think we can measure when we talk about rating teachers and students.

Reading is Complicated; Literacy is really Complicated

I 'bumped into' a former colleague from my university teaching days. She's an English professor with a background in literacy programs, so we fell to talking about reading, schools, etc. She sent me an article about the role the bedtime story plays in various kinds of family environments. The article summarized the findings of an ethnographic analysis of three different groups and how they enculturate their children into the reading and learning process.

Group 1 begins reading with their children very early, provides a wide variety of reading material, and teaches (consciously or not) that there is much to "take away" from reading material. These parents show, demonstrate, teach their children to, I as I put it to my 8th graders, "interact with the text." Make inferences, draw conclusions, evaluate context, make comparisons, etc.

This 'order' of things, by the way, is how school is organized. Or, more to the point, school is designed by people who are good at this, and who think this reading/learning process is good and right. Kids from this kind of background will be much more likely to succeed at school than will kids from Group 2.

Group 2 parents read less to their children and tend to be more ambivalent about the material. Their interactions tend to involve more of the recall/recapitulation process, and the so-called higher level reading skills are engaged less deeply. Children in this group are less likely to grow up thinking there is a breadth and depth of worthwhile material in a piece of reading, and less likely to have the skills to find that breadth and depth. In other words, they're less likely to be effective at reading, though these prospects are worse for Group 3 children.

Group 3 parents don't read much at all to their children and tend to see much less value in the process of reading or in the reading material than do Group 1 parents, or even group 2 parents. These children are the least likely to be successful in the reading-driven elements of school.

Parents in all three groups want and hope for their children to be well-educated. The study makes clear that the latter two groups may be less clear about the best things for enhancing their children's learning prospects.

The study was concerned to show a number of things, but I am interested in a variety of implications for the process of schooling.

First, to the degree that one class room has a mix of these kind of students, a teacher essentially has 3 different kinds of teaching tasks. And when we're talking about higher grades, this problem becomes the more complicated. The group 1 students who've been keeping up with grade level learning expectations will be in a very different place for the whole of their academic experience than will Group 3 students who have likely fallen further behind with each passing year.

Second, the teaching of reading mechanics (decoding, fluency, etc.) is far from the most important aspect of literacy. The article points this out, but it can't be reiterated enough. At school we're fond of saying "Every child can learn," but the differences between these kinds of students compels us to think seriously about this trope and the degree to which we really can commit to the idea implicit therein.

Can a Group 3 child learn in the same way that a Group 1 child does? Is school constructed to reach one group more than the others, and to what effect on those other groups? Should school be constructed in some other way? Should and can we transform Group 3 kids into Group 1 kids? If so, how? Or should we try to create more work and learning opportunities engaging Group 2 and 3 kids on their own terms? If so, at what opportunity costs? Is the current model, structure, organization, etc., of schooling adequate to this diversification?

Third, the big question...is the standardized testing process inclined in favor of one or two of the groups and against the others? Would it be reasonable to figure out a different metric and learning goal for the different groups of kids?

Fourth, and I think most importantly, is there some way to engage Group 2 and 3 parents earlier, and get them to help their kids see the value of the reading and learning process? There is only so much a teacher can do to reconstruct a child's enculturation toward reading and learning. Getting more done earlier will be better for a child.

Fifth, does this mean that programs like Head Start are indeed important?

Sixth, I'm sure there are more implications to consider...let me know what they are.

...from that same colleague....Who's to say that group 1 literacy is the only way to go? There are certainly benefits to the second and third situations that get denied or ground out of students in schools.

Awful

Out of Virginia, the tragic news that a teenager committed suicide, in part, apparently, over his despair about the discipline his school meted out to him over his purchase of a legal drug substitute. The Washington Post's Education blogger (I guess you'd call her) wrote of how the zero tolerance policy is counterproductive, even, so it seems in this case, destructive.

Fair enough. This very sad case seems to have some elements of the absurd.

So does much of school life, unfortunately. To wit, zero tolerance.

I agree that zero tolerance rules tend, on average, to be onerous and inflexible. The bind schools often find themselves in, though, is that flexibility in discipline, which any parent knows is required in child-rearing, will be challenged the moment a parent gets the idea that disciplinary decisions have generated different outcomes and therefore are unfair.

There are certainly kids in my school whom I discipline differently because I can discern regret, remorse, repentance, etc., and I know they've 'learned' just by our conversation. Other students, though, seem undaunted by even the prospect of 3-hour Friday detention after school.

For the sake of fairness (and not wanting to have to deal with the parents who'd cry foul), the easiest route is to implement completely even discipline--zero tolerance.

Also, though, what do you think the coverage would be like if the school had given a 'light' punishment to a student who then goes out and makes real trouble after being 'let off' by the school? This is why risk-management (i.e., lawsuit avoidance) is so powerful an idea (and dept.) in school and society

I don't like zero tolerance, but the schools (indeed, public agencies generally) are in a tough spot either way. Imagine trying to implement a 'some tolerance' policy, especially in an environment of low trust, as people now generally have for the schools.

Markets don't solve everything

Or, at least philosophical claims about what the market can accomplish don't solve everything.

The appeal of market logic is compelling. I enjoy the wider availability of good things at low prices because of markets. The logic of free trade is ineluctable. Just read The Choice. But then read Politics and Markets, too, so you can think about the balances we as people strike when we organize our political economic systems.

Yes, I'm a free marketer, but at the same time I'm glad we have labor laws protecting children, and industrial laws protecting adults, for instance. These are choices we make in political society. The market would't 'care' (I use the quotes because the market doesn't have agency, so I don't really like to personify it) whether a child were put into dangerous working conditions, but people can and do care about just such things.

My point is that the claims of market supporters are well-founded--efficiency gains, maximization of production at lower prices, etc. But the moral choices we make as a society are a feature of social or political choices. Markets don't choose because they don't act. People act and choose.

So, all that to say, I find both the left and the right tiresomely ideological about so much of what they say in politics. But I'm most perplexed by the right, because that's the direction I incline. I'm toughest on those who sound and think like I do, or maybe it's that I think like they do, but I don't want to sound like they do at all.

Anyway, the Evergreen Freedom Foundation's market logic applied to schools--My comments in regular font.

Summary of findings
Many school districts could boost student achievement without increasing spending if they used their money more productively. An Arizona school district, for example, could see as much as a 36 percent boost in achievement if it increased its efficiency from the lowest level to the highest, all else being equal.

What is more productive spending? How do we know it when we see it, other than seeing that success measures (scores, presumably) go up. How is efficiency defined in connection to test scores without the whole thing becoming a tautology? We need to have an a priori definition of efficiency and productive spending, and this summary does not provide that.

Low productivity costs the nation’s school system as much as $175 billion a year. This figure is an estimate; our study does not capture everything that goes into creating an efficient district. But the approximate loss in capacity equals about 1 percent of the nation’s gross domestic product.

Again, what is efficient? I suspect they define efficiency (as some have bandied about in blogs and comments) as things like cutting the excess principals in a building. My school is operating with one instead of two principals right now. It makes a lot of things much less productive. Discipline management, for instance, is less clear and cohesive, and that intrudes into every aspect of the school day. Talk about productivity disruptions, but I don't know how to quantify them. Of course, if you want the best test scores for the least spending, you need to figure out how to get rid of the low performing students. They're a horrible drag on efficiency. The struggling learners take up a lot more time and attention than the students who are on grade level.

Without controls on how additional school dollars are spent, more education spending will not automatically improve student outcomes. In more than half of the states included in our study, there was no clear relationship between spending and achievement after adjusting for other variables, such as cost of living and students in poverty. These findings are consistent with existing research: How a school system spends its dollars can be just as important as how much it spends, at least above some threshold level.

Again, seems easy to say, but it remains unclear how a school system should spend its money. And a little understanding of Kenneth Arrow's impossibility theorem would help here. Arrow showed that the collective choice among 3 options might not actually reflect a rationally preference ordered choice of the largest number of deciders in the group. Individual preference orderings don't always coherently transfer to collective choice. Arrow showed this for choices among 3 candidates in an election. Imagine how much harder the collective choosing process is when you're talking about much larger option sets with much more complicated tradeoffs among them. We had this problem just the other day when talking about our master schedule. I don't know what the efficient outcome would be in the case of scheduling. And, frankly, what works effectively for some students won't work as well for others. That second group may need something that turns out to be detrimental to a third group, whose dominant need actually undermines group one...Arrow's impossibility....!

Efficiency varies widely within states. Some districts spent thousands more per student to obtain the same broad level of academic achievement. After adjusting for factors outside of a district’s control, the range of spending among the districts scoring in the top third of achievement in California was nearly $8,000 per student.

What, I wonder, were the factors "outside the district's control." One factor that has a lot to do with success is level of parent involvement and general family stability. My district probably looks pretty efficient--we spend about $6000 per student, and our test scores are pretty good. But I tell all who ask that we've got pretty involved parents and pretty stable families. If parents and families matter, then to maximize efficiency, make sure your district is doing well in this area.

More than a million students are enrolled in highly inefficient districts. Over 400 school districts around the country were rated highly inefficient on all three of our productivity metrics. These districts serve about 3 percent of the almost 43 million students covered by our study.

And, at the same time, The Alliance for Excellent Education reports the following:

Approximately two thousand high schools (about 12 percent of American high schools) produce more than half of the nation‘s dropouts. In these ―dropout factories, the number of seniors enrolled is routinely 60 percent or less than the number of freshmen three years earlier.

Eighty percent of the high schools that produce the most dropouts can be found in a subset of just fifteen states. The majority of dropout factories are located in northern and western cities and throughout the southern states.

Unless those 2000 high schools are all from those 400 districts, efficiency and retention may not be highly correlated. So, which do we, as a society, focus on? Or, to say it differently, how do we get efficient and retain students at the same time?

High-spending school systems are often inefficient. Our analysis showed that after accounting for factors outside of a district’s control, many high spending districts posted middling productivity results. For example, only 17 percent of Florida’s districts in the top third in spending were also in the top third in achievement.

Again, which factors? If those districts were paying for mandated programs to some target population, they may not be able to get out from underneath that spending requirement. A district with a high volume of special needs is going to be much less efficient, whatever way you define it.

Students from disadvantaged backgrounds are more likely to be enrolled in highly inefficient districts. Students who participated in subsidized lunch programs were 12 percentage points more likely to be enrolled in the nation’s least-productive districts, even after making allowances for the higher cost of educating lower-income students.

If low socio-economic status correlates with lower parent involvement, lower reading success at early grades, etc., then those districts will have more special program requirements to meet and will be less efficient. Further, kids who fall behind early will require remediation, and while those programs achieve their curricular goals, without reinforcement and persistence outside of school, the gains are diminished and the student remains somewhat behind, thus requiring more remediation, and so on.

Highly productive districts are focused on improving student outcomes. We surveyed a sample of highly productive districts to learn more about their principles and practices. The districts that performed well on our metrics shared a number of values and practices, including strong community support and a willingness to make tough choices.

This bit is non-sensical. A good district focuses on student outcomes, and they "performed" well on strong community support. An implicit acknowledgment that community/parent involvement is a boon to a school. Of course, in great degree, community support is up to the members of the community, not the school.

States and districts fail to evaluate the productivity of schools and districts. While the nation spends billions of dollars on education, only two states, Florida and Texas, currently provide annual school-level productivity evaluations, which report to the public how well funds are being spent at the local level.

The quality of state and local education data is often poor. In many instances, key information on school spending and outcomes is not available or insufficiently rigorous, and this severely impedes the study of educational productivity. For instance, we did not have good enough data to control for certain cost factors, such as transportation. So a rural district with high busing costs might suffer in some of our metrics compared with a more densely populated district.

The nation’s least-productive districts spend more on administration. The most inefficient districts in the country devote an extra 3 percentage points of their budgets on average to administration, operations, and other noninstructional expenditures.

Some urban districts are far more productive than others. While our main results are limited to within-state comparisons, we were able to conduct a special cross-state analysis of urban districts that recently participated in a national achievement test. After adjusting for certain factors outside a district’s control, we found that some big-city school systems spend millions of dollars more than others—but get far lower results on math and reading tests.

Tiresome to the left and right

The Evergreen Freedom Foundation (or whatever it's now called) is an organization that follows WA state politics, including education.

They rank schools. Here's my school's ranking.

2008-09 Rank 71/431 Rank in the most recent 5 years 105/382

School Information
Grades	6-8
Enrollment	646
Low Income (%)	15.7
Ethnicity (%)	Wh: 58.2 As: 12.1
Tests not written not exempt (%)	0.3

Academic Performance	2005	2006	2007	2008	2009	Trend
Avg level: Reading	3.1	2.9	3.0	3.1	3.1
Avg level: Writing	2.6	2.7	2.8	3.0	3.2
Avg level: Math	2.6	2.5	2.6	2.7	2.8
Avg level: Science	2.2	2.5	2.5	2.6	2.9
Tests below standard (%)	39.8	39.7	34.4	30.3	26.6
Low income gap: Reading	n/a	n/a	n/a	N 0.5	N 0.6	n/a
Low income gap: Math	n/a	n/a	n/a	N 0.7	N 0.7	n/a
Overall rating out of 10	6.5	6.5	6.9	7.3	7.5

Those 2 and 3 scores are on a 4-point scale. 3 and 4 are at and above standard.

The score they list is the average for that whole grade level.

Nice, too, that they tell how many tests were failed in the year.

Most of my colleagues don't like EFF. They're a conservative group that thinks school teachers get paid too much and accomplish too little, so the thinking goes.

I'm more bothered that they simply offer up the same kind of material (data, I guess it is) as anybody else.

Won't anybody with any policy weight discuss this any differently?

Math made easy

These gags--which I assume are fakes--go around the teacher e-mail circuit about once a year. Real or fake, they're fun.

A most poetic response to math and science. I can't understand how this earned a zero.

More importantly, read question (b) again closely.

"Does the object continue to move after it comes to rest?" This needs to be asked?

Meaningless Language

From Politico--Obama’s education secretary Arne Duncan sounded surprisingly like the Republican governors when he told teachers unions and administrators at a conference Tuesday in Denver, “Clearly, the status quo isn’t working for children.”

Really? Seems likes the status quo isn't working for adults...taxpayers, budget writers, lawmakers, etc.

And just what is the status quo? It's a lot different at my school than it is at my children's schools, which is different from what it is at schools in other parts of our city.

I find it almost fortunate that something as sprawlingly amorphous as 'the schools' probably really isn't fixable on the grand scale...anything we think is a fix is probably as destructive as it is helpful, so I'm glad such fixes are hard to implement.

3rd Quarter of the Year

Our third quarter just started. We have officially entered 'Testing Season.' Be on the lookout for any free-roaming 'bubble' students.

Let me explain.

With about 2 1/2 months until we take the test, we all begin to think more specifically about it. As I've said before, I spend some time on test strategy, reading skills specific to the test, and extra work for the so-called bubble students--those who were very close to passing or just barely passed last year.

After having read the very interesting Nudge, I can't help but think about the testing season as a 'choice environment.' We make (or have made, or have had made for us) choices about how we undertake teaching, and what we think of as 'learning,' and the incentive structures that the testing season creates shape teaching and learning in a particular way.

We are, for instance, planning a little test-boosting 'extended learning opportunity' (i.e., after school help) in the next few weeks. This will be offered to these bubble students, and most won't take it. Last time we offered this, 2 of the 20 or so invitees came. We'd love to get some to do the extra preparation. Just think if we got 10 'almost passed last year' students to 'passing this year.' That would make our overall test numbers go up nicely.

Think of what's not being done, at the same time. The students who are safely in the passing zone, who could probably forego even the test prep we do in our regular language arts day, are left doing less than they probably could do. During test season, we don't really mind, though. We're not really held accountable for changes in student scores, only whether they pass or not.

The more I can get to pass, the better. It doesn't really matter if I get students to enjoy reading more, or do more of it. If I teach test strategies and those numbers go up, I'll be a hero. And believe you me, at this point, test strategies probably get as much value-added (for test scores) as anything else.

So, the question still remains, what does passing the 8th grade reading MSP prove other than the fact that you passed the 8th grade reading MSP? I'm not sure. If I can get marginal readers up over the magic bar, does that mean they're now better students, will be more successful in their pursuits, are smarter than before they passed?

It's unclear that we can say YES to any of those. But mine is not to question why, and all that good stuff.

Time and Place

So, blogging about students can be tricky....

My colleagues and I often tell students who are being a bit too cute or funny or silly, "time and place." There's a time and place for that...it's not now and here, but there is a time and place for that.

Have I ever complained to colleagues about students? Yes. It's commiseration that helps you stay sane.

But there's another sort of odd thing going on, for me. I sometimes 'complain' to my colleagues in order to check or monitor my own conduct. Sometimes it's like a strange confessional process.

"Can you believe Billy did....?"

"So I told him...."

And there's a little bit of trying to commiserate with a colleague in order to gauge how close I am to being out of bounds.

There is some guilt mitigation going on, because sometimes my frustrations take hold of me, like when....Oh forget it. I couldn't describe it. You should just come to my room for a day.

Democratic Centralism?

I'm giving away my old Soviet politics/Cold War academic background to call our staff meetings at school 'democratic centralism without the Leninism.'

Communist parties often run along democratic centralist lines. Elections are supposed to select the leadership of a rigid hierarchy that really makes policy...without too terribly much input from the electorate. It's more centralism than democratic.

So, why are our staff meetings like this?

Today, we met to give our wish lists for school schedule parameters. The discussion is complicated by at least two important realities. First, preferences diverge. More than once someone suggested one thing (common grade level planning time, for instance) only to have someone else say their group didn't want that. One group finally just said we want 'this' and 'not this,' both.

Second, the sets of issues that we're considering are not of the same order. Whether we have a common planning time may not have anything to do with avoiding multi-grade teaching responsibilities. In such a situation, the preferences are particularly difficult to rank, as we (or at least, I) have difficulty comparing them.

Such is the nature of 'democratic' discussion.

So, ultimately, the district leadership will take a significant role in the whole thing. Thus, the centralism part.

As long as it's evidence-driven research-based best practices of what's good for kids we'll be fine.

Hear hear!

Saw a short piece on Why I Dislike Education Reform. Sums it pretty well, I think. I don't know about the 'cheating' observation...not sure I really understand what he means.

But I--and my colleagues--have thought about the other 6 quite a bit.

"Status increases aggression"

The higher your social status, the more aggressive you become. The movies got it right...at least for girls. The study reviewed at that link used survey results over the course of a year (I think) to determine that as a teenager moves up the social hierarchy s/he picks on people more.

Correlations and causations are notoriously difficult to evaluate, especially in a study based on self-report survey data. Asking students to name those they've picked on and that have picked on them in the last year (or whatever time period) might just as likely elicit a peculiar kind of status-establishing response.

Such reasoning would be unconscious, but would go like this, "I've picked on people that I would have used to have thought a bit cooler than I was, so since I've picked on them, I must be cooler than I was before. And that really cool person picked on me....Not bad, I'm movin' up."

Is it possible, in other words, that coolness self-talk includes an inventory of how a respondent has used the idea of bullying for social mobility, in his/her own mind?

Harassment, Intimidation, Bullying (HIB, it's now called) are widespread and corrosive at school, no doubt. They're also nuanced, subtle and often hard for adults to detect. The part I find most difficult is that a lot of kids don't want to imagine that their behavior might be destructive to someone else. The rationalization, justification, deflection impulse and machinery are amazing.

I see this problem most clearly with the pervasive "How many of you have ever been bullied?"

I hear a lot fewer adults asking students whether they have ever bullied. Clearly, almost of all of us have said destructive or hurtful things to others. We're both perpetrators and victims. Maybe it would be helpful for more of us to seriously examine our role as even low-level perpetrators.

More on Reading

Just got an e-mail for more continuing education opportunities, this time about reading, so I should pay attention. Doubly so, since they promise to tell us about the Matthew Effect. Never heard of this, so a little Googling is in order. Simple concept, really. The rich get richer and the poor get poorer…in this case, as readers.

Those who learn to read well by 3^rd grade tend to just get better and better at their reading. Those who don't, don't. It must be true…studies show it.

Obviously, reading teaching and intervention for strugglers really need to be thorough and intensive through age 8 or 9. Again, reading patterns and modeling at home seem all the more important. If the schools and parents together don't get a youngster to effective reading in those first few years, there will be persistent difficulty in reading for that student.

I am perfectly willing to say the schools need to be sharp about this. But at the end of 3^rd grade a child will have been in school for the equivalent of about 600 full school days. Even if 1/4 of the school day is given over to reading instruction, that's about 900 hours in 4 years. 1 hour of reading instruction in school, though, is probably worth about 10 minutes of what some one-on-one out loud reading at home would do.

So those signs on the school readerboard that say "Read 20 minutes a day with your child" should add, "it's a day's worth of school."

A Brag and an Observation

My 15-year-old son took some reading test or another at school yesterday. A computer-driven test, they got the results immediately. He got a 1700+. Since I have no idea what the test is or what the scoring scale is, I don't really have much of an idea what that score means. Except his teacher did say he'd never seen a score that high. My son has typically gotten very good scores on the various reading tests the school has given him. The test they took in middle school gave a percentile score, and his was typically in the high-90s, often 99th.

So, yesterday's test was to read a passage and then choose the best word (I think he said) to complete the sentence to make the best summary of the passage. I've not seen this particular (and quite specific) assessment tool, though I'm sure there's plenty of research-based evidence that using it is the best practice.

But what am I to make of all the data about my son? He had all these great scores, and for a few years of his school career he didn't do all that well on the one test that mattered--the WASL. Oh, he passed, but one year it was a little closer than I was comfortable with. And his score certainly was not in the 99th percentile.

How can I be sure that what the WASL (now Measurement of Student Progress) measures is really the important stuff to measure? How can I be sure that what all those other tests measured was the really important stuff?

I will elaborate in a forthcoming post.