Education: Inside The New SAT

(9 of 10)

The other reason the writing test will be less reliable is that human beings, not machines, will grade the essay. In June, I participated in a mock grading session with members of the College Board's writing-development committee. We read 15 essays by kids who had taken a pretest; they had been given 25 minutes to write on a topic I can't reveal, since it may appear on a future SAT. We scored the essays on a scale of 1 to 6, 1 meaning "very poor" organization and development and 6 meaning the student organized her thoughts, displayed "facility" with language and "insightfully addressed the writing task." Such standards are quite rubbery, as we discovered: of the essays we read, we 15 readers uniformly agreed on a grade for none. On most of the essays, the lowest score was 3 full points away from the highest.

Our grading also rewarded the blandest essays. I gave a 5 to a kid who had written a funny, subtle first-person account of a friend who had slacked off his studies and begun dressing "like a pimp" in order to impress the cool kids. The other graders gave the essay 2s and 3s; there was one 4. (Our scores didn't count for anything. On a real test, the raw 1 to 6 score will be combined with the raw score from the multiple-choice grammar segment and translated into an overall writing score on the traditional 200-to-800 scale.)

College Board officials are acutely worried about the subjectivity. Parents have already called to say they don't understand how a 3 differs from a 4. In response, the board is conducting further studies of the test and developing more consistent, less pliable 1-to-6 scoring points. Graders hired by an Iowa City, Iowa, company, Pearson Educational Measurement, will actually score the essays. Pearson trains its scorers to follow the 1-to-6 guide closely. Two of them will read each essay, and on the basis of the firm's experience with exams in Texas and elsewhere, they will disagree only 30% of the time. When they do, a third--and, if necessary, fourth--scorer will resolve disputes. It's a thorough system, but it will be expensive. The pressure to read fastand to reward competent but formulaic essays will be massive.

In some ways, Gaston Caperton has an excruciating job these days: he must sell the New SAT even as he defends the current test against its critics. He cannot say the College Board was wrong about the SAT all these years; nor can he say the board was wholly right about it. That's why he argues, on the blade of a knife, that the SAT is not becoming a typical achievement test but that it is coming "into a great balance" between a test of "critical reading, comprehensive writing and higher mathematics" and a test of "learned skills that you use to reason."