Jane Says . . .

"I am puzzled by your statement that a reader can't rate acontextual writing. (Maybe that "a" is a typo?)"

No, here's my thinking -- a reader will find it difficult to rate "real" writing outside the context within which it was written. Here's my experience with the issue. As part of SWIC's outcomes assessment initiative, I was on the cmte tasked with developing some way of assessing (efficiently) the writing SWIC students are doing across the curriculum.

Our first task was to develop some understanding of how we are defining good writing. We spent a great deal of time developing a workable rubric to evaluate writing across the curriculum. Then we collected essays from different disciplines, asked for volunteer readers, did a few norming sessions, and then evaluated papers in earnest.

Let me make a long story short -- we simplified the grading rubric to a bare-bones version, asked the classroom teachers for the assignment and grading criteria, but it was still difficult for us to assess the writing other than on a very basic level -- does it have a purpose, a sense of structure, adhere to conventions, etc, but when it came to making fine-tuned sorts of decisions about the quality of thinking / purpose, or whether a piece of writing was organized well, it was very difficult because we were evaluating the writing outside of the context within which it was written.

For example, we all know teachers who privilege obedience over substance, and we were given papers from teachers such as this. The teacher said in the instructions the lab report should be a single paragraph, which is just wrong and bull-headed, so we were given student samples that were three pages long, with no paragraph breaks, as the teacaher had requested; however, there were other students, recognizing this is simple wrong rhetorically, who shaped their writing rhetorically. As a group, we struggled with how to rate such writing.

Or, worse yet, were papers written for a math class. These papers were explications and applications of mathematical and geometric principles; there were few of us who knew whether the quality of thought displayed was acceptable or not -- acontextual.

I think I read an article by Peter Elbow, no, wait, it was an article by Yancey about the development of big-picture, insitutional and for purposes of placement, writing assessment. She broke it into three stages: the first stage was indirect assessment, based on grammar tests. This is highly reliable, but of questionable validity. The second stage attempted direct assessment, but had difficulty with reliability. So, raters were assessing real writing, but had consistency rates slightly better than chimpanzees. The third stage, spearheaded by Elbow and Belanoff attempted to ensure validity, measuring writing directly, while improving relaibaility, and their solution was to assess writing contextually, while the raters, classroom teachers, did so collectively, thereby, negotiating the "community standard" and improving reliability.

This is why I say it's impossible to assess writing, well, outside of the context within which is was written.

Jane Says (Have you heard of Jane's Addicition, the band? They have a song called "Jane Says" and everytime I write that, I have this melody playing in my head):

"With the emphasis on high stakes testing today, I'm inclined to consider the timed essay test (or what we call "constructed responses. . . .)" a specific mode of writing with its own peculiar context -- an arbitrary prompt, an artificial time limit, and no obvious purpose or audience other than to convince the reader that you know how to write. This IS, in fact, a situation that occurs on occasion even for adults (e.g. some job applications), and I don't object to having school and college writers practice it.The problem comes when we make the assumption that performance on this mode of writing is representative of a writer's competence in any other mode.What do you think?"

I agree with you that this sort of writing is out there, and students should understand when they are presented with this sort of situation what's expected of them and what they need to do. My assumption, not always correct, is that most students know how to do this already, namely because they been asked to do it many times in their past as writers. Like the five-paragraph theme, most of my students adopt the sort of stance required of these high-stakes timed tests in all their writing situations. This is writing that the writer has little relationship to, or with the ideas and materials presented, and this is what they try to do when they get into college. In some cases, this sort of writing is re-emphaized for them in other classes they are taking outside the English department, and they are shocked to find that in our classes, they cannot get away with this sort of writing. It's really a form of non-writing rather than writing.

Damn it, I can't think today -- somebody, my brain is resisting me, calls this sort of writing THEMEWRITING, and it is characterized by form (5PT) and content (empty). Okay, got it, William Coles.

I think, again, students need to be able to recognize the difference between the high-stakes sorts of standardized test writing and "real" writing, and be able to recognize both, know when to do each, and what is required of each.