A new discussion has just popped up on the TechRhet list regarding a recent article in Science, "Computers as Writing Instructors" (subscription required). In any case, this is old news in our field to some extent. The GREs and similar tests have been machine grading essays for years. This article also touches on the potential value of computers in providing feedback to student writers in helping them write their essays.
The fundamental argument for the validity of machine grading is simulation. It's essentially a version of the Turing Test. Imagine three rooms. In one room is a machine grader. In another room is a human grader. A series of essays is given to both the machine and human graders who turn back grades. If a second human sitting in the third room cannot distinguish the human from the machine, then the machine is an effective grader.
In a way this isn't suprising. What is a norming session if not an attempt to turn humans into predictable grading machines? To get humans to simulate some "ideal" reader? The faulty premise is the idea that readers should respond in the same general way to a given text, that a text should elicit common responses from readers. To create such responses requires turning reading into a kind of mechanical, cybernetic process. Certainly there are elements of reading that might be understood as mechanistic and/or cybernetic. I would think that nearly any human activity might be partly understood in such terms.
So there is a curious outcome here. Students are trained by computers to write essays that will be read by computers. They are taught to simulate writing. And perhaps simulation is good enough? It is a curious exigency though, isn't it? To compose for the purpose of satisfying a machine. No doubt there is a degree to which such activity is important: search engine optimization for example. But what about beyond that?
The primary criticism of machine reading is that computers can't tell if an essay really means anything. Ah, but that's a criticism that cuts in several directions.
Students write an in-class essay or the conventional research paper. Do these texts mean anything? They get a grade, so that's some value. But to whom do they have meaning? Does anyone really care about them? Do they serve any real purpose? Perhaps the activity of writing the text has had an educational purpose… hopefully. But what about the text that results? For whom does it have meaning? Our students are often asked to write as if their texts had meaning and purpose. So we are already engaged in simulation.
Besides, machines cannot measure the meaning of a text, because the meaning is not located there. It is supplemental, in the Derridean sense. Nor was the meaning present in the authorial mind. Meaning arises in the mind of the reader in the intersection with the text in a particular networked event/moment. Norming is a kind of cybernetics that is meant to enact a gravitational pull on the event of reading, to draw those events down onto a representational plane in predictable ways. But if you norm differently obviously you get different results, different meanings from the same body of texts. So exactly what kind of screwing around are we engaged in here anyway?
The only reason we find ourselves in this situation is because we began using humans as computers (remember the word "computer" used to refer to a human who performed calculations before machines took over). Human readers served as an interface between a text and the machines that would take our scores and analyze them. Eventually the machines have become (or will become) smart enough to do the job on their own.
Next week, my daughter will be taking the 4th grade ELA test. I'm sure she'll do fine, and the machine graders (human or computer) will respond appropriately. Today I got letters from two 4th grade Nicaraguan children that we sponsor. The letters are not so well-written in the machinable sense. Yet they have a degree/intensity of meaning that I doubt my daughter will be able to summon on her test. What is it that we are trying to accomplish here? Are we trying to teach students to produce machine-like, meaningless pablum? (how can you eat your pudding if you don't eat your meat?)
The Science article explains that these computer programs are necessary because teachers cannot read and respond to as much student writing as the students should be doing; so the machine reads them instead. Hmmm…. what other possibilities could there be I wonder? …. Maybe the other students? Maybe the could be reading each others' work? Maybe they could even actually be writing to one another? Maybe they could be using these networks to write to other students around the world? Maybe they could be composing texts that were addressed to other humans rather than to machines and which might actually have some real meaning and value?