Subscribe via RSS Feed

The Next Step in Destroying Academic Labor

[ 128 ] April 18, 2012 |

Our highly valued commenter John Protevi leads us to the latest plan to eviscerate the humanities: get undergraduates to grade papers for free!

Koller, an artificial intelligence specialist who has taught computer science at Stanford since getting her Ph.D. there at age 25, said that the challenge of assessing student work in humanities-oriented MOOCs could be addressed through a system of “calibrated peer review.” Human readers, plucked from the ranks of the course registrants, could read short essays written by their peers and rate them according to a rubric developed by the professor. A critical mass of deputized students should be able to evaluate an essay “at least as [well] as a pretty good [teaching assistant],” Koller said in an interview.

I think I’ve written about Koller before though I can’t find any references to it, but I love the idea of a university professor spending her career dedicated to helping universities not hire academic labor. It must be very rewarding. And hey, get rid of all the TAs and we can hire another administrator for 125,000 a year!

Share with Sociable

Comments (128)

Trackback URL | Comments RSS Feed

  1. DrDick says:

    I am particularly intrigued by the idea of having semi-literate undergraduates, with little or no knowledge of the subject matter, do the grading. This is especially true given the incentives to give everyone else a high grade in the hopes they will do the same.

    • Erik Loomis says:

      Although whenever I’ve seen undergraduate peer review of papers, damn they can be brutal to each other.

      • Eleanor says:

        How did you get your undergraduates to do that? Whenever I’ve tried peer review they all only say nice things, or circle one (out of many) spelling or grammar errors.

        The other problem, which I’d think a person w Koller’s background might have stumbled on, is that there is a fair amount of research suggesting that without a lot of prior training in using them, even shared rubrics don’t produce the same grades. There’s a term for that… but I can’t remember it. Frex: a department develops rubrics together, hands them out, every paper gets graded by at least two faculty members, and the correlations are not flat. Even using a collaboratively developed rubric doesn’t result in identical, or even necessarily similar grades. Especially for the bulk of papers in the middle (usually it’s easy to spot the best and the worst…).

        • Erik Loomis says:

          I don’t do it, I’ve only seen others do it.

        • DrDick says:

          That is my experience as well. The term you were looking for BTW is “inter-rater reliability.” It is something we cover in social science research methods.

          • Eleanor says:

            Yes! That’s the term. Thanks. I didn’t have to take social science research methods (Phd in History in the 90s)…. and, in retrospect, it would have been worth the time. (My mom was always after me to take statistics too, but, did I listen?)

        • Bijan Parsia says:

          My guess is that she’s not aiming for high interrater reliability per se, but synthesizing a highly reliable assessment out of somewhat noisy input.

          (There’s other problems even with highly trained rubric users: e.g., There can be odd biases that get suffused through the trainees that distort the assessment. I just read a paper which discussed this though it eludes my grasp at the moment. They looked at some English composition programs with really elaborate rubics and training and cross checking and found weird systematically odd application of the rubric based on outside the rubric considerations, e.g., some errors were forgiven or downplayed because “the student seems to be a good writer, so this is a minor error” vs. other essays where similar problems got harsher marks. I’ll see if I can dig it up.)

        • justaguy says:

          “How did you get your undergraduates to do that? Whenever I’ve tried peer review they all only say nice things, or circle one (out of many) spelling or grammar errors.”

          Give them a rubric that asks for specific positive and negative feedback, and collect the feedback forms with the final paper – telling them that part of their final grade will be based on the quality of feedback that they give their peers.

          I’ve done this with rough drafts on big papers and its worked fairly well.

    • joe from Lowell says:

      Good point.

      Having students read and comment on each others’ papers can be a valuable exercise, both for the reviewer and for the writer who gets the feedback, but it’s not supposed to replace grading by the instructor.

    • mds says:

      Perhaps someone could acquaint Professor Koller with game theory. Because there do seem to be a few potential outcomes she apparently hasn’t considered.

      • Bijan Parsia says:

        I really wouldn’t condescend to Koller in this way. Just look at her website and you’ll see how silly and offensive this comment is.

        AFAICT, all the comments in this thread about Koller’s personality and career are worst than off base.

  2. George bush's feminism says:

    Getting a comp sci PhD at 25 basically guarantees that you’re not the compassionate or empathetic type. Why should she care at all about the world?

  3. rea says:

    YOu know, you could also clean the University’s bathrooms through a system of “calibrated peer review.” Human cleaners, plucked from the ranks of the course registrants, could inspect and scrub campus bathrooms and rate them according to a rubric developed by the professor. A critical mass of deputized students should be able to clean a bathroom at least as well as a pretty good teaching assistant.

  4. LeeEsq says:

    Erik, yeah but this is probably more out of meaness than anything else.

  5. Linnaeus says:

    I think I’ve written about Koller before though I can’t find any references to it, but I love the idea of a university professor spending her career dedicated to helping universities not hire academic labor. It must be very rewarding. And hey, get rid of all the TAs and we can hire another administrator for 125,000 a year!

    And continue raising tuition rates even after shifting more work to the students!

    Another example of the McDonaldization of society.

  6. elm says:

    There’s a way in which this isn’t a bad idea: part of the problem in the humanities job markets is an oversupply of PhD students because departments bring in large grad classes because they serve as cheap labor for large undergrad classes like English Comp and language courses and whatnot. Something that changes this dynamic could address that long-term structural problems in these job markets. Not that I’m saying that that’s Koller’s goal, mind you.

    • elm says:

      I should add, there are clear implementation issues with the idea as DrDick and others point out above, but I don’t think these problems are necessarily insurmountable. (It would involve spot checking, at the least, of undergraduate evaluation to make sure they know what they’re doing and grading fairly.)

  7. Ion says:

    When I was a TA, I spent a large amount of time helping my students learn the material. Is turning out graduates who are competent in their fields passe now?

    Besides, nothing proves you know the material like being able to teach it. TAing is a valuable experience for the TAs, too. (Assuming the university isn’t trying to get an adjunct professor’s amount of work out of them, which happily wasn’t the case at my school.)

  8. dave3544 says:

    I wonder how this works with FERPA; students are not supposed to know the grades of other students.

    This is also why the graduate employee union at the UO has been so strident about not having undergraduates grade papers, lead labs, or teach classes on our campus. A TA position is a tuition wavier, health insurance, and a fee waiver in addition to the salary and we don’t want those jobs replaced by undergraduates working for minimum wages or for free.

    • Bill Murray says:

      FERPA is just evil government regulation that needs to go away so that corporations can work more effciently

      • (the other) Davis says:

        Oh, but schools love FERPA when they can hide behind it and refuse to release data that might make them look bad. No matter if the data in question is effectively anonymized and aggregated.

  9. homunq says:

    This could actually be a good idea if done right. The fact that the people who like the idea most have no motive to do it right, while the people with a motive and ability to do it right mostly hate it, means that it will be done badly at least 5 times before anyone gets it halfway right.

  10. wengler says:

    The only way this would work is with supervision of every single peer-reviewed paper. Thus saving no work whatsoever.

    I’ve had many a lazy teacher do this in any number of grade levels though. I think this was in some ways made illegal by the Supreme Court however.

  11. somethingblue says:

    And hey, get rid of all the TAs and we can hire another administrator for 125,000 a year!

    Well, maybe part of one. I suppose you might be able to get a Deputy Associate Special Assistant to the Vice Provost for that. If you threw in hockey tickets and a reserved parking spot.

  12. JL says:

    Maybe there’s some background about Koller that I’m not familiar with here, but how much of this might be field myopia? I’m in CS. Grad students in CS tend to be supported through RAships more often than through TAships. And even among TAs, in some programs grading is considered undesirable low-level gruntwork – I took a number of lower-level classes as an undergrad where the TAs taught recitations and held office hours but grading was done (for pay) by senior-level undergrads who had aced the course when they took it.

    Obviously that’s not how it works in the humanities, which is why I’m bringing up the possibility of field myopia.

    Also, echoing what elm said, I’ve always been told that in the humanities they admit way more grad students than the labor market can support so that they can exploit them for cheap TA labor, leaving many of them unable to find jobs in their field later. Is that not actually true?

    • Hogan says:

      The article talks specifically about MOOCs (massively open online courses), which can have enrollments in the low five figures (and don’t count toward degrees). Why you would assign papers in such a class I have no idea, and maybe peer review would be no worse than the scant attention the papers would get from a professor with a normal-sized cadre of graduate assistants.

    • Linnaeus says:

      At my university, some of the lower-level CS courses are TA’d by undergraduates. They do all of the duties of a TA.

      I’ve always been told that in the humanities they admit way more grad students than the labor market can support so that they can exploit them for cheap TA labor, leaving many of them unable to find jobs in their field later. Is that not actually true?

      Well, of course they don’t tell you this when they admit you, but it’s hard not to draw the conclusion that the real value of graduate students to the institution is the labor they provide and that Ph.D.s are essentially a waste product. Although universities that have been pursuing a “growth strategy” in response to declining state funding have found that strategy hard to continue since the recent economic downturn and now we’re hearing a lot more about “doing more with less”.

  13. Heron says:

    Koller’s Comp Sci, and notice how she singles out Liberal Arts. If the opinions of engineering and science faculty I’ve encountered are any indication, she probably doesn’t consider Lib Arts majors to be “serious” subjects.

    • Hogan says:

      Well, she doesn’t consider the student products in those courses to be machine-gradeable. Which is probably true.

    • Bijan Parsia says:

      This is false, right? She teaches an online computer science course.

      If she didn’t think they were serious, she wouldn’t try to support them in the massive online context.

    • kStudent says:

      Echoing the comments of Bijan and others, this seems like a lot of (slightly mean) guesswork.

      In fact, much of the computer science grading at Stanford is already done with substantial amounts of machine help. As far as I can tell (and I’m working there in a very closely related field), this seems to be appreciated by the CS grad students, who can spend more time on their research. Given that this stuff was developed for this reason in Koller’s home field, and that it works, it seems uncharitable to assume other motivations.

      • mds says:

        Given that this stuff was developed for this reason in Koller’s home field, and that it works, it seems uncharitable to assume other motivations.

        From the linked article:

        assessing student work in humanities-oriented MOOCs

        A critical mass of deputized students should be able to evaluate an essay “at least as [well] as a pretty good [teaching assistant],” Koller said in an interview.

        [Emphasis added]

        Yes, I’m sure such a move would be welcomed by humanities graduate students, who could then spend more time on their research once their shift at Wal-Mart is over.

        Meanwhile,

        The terms stipulate that Coursera may use “non-personal” information it collects from users “for business purposes.” They also indicate that Coursera may share personal information with its “business partners” so that registered students might “receive communications from such parties that [students] have opted in to.”

        Let’s not be uncharitable here. Who’s to say a for-profit model that treats students as a product to be sold is a disturbing precedent? We wouldn’t want to be mean, just because someone is handing yet another weapon to those seeking to further commodify higher education. It’s with the best of intentions.

        • Furious Jorge says:

          Butbutbut – she’s brilliant! A leader in her field! How could her work possibly have any negative repercussions? It’s so mean to assume that they would!

          • Bijan Parsia says:

            No one said that there couldn’t be negative repercussions. I’d, in fact, welcome such a discussion. Even without her work, distance learning is deeply threatening to the university model. Videoing lectures potentially threatens jobs.

            The “critiques” in this post and thread don’t engage any of that. They consist primarily of:

            1) Attacks on Koller’s motivations. Yes, she founded a for-profit spin out and (thus) hopes to make money. She also does other things partly motivated by money (e.g., teach at Stanford). Unless someone pulls up something concrete about her hating labor, or grad students, or whatever, then, frankly, given the available public evidence, this is anti-warranted.

            2) Attacks on Koller’s personality. Again, the public evidence suggests that she is a normally adjusted, rather pleasant person who cares about teaching, students, and expanded access. Absent some concrete evidence to the contrary, the sliming is anti-warranted.

            3) Attacks on the Very Idea of “calibrated peer review”. There’s definitely reason to wonder about approaches to this very knotty problem. But suggestions that Koller hasn’t considered the obvious issues are, frankly, ridiculous.

            So, there’s little semblance of useful critique or discussion. This is not appreciated by me, for whatever that’s worth.

            • Linnaeus says:

              1) Attacks on Koller’s motivations. Yes, she founded a for-profit spin out and (thus) hopes to make money. She also does other things partly motivated by money (e.g., teach at Stanford). Unless someone pulls up something concrete about her hating labor, or grad students, or whatever, then, frankly, given the available public evidence, this is anti-warranted.

              For me, it’s less about Koller’s personal motivations here and more about the structural problems that may (and I stress “may”) happen. It’s entirely fair to have a few questions concerning the possible conflicts between profit motive and educational mission.

              2) Attacks on Koller’s personality. Again, the public evidence suggests that she is a normally adjusted, rather pleasant person who cares about teaching, students, and expanded access. Absent some concrete evidence to the contrary, the sliming is anti-warranted.

              I don’t know much about her, so there’s not much I can say to this other than, yes, we ought to think twice about personalizing critiques of her ideas.

              3) Attacks on the Very Idea of “calibrated peer review”. There’s definitely reason to wonder about approaches to this very knotty problem. But suggestions that Koller hasn’t considered the obvious issues are, frankly, ridiculous.

              Admittedly, I don’t know a great deal about “calibrated peer review”, so I’m willing to temper any skepticism I have about it and be open to its possible advantages. On the other hand, it’s incumbent upon people like Koller who are arguing for the idea to demonstrate that it will do what they suggest it will do.

              • Bijan Parsia says:

                For me, it’s less about Koller’s personal motivations here and more about the structural problems that may (and I stress “may”) happen. It’s entirely fair to have a few questions concerning the possible conflicts between profit motive and educational mission.

                I trust it’s evident that I’ve not said anything otherwise. However, it’s hard to see Erik’s original post, even, as addressing that aspect. The spin out and for profit aspect problems were not there. (Just to pick a prime example.)

                The fact that distance learning is dominated by for-profits (as I pointed out below) is really worrying.

                Admittedly, I don’t know a great deal about “calibrated peer review”, so I’m willing to temper any skepticism I have about it and be open to its possible advantages. On the other hand, it’s incumbent upon people like Koller who are arguing for the idea to demonstrate that it will do what they suggest it will do.

                Of course. I’ve tried to suggest ways in which it could be reasonable. But without more concrete details it’s impossible to assess.

                On the flip side, the content of the critiques of the possibilities in this space seem rather thin (see many comments I responded to).

                On the flip flip side, I finally googled for calibrated peer review and it seems to be an existing technique, not something Koller is originating. The first few paged on Google scholar are mostly for science based writing. I’ll see what I can dig up on humanities.

                • Bijan Parsia says:

                  This paper is interesting.

                  Average instructor-given scores (± standard deviation) were significantly lower than CPR-given scores
                  for the second (7.9 ± 1.6 vs. 8.9 ± 1.1; t
                  19 = -4.25, p < .001) and third essays
                  (8.1 ± 1.8 vs. 8.6 ± 1.6; t 19 = -2.19, p =
                  0.04), but were not significantly different for the final essay assignment (8.8 ± 0.9 vs. 9.0 ± 0.8; t19 = -1.62, p = 0.122). However, the instructor assigned scores were very similar to the peer-review scores (i.e., scores excluding performance on calibration essays, peer review, and self-assessment), with differences between
                  scores for the second (7.9 ± 1.6 vs. 8.4 ± 1.4; t 19 = -1.82, p = 0.085), third
                  (8.1 ± 1.8 vs. 8.1 ± 2.1; t19 = -0.002, p = 0.998), and fourth essays (8.8 ± 0.9 vs. 8.5 ± 1.1; t 19 = 1.54, p = 0.14) all being insignificant (Figure 2). Most assignment grades therefore reflected a generally high level of performance on calibration essays, peer review, and self-assessment. In fact, only 2 essays of the 60 analyzed received negative consequences in scoring due to these factors.

                  My read is that, basically, there wasn’t a substantive difference in the assessments (i.e., the direct scores of the assignments). So peer review performed as well as expert review, but the full “calibrated peer review” was easier on students. (But only by a small amount, i.e., less than a point.) However, there seemed to be no extra benefit to the peer review. (No way I’m fighting the copy and paste again. The article’s short!) Moreover, incorporation of the writing component didn’t result in improved outcomes. (Might be orthogonal to calibrated peer review. They note in related work that other studies have seen improvements.)

                  It’s a very readable paper. So one good thing about this mess is that I have something new to explore!

        • Bijan Parsia says:

          Yes, I’m sure such a move would be welcomed by humanities graduate students, who could then spend more time on their research once their shift at Wal-Mart is over.

          I was a humanities grad student. I would have appreciated some assistance with the grading. Obviously, if this defunds me, that would suck. But if the people exploring radically massive online learning succes, universities are going to have problems across the board.

          But we have problems now. The tuition spiral and adjunctificaiton/studentization of teaching are big, big problems.

          Let’s not be uncharitable here. Who’s to say a for-profit model that treats students as a product to be sold is a disturbing precedent?

          This is a more substantive point. I wasn’t too happy about reading that. OTOH, compared to ending up in horrible debt, it’s unclear that it’s so bad. Indeed, exactly what dignity there is in going massively in debt over having my one’s info sold by Yet Another Online Thing? Or even paying $1000 for a course? Or $500?

          We wouldn’t want to be mean, just because someone is handing yet another weapon to those seeking to further commodify higher education. It’s with the best of intentions.

          Still silly. I don’t see why Koller’s expressed desire to expand access is to be entirely neglected in analyzing the situation. It’s not her only motivation, nor does motivation alone mitigate harmful outcomes.

          It’s not just mean, though it is (inappropriately) mean, it’s analytically poor.

  14. Corey says:

    LOL @ “academic labor”. Your self-martyrdom tendencies are embarrassing, Loomis.

    • Erik Loomis says:

      I’m sure the plethora of academics on this blog will totally agree with you that their work is not labor…..

      • Walt says:

        Eric, just delete comments like this as soon as you see them. Corey is just wasting your time.

      • Professor of Conservative Fantasies says:

        Actually it is all fun and games. I can not believe the state pays me such a princely sum especially as I the bulk of my day indoctrinating undergraduates in obamanism. Occasionally, I conduct some research on all they ways conservatism has failed or why America sucks, but I foist most of that work on the multiple grad assistants of both genders that I am involved with. I will miss those special office hours, when I retire to a tropical paradise at 50, with an unbelievably lucrative pension and a Young Republican, I have recruited into the lifestyle.

    • Linnaeus says:

      “Academic labor” means work done in an academic context, like teaching or research, typically at an institution of higher education. It’s a pretty uncontroversial term; there’s no self-martyrdom involved.

      • DrDick says:

        It also entails really long hours (60+ a week generally) and can be quite demanding. That we generally love much of what we do is beside the point. I know mechanics who love their jobs.

        • Linnaeus says:

          Yes, I also find “psychic wage” arguments to be rather unconvincing. As if the finance folks on Wall Street are making so much because they hate their jobs.

          • Bijan Parsia says:

            Indeed. I love my job, but I’d love it even more if you paid me bankers wages. “I get to teach and do research and then go home to my SECOND YACHT! WOOHOO!!!!”

            I believe law school reveals that this is a widely shared preference :)

    • Furious Jorge says:

      The word “labor” does have a pretty standard meaning at this point. From an economist’s point of view, pretty much everyone who works performs “labor” – ever hear them talk about the “labor force?” Yeah, we’re part of that.

      No, we’re not working in coal mines or anything (and I don’t see anyone claiming we are), but we are working, and we aren’t exactly management either.

  15. I’ve had petty admins (not my department chairs, they know better, but social science and literary types given “associate” and “assistant” versions of bigger titles) tell me this for years: “You can assign all the writing you want without adding to your workload if you let them read and evaluate each other’s work”….

    I never felt my discipline so undervalued by my own academic colleagues….

    • Bijan Parsia says:

      That is the crap aspect of work on peer evaluation. My experience is that it often takes more work to do right and there is certainly an instructor learning curve. It can “save time” in the following sense in almost every case: Student work can get a lot more individualized attention (since they are getting it from each other). (Even there, standard participation problems make it complex.)

      I’d love tools that made it easier for me to use peer evaluation effectively.

      I’m teaching an academic writing class for our first year PhD students. Last year, we met as groups once a week to read two of the papers and everyone else was to provide critique. (Groups were up to 10 people.) It just didn’t work at all.

      This year, we switched to smaller groups and had them do reviews in advance using a conference management system. Our first inclass discussion was today and it was awesome. They were super active and me and my co-instructor were able to go around the room and provide targeted and generally minimal intervention — instead of doing all the evaluation we could interject bits to turn things the right way.

      Now, we’ll see how the outcomes are, and whether it was this cohort, but it feels promising.

  16. Worth noting, too, that Koller is actually one of the co-founders of Coursera (should that be CourseRA?) so it’s not like she’s a disinterested party here. One might even say there a conflict of interest present.

    In related news, this whole scheme reminds me of that demotivational poster that read “None of us are as dumb as all of us.”

  17. Dave says:

    Better idea: administrators assign grades directly based on estimation of students’ servility in a classroom setting. Not servile? For a nominal fee, you can just get whatever grade you want. Saves labor, teachers; can be outsourced to private contractors.

  18. Bijan Parsia says:

    I think I’ve written about Koller before though I can’t find any references to it, but I love the idea of a university professor spending her career dedicated to helping universities not hire academic labor

    Erik, this is really shoddy and Koller is undeserving of your scorn.

    A quick glance at her publications would show that she’s not “spending her career” against labor. She is a giant in the area of machine learning (see a popularizing article about her work). In general, one of the leading computer scientists of our generation.

    And there’s reasonable evidence that she’s a dedicated educator (she’s one of the Stanford crew pushing free online courses as well as offering one).

    Peer instruction is nothing new and, done right, has a lot of pedagogic benefits. Indeed, getting students to think critically about their own and others’ work is one of the key skills…for anything!

    Without question, Koller is aware with the obvious challenges of using peer evaluation either formatively or summatively, so the quick comments about “oh semi-literate undergrads” and “every peer reviewed paper would have to be supervised” are unlikely to hit home. My guess, from the brief quoted comment, is that she intends to 1) control the problem by providing structured evaluation metics with training and 2) use various crowd-sourcing mechanisms to provide a better total evaluation than any particular individual evaluation is likely to be. (An obvious first idea is to look to interrater correlations in a variety of ways, perhaps with simple similarity to model answers and some history and context thrown in.)

    So, this is likely to be most successful in fairly controlled circumstances wherein the peer reviewing is a useful part of the education process.

    Now, there is an important issue of how this affects the academic workforce (or even simply other universities…if Stanford, MIT, etc. master scaled university learning, even at the remedial through first two year level, other universities (and perhaps community colleges) could be hit really hard). But, afaict, this is focusing on the drudge work (e.g., grading 200 essays).

    I don’t believe university administrators will do the right thing, but they aren’t doing the right thing now.

    (As an aside, I have a PhD student working on automatically generating Multiple Choice questions for both formative and summative assessment. I believe it would eliminate a lot of drudgery, improve assessment (including self-assessment), and allow people to focus on other aspects of teaching. Again, there are a lot of restrictions, but imagine we who work in this area achieved full success: From articles, slides, textbooks, etc. you could reliable generate appropriated targeted MCQ exams. This almost certainly would mean that less labor would be needed to generate all sorts of exam.

    So, what exactly should follow?)

    (Let me add that computer science is notoriously crappy to women. So I find this, afaict, unmerited slamming of Koller to be particularly unsettling. If there’s some other thing driving this animus, i.e., she’s bad on grad student unions or something, then put it out there. As it stands, this is wrong.)

    • JL says:

      Yeah, I am usually a big fan of Eric’s posts (including most of the ones that get massive outrage from the commentariat here) but I have to agree with most of this.

      I could see the implementation of this going wrong, but what is wrong with trying to get some of the gruntwork out of teaching? Wouldn’t you rather have your TAs preparing for lectures or discussions, answering student questions about the material, working one-on-one with students during office hours, writing exam questions, that sort of thing?

      Also, as a woman grad student in CS, I had the “unsettled” reaction that Bijan describes in his last paragraph.

      • Linnaeus says:

        but what is wrong with trying to get some of the gruntwork out of teaching? Wouldn’t you rather have your TAs preparing for lectures or discussions, answering student questions about the material, working one-on-one with students during office hours, writing exam questions, that sort of thing?

        I’ll admit that grading papers is among my least favorite of teaching tasks that I did as a TA and as an instructor and that the prospect of reducing/shifting that task is appealing in some sense and may even have some pedagogic value.

        I think the concern is that spending more time doing the other tasks you mention won’t actually happen and things will actually go in the opposite direction: TAs will be further deskilled or even eliminated.

        Case in point: several years ago, my department (history, btw) had to deal with the problem of (then) rising enrollments, but with fewer resources to employ TAs to help teach the larger classes. What it did was create large history courses with out discussion sections, which are typically within the domain of a TA’s work. So a TA in that course was responsible for more students (up to 150, instead of 50-75), but didn’t hold discussion sections. We referred to these TAs as “super-graders” because that’s effectively what they were. There were a lot of concerns voiced at the time that this was a path to deskilling and reducing TA labor. So far, the number of non-sectioned courses has held steady, but there’s always the chance that they will increase in number now that they’ve been established.

        So, with that as an example, coupled with other trends in the direction of the casualization of academic labor – more adjuncts, outsourcing TAs, etc. – and the potential for profit, I can see pressure not to give TAs “higher-level” tasks, but simply to push them out altogether. This may be less of an issue in wealthier science and engineering departments, who can fund their students to do research, but in departments whose students rely much more on teaching for support, the impact could be much greater.

        • Bijan Parsia says:

          So, with that as an example, coupled with other trends in the direction of the casualization of academic labor – more adjuncts, outsourcing TAs, etc. – and the potential for profit, I can see pressure not to give TAs “higher-level” tasks, but simply to push them out altogether.

          Yes. This is a big problem. On the other hand, exploiting them ever more with overloaded amounts of work isn’t really a great alternative. Giving them exceeding crappy jobs while students and leaving them with no sane prospects afterwards is nuts as well.

          I certainly don’t welcome a collapse, but I’m unclear that this marginal bit is supersignificant.

          So, for example, I’m pretty skeptical that the technique can be reliably applied in credit granting situation for quite some time. At least, in a convincing enough way such that degree granting institutions are going to adopt it. Thus, it will almost certaintly need a fair number of Expert Distinterested Graders (aka TAs) as facilitators and probably sample graders. (I.e., the TAs would provide feedback on some fraction of the assignment to provide some grist for comparison and perhaps spot check student evaluations).

          Of course, the structural winds make good outcomes unlikely.

          • Linnaeus says:

            On the other hand, exploiting them ever more with overloaded amounts of work isn’t really a great alternative. Giving them exceeding crappy jobs while students and leaving them with no sane prospects afterwards is nuts as well.

            No, it’s not a great alternative. That suggests to me that one thing we could do is try to come up with some other alternatives. Easier said than done, I know.

            Or, if we as a society are committed to the current path, let’s at least be more honest about its effects.

            • Bijan Parsia says:

              Yes and yes.

              I don’t have any brilliant ideas, really. Just dealing with our administration is frustrating even before we get to the disaster the UK government is making for higher ed.

              What’s particularly frustrating is that the certification function (which is a powerful weapon) is almost certainly going to continue to be misused and fail hard as for-profits rise higher.

        • JL says:

          One thing I’ve wondered about that seems at least somewhat relevant here…is the relative lack of RA-based support in the humanities because they just don’t have the money (i.e. the professors’ grants are just not as big/frequent in those fields)? Or is it because humanities profs feel that the nature of their work doesn’t benefit from research assistants? Or some combination of the two? Or something else?

          I know many STEM grad students but very few in the humanities, and I went to a STEM-based university for undergrad, so this is something I’ve had limited opportunity to figure out. My little sister’s about to be a humanities PhD student, but since she isn’t yet I doubt she has much insight.

          • Bijan Parsia says:

            It absolutely is the lack of money. When I was in philosophy grad school I was an RA one semester (and knew a few others) and we were regarded with delight by our profs. Mine had me go through many issues of the philospher’s index and get photocopies of all papers on certain topics.

            (It, overall, was a good experience and that wasn’t all I did and I was able to get my own papers concurrently and it was a break from grading. But it was a good deal of drugery.)

            Of course, with digitized journals, you don’t need a student for a semester. Technological obsolescence.

            Other people did proofreading, index collation etc.

            100% funding. (Some teaching is “needed” because teaching experience out of grad school has become standard. But I really suspect that that is a result, not a driver.)

          • Linnaeus says:

            To riff off of Bijan Parsia’s comment, it’s mostly money. I think humanities scholars aren’t against research assistants at all, but humanist scholarship tends to be solitary in nature, so the need for research assistants isn’t as great. Humanists may coauthor a book or coedit a volume of collected essays, but that’s about as big as most projects in the humanities get. And as Bijan points out, some of the tasks that humanities RAs do are made less necessary though digitization.

            Humanist scholarship, then, doesn’t require a lot of expensive infrastructure to do, and that’s one reason why the grants are 1) less commonly available and 2) smaller. In my field, the biggest reason you need money is for travel to and living expenses in the places where your primary sources are.

            I suspect another reason why research grants in the humanities are not as common is because those fields are seen as not yielding tangible benefits. They don’t “do” anything, the argument goes, and so why expend resources on them that you don’t need to?

      • Bijan Parsia says:

        Hey JL,

        It is disappointing, esp. as some people have doubled downed. Maybe I’ll pester Erik on twitter about it.

        Feel free to ping me if you’re working on anything I’d be interested in (and I’m interested in most everything :))

        Cheers!

  19. Bijan Parsia says:

    Interesting article about the destruction of higher ed:

    The response to the falling rate of profit also played a role in changing education. Tax reduction had the attraction of partially restoring profits, but it also had an important effect on education. Growing budget deficits would ramp up pressure to privatize what had been previously public responsibilities. By largely defunding education, universities became increasingly dependent on corporate money. Administrators became cautious about allowing expression of ideas that might seem upsetting to business. These factors took an enormous toll on higher education.

    Tuition began a rapid ascent. Student debt accumulated. University funds were concentrated on programs that cater to business needs, such as biotechnology and engineering, and, naturally, business schools.

    • Linnaeus says:

      This is, arguably, what’s happening at UW, although I wouldn’t say the institution is being “destroyed”. Rather, it’s being transformed. How much of a good thing that is can be debated.

      • Bijan Parsia says:

        Hmm. How is it not destructive? I mean, what’s the aspect of the transformation that’s non-destructive?

        • Linnaeus says:

          Well, yes, the transformation is “destructive” in the sense that older models of funding and providing education are being replaced. I’m just trying to be careful not to be hyperbolic about the changes in higher education that are happening and appear to be accelerating.

          I mean, UW’s still around, of course. Its basic institutional structure remains. But how it does what it does – and the purposes to which its mission is directed – are being transformed, and I worry about some of the consequences of that.

  20. justaguy says:

    This plan is dumb for a variety of reasons.

    Rubrics can be very helpful in getting students to evaluate each other’s work – but on their own aren’t very useful. That is, I’ve only gotten undergrads to take grading by rubrics seriously if I’ve told them that the quality of their feedback will be part of their final grade. And even then, they’re not good for giving the specificity of feedback that you would get from a grad student TA.

    Also, grading isn’t just about the feedback students get – it is also valuable feedback for professors. It lets you know what students are having trouble understanding.

    While there are students that understand the course material and academic writing well enough to give feedback to their classmates, how would you know who they are if you don’t interact with them?

    And this isn’t just about the humanities. Sure, a math problem is really easy to machine grade. But if you want to learn to do math, knowing THAT you got something wrong is fairly useless – you need to know WHY you got it wrong.

    But, if they’re going to be giving these courses away for free, its hard to see how they could do so with TAs. They’re talking about tens of thousands of students. So, if you figure that you have one ta for 100 students (a large workload, but lets keep it with round numbers) that’s 100 TAs for a 10,000 student course. Pay them 2K a month for a 4 month semester, and that leaves you paying $800k in TA salaries (plus fee remissions).

    Which is all to say that, I don’t think that IT makes education scalable to the degree that these people seem to imagine it is. I wonder what experience these people have of actually teaching classes.

    • justaguy says:

      Oh, and the fact that nobody wants to give students anything resembling a diploma for their coursework is telling.

    • Bijan Parsia says:

      This plan is dumb for a variety of reasons.

      Really not obvious at all. It might not work out, of course.

      Also, grading isn’t just about the feedback students get – it is also valuable feedback for professors. It lets you know what students are having trouble understanding.

      Done right, you can recapture some of that. Examining work that has been presorted and markedup according to a rubric is a good way to spot problems (esp. the prevalence of problems).

      While there are students that understand the course material and academic writing well enough to give feedback to their classmates, how would you know who they are if you don’t interact with them?

      Again, I would imagine that there is quite a bit of interaction (e.g., training), but also that they intend to exploit the “wisdom of the crowd” (oh, how I hate that phrase). However, you can in a variety of situations use analysis of a set of responses to get more reliable results than a single response (cf prediction markets).

      Obviously, it’s not magic pixie dust. Socrates’ point in the Crito is still important. The challenge here is to structure things so that you capture at significant part of the “Graded by expert” value.

      (Note that a lot of the effort of expert graders (TAs, etc.) is wasted (as students don’t look at it) or of extremely variable quality (both between graders and within a grader…fatigue and burn out take a toll). So the question isn’t whether the peak performance of traditional grading will outshine the peak performance of calibrated peer review, but what the median, average, and low performance is.)

      And this isn’t just about the humanities. Sure, a math problem is really easy to machine grade.

      Depends on the level of math. Proofs are rather hard.

      I wonder what experience these people have of actually teaching classes.

      Koller teaches continually, is currently teaching a massive online course, and has a lot of other teaching innovation under her belt. Her PhD students do rather well and seem to like her a lot.

  21. BradP says:

    I can’t really comment on whether this is a good way to grade tests and essays, but it seems like an attempt to raise teacher productivity.

    Increases in teacher productivity should be considered a good thing. The already existing problems that direct the revenue to already bloated administrations are the problems here.

    • Slocum says:

      “Productivity?” But production of what? Students who take a senior seminar with extensive and personalized face-to-face feedback are getting a very different, and arguably superior, “product” than what comes out of CourseRA. To exaggerate, its like saying that professors are more productive if we can find a way to get them to make Beanie Babies while they lecture to large classes on microeconomics.

      • BradP says:

        To exaggerate, its like saying that professors are more productive if we can find a way to get them to make Beanie Babies while they lecture to large classes on microeconomics.

        Like I said, I don’t know if it is a good idea in particular, but freeing teachers up from the more menial aspects of their job should allow them more time to focus on their more specialized talents.

  22. mtraven says:

    Welcome to the future. Academic labor surely is labor, and just as surely, it is going to get hollowed-out by technology just like other fields. It happened to manufacturing jobs (spawning the original Luddites and continuing on today), it has done a job on newspapers, and it is poised to happen to a variety of white-collar jobs, from insurance claims processors to lawyers and teachers.

    Welcome to capitalism. This is a great example of it at work: it’s producing added wealth (more people can learn from Koller and others thanks to their free online courses); at the cost of changing from a hand-crafted personal product to a mass-manufactured one of lower but acceptable quality; and everybody in the same field who isn’t Koller or a comparable superstar gets the shaft. Let’s say Koller is the best person at the world at teaching Bayesian Networks</a?; once her product is out there for free, the value-add of the 2nd through 999th best teacher of the same subject starts to fall.

    Academic institutions, like unions, provide some insulation from these market pressures thanks to the lingering pre-capitalist values that keep them going, but not a lot.

    BTW, my point is not so much that the Luddites were wrong; it's that they failed.

    Also, re the personal judgement of Koller: do you think she would be a better person if she put more value on protecting her guild brethren (academics) rather than trying to provide wealth to the world at large? I don't think I phrased that as well as I'd like because I actually don't know the answer. I tend to think she's doing good, but from the union point of view she's kind of like a scab, a tool of the bosses. Which side are you on?

    • John Protevi says:

      Welcome to the future.

      The essence of closing off critical thought is naturalizing / inevitabilizing a social process.

      It happened to manufacturing jobs (spawning the original Luddites and continuing on today), it has done a job on newspapers, and it is poised to happen to a variety of white-collar jobs, from insurance claims processors to lawyers

      Without the slightest bit of Mr Chipsism, educating students =/= producing chairs (table, rugs, cars, TVs …), newspapers, insurance claims, and law citations. That’s because students are human beings, and not lumps of raw material. In other words, you need to work on your ontology, which is a fancy way of saying you need to learn how to tell the difference between people and things (this is as far away from a precious little humanism as you can get; you can and should make this distinction purely in terms of complexity and unpredictability of the material systems involved).

      Academic institutions, like unions, provide some insulation from these market pressures thanks to the lingering pre-capitalist values that keep them going, but not a lot.

      You say that as if it’s a bad thing. Moreover, you need to think through the inevitabilist march of progress narrative you’ve constructed, because otherwise you are begging the question (and yes, I’m using that correctly; you’re assuming that anti-capitalism is an atavism ["lingering pre-capitalist values"] standing in the way of some linear progress, but that’s what needs to be shown, not assumed). In simpler terms, there has always been conflict between solidarity and exploitation; some social systems do a better job at supporting one side or the other, but there’s no inevitable process that has resulted in having our system support exploitation; to say otherwise is fatalism or Whiggery.

      • John Protevi says:

        All that by way of demonstrating the type of analysis an experienced professor can bring to a student paper that a “critical mass of deputized students” is probably not going to be able to match.

        • John Protevi says:

          Which is not to say I’m as snotty to my students as I was to mtravers! Psychology matters, personal trust matters, encouragement and hell just plain caring about students matters. None of that can be replicated by a “critical mass of deputized students.”

          I always recommend Hubert Dreyfus, On the Internet, for a start on this issue.

          • mtraven says:

            I’m not as snotty about Dreyfus as some other people in the computer community, but the book you cited starts out with him admitting that the predictions and complaints in the original 2001 edition were completely wrong:

            The most radical change is in Chapter One. There I endorsed the current pessimism concerning the possibility of succesfully searching billions of meaningless hypertext websites. Now, that pessimism has turned to optimism thanks to Google…likewise, most of Chapter Two predicting the failure of idsembodied distance learning and ridiculing the enthusiasts…had to be scrapped. It is now clear that distance learning has failed. The major universities have given up on it…

            One might plausibly conclude that listening to Herbert Dreyfus is not a very good way to grapple with potential future developments and deployments of technology.

            And since this will probably also be misunderstood — the world desperately needs good critics of technology, some counterweight to the mindless enthusiasm and enormous energies of capitalism. But (judging solely from this page) Dreyfus is not doing a very good job of providing it.

            • John Protevi says:

              The name is Hubert, not Herbert. But if that’s payback for calling you mtravers instead of mtraven, that’s cool.

              You should learn more about him. Dreyfus is famous for distinguishing meaningless machine calculation from meaningful interpretation. His attack on GOFAI (good old-fashioned artificial intelligence) is rightly credited with inspiring the embodied cognition movement.

              The reason he says he Google succeeds (in the part you excise) is that they are “gaining followers from old-fashioned meaning-based ordering of information.” So he was right that meaningless calculation would never work for web searches, since that’s not what Google does.

              Similarly, on his second point, he is right that universities have scrapped distance learning (for degrees); the current project is non-degree-based.

              • Bijan Parsia says:

                Dreyfus is interesting on some things, but he has a tendency to make rather dramatically wrong predictions about technology. I’m pretty skeptical about the influence there (pointers?).

                It’s been quite some time since I looked at Dreyfus (though I enjoyed some of his existentialism lectures on iTunes University), but my impression is that there was more accidental convergence with modern views (plus, some rather silly foils in early AI), rather than him articulating the basis of current work.

                Again, if you have some particular bit you think would shake this impression, I’d welcome it.

              • Bijan Parsia says:

                Sorry I missed the key points to critique:

                The reason he says he Google succeeds (in the part you excise) is that they are “gaining followers from old-fashioned meaning-based ordering of information.” So he was right that meaningless calculation would never work for web searches, since that’s not what Google does.

                I have no idea what this is supposed to mean or what you think Google does. Google’s algorithms are certainly not primarily “meaning based ordering of information”, but statistical algorithms based on such things as number of links (to vastly over simplify).

                (“Gaining followers”? You mean users?)

                Similarly, on his second point, he is right that universities have scrapped distance learning (for degrees); the current project is non-degree-based.

                This is totally ridiculou. The Open University is going strong. My university is looking to expand it’s (credit bearing) distance learning. Etc. The caveat “for degrees” is pretty telling: Why would Dreyfus make the distinction? I mean, it you can get a degree from it then you have a non-learning based reason to do it! If it’s crap for learning and qua experience then we should see fewer free ones!

                There are, of course, huge issues with our online lives, but I would think that the tremendous time we all spend in virtual interaction would provide at least some push back on a Dreyfusian perspective.

                • John Protevi says:

                  YIKES! My bad. In my haste to one-up mtraven by quoting what he excises, I myself excised the important part of what Dreyfus says, which is that Wikipedia uses meaning-based process, not Google! (You could say that Google uses meaning at a remove, since it ranks links and hits processed for the most parts by meaning-based human interpretation — notwithstanding the bots and so on sometimes used to game the system.) Apologies and chagrin!

                  As for Dreyfus’s legacy w/r/t embodied cognition, it’s a little complicated. His critique of GOFAI comes from a Heideggerian perspective, which is not that of embodied cognition, since for Heidegger the action is in the “clearing” or background opened by social practices and “cognition” even as an organism’s navigating its environment (as in a Varelean approach) is one-step away from the “clearing.”

                  Quick references: Mike Wheeler and Julian Kiverstein have edited a collection on Heidegger and Cognitive Science with a Foreword by Dreyfus.

                  Wheeler’s 2005 MIT book is neo-Dreyfusian, if you will.

                  More links following.

                • John Protevi says:

                  Two recent books from the non-Dreyfus enactive school:

                  An essay collection from MIT.

                  Evan Thompson’s 2007 Harvard book, Mind in Life.

                • Bijan Parsia says:

                  YIKES! My bad. In my haste to one-up mtraven by quoting what he excises, I myself excised the important part of what Dreyfus says, which is that Wikipedia uses meaning-based process, not Google!

                  Wikipedia is a pretty good example of something akin to calibrated peer review and disembodied activity (i.e., it is mostly virtual).

                  (You could say that Google uses meaning at a remove, since it ranks links and hits processed for the most parts by meaning-based human interpretation — notwithstanding the bots and so on sometimes used to game the system.)

                  I don’t think so. This would stretch the distinction beyond all comprehension. A rule based natural language processing system would then be meaning based because it is sensitive to the texts.

                  Google doesn’t “interpret” the links and certainly doesn’t read them the way people intend them.

                  Re: the history…I think the history is more meaningly informed by the rise of statistical methods, which, prima facie, don’t seem very Dreyfusesque to me. They exactly don’t make use of “meaning”. (They don’t use “uninterpreted symbols”, of course, but I’m not sure why that’s a win for him. I started looking at a paper of his on phenomenology and neural networks but, frankly, it seemed quite poorly informed about machine learning and I couldn’t immediately tell how old it was to see if it was the time or his lack of knowledge.)

                • Bijan Parsia says:

                  John, don’t forget the wrongness about distance learning! See e.g., this article. (Has an excellent point that distance learning tends to require more instructor effort but is not compensated for in standard faculty load models.)

                  The literature is replete with evidence of the growing demand for distance education. The annual market for distance learning is currently $4.5 billion, and it is “expected to grow to $11 billion by 2005” (Kariya, 2003, p. 49; see also Pond, 2003; “Lifelong,” 2002). As Oblinger and Kidwell (2000) have noted, the International Data Corporation (IDC) expects a 33% growth rate in distance education over the next several years. Some analysts predict that demand for distributed education will grow from “five percent of all higher education institutions in 1998 to 15 percent by 2002” (p. 32; see also West, 1999). Others have asserted that up one-half of traditional campus programs will soon be available (alternatively or exclusively) online (Finkelstein et al.2000; Bishop, 2003; Dunn 2000; Winsboro, 2002).

                  And from here:

                  Enrolments in fully online courses are around the 11% mark of all enrolments, and this is expected to increase to 20% by 2014 (about 4 million enrolments). Still more than half (55%) of all US degree-granting institutions offer no fully online courses (down from 69% in 2005). However, Eduventures estimates that of the adult market (25+), 24% are currently in online programs and this is expected to increase to 35-40% by 2014.

                  Worryingly:

                  For instance the for-profit sector has a much higher proportion of the total online market (around 32%) compared with its share of the overall higher education market (about 7%), and seven of the top ten institutions in terms of the number of online enrolments are for-profits.

                  In my school this is viewed as a problem, i.e., that we aren’t competing effectively in the distance learning market.

                • John Protevi says:

                  Bijan @3:28: bad phrasing on my part. When I say

                  Google uses meaning at a remove, since it ranks links and hits processed for the most parts by meaning-based human interpretation

                  I meant that human interpretation is what makes the links and hits: for instance, Erik read my post and linked to it. Google then picks up the link, which improves the ranking of New APPS. So it’s a two part process: the humans make the links and Google ranks the sites based on the links. So Google is one step away from meaningful interpretation.

              • mtraven says:

                I know all about Dreyfus’s critique of AI, and like I said, I probably am a lot less contemptuous of that critique than most computer people.

                Actually it is in at least one way of a piece with this newer work that you cite. His AI critiques was titled “What Computers Can’t Do”, but it really was “what computers can’t do yet”, or, to give him more credit, “what computers can’t do if we have extremely naive views of the relation between computers and thought”.

        • Bijan Parsia says:

          All that by way of demonstrating the type of analysis an experienced professor can bring to a student paper that a “critical mass of deputized students” is probably not going to be able to match.

          You say that as if it were a bad thing.

          Putting aside tonal concerns, I think you were misdirecting your response with the whole “students are human beings, not tables”. If you had attended to earlier parts of the thread you would have already seen a bit of floundering about academic work as labor. Many aspects of academic work, including instruction, are reasonably labor. And, as mtraven pointed out, many aspects are vulnerable to technologies shifts (just as it is increasing vulnerable to classic management abuse as self-administration and faculty governance gives way to corporatesque structures and actual corporate control).

          Indeed, mtraven’s analogy seemed both apt and not overdone.

          • John Protevi says:

            Yes, this exchange has been more antagonistic than it could have been and I’ll take responsibility for that.

            • Bijan Parsia says:

              If only your feedback had been calibrated!!!

              (But, interestingly, it does demonstrate that individually supplied feedback is subject to the variance of that individual. Which is a real and perennial problem in assessment.)

              • John Protevi says:

                But that’s part of the discussion. I can’t calibrate feedback to an online interlocutor the way I can to a student in one of my (very reasonably sized, I have to say) classes.

                • Bijan Parsia says:

                  That may be true (I’m a bit skeptical), but it’s not particularly true of a lot of classes.

                  I was being snarky, but if we look at your proposed exemplar of high quality feedback it 1) involves a misreading, 2) involves poor tone, and 3) is a bit repetitive.

                  2, whatever. You were grumpy. 1 doesn’t seem to be a matter of online/offline, high contact/low contact, etc. per se. It seems perfectly possible to have done better in your reading without having any more contact.

                  I feel confident that I, at least, could have made a similarly crappy piece of feedback in either the attenuated or the full contact situation.

                  (And do you really mean “calibrate feedback to” rather than “tailor feedback to”? Calibration would be ensuring your feedback met some standard. So, for example, since we agree that the comment, qua student feedback, was a bit hostile, I don’t see that there’s anything specific to your interlocutor that makes it so. The failed calibration is to a general standard, not “to a student”.)

                • John Protevi says:

                  Not to be too defensive here, but the original comment at 19 April, 10:57 am, did not have any misreadings, and contains what I would maintain would be hard to capture in a rubric given to UG peer graders, that is, detecting and critiquing the techno-inevitabilism tropes of “welcome to the future” and “welcome to capitalism,” etc. The misreading occurred in the subsequent discussion of Dreyfus, which became something of a red herring in distracting us from the original critique of mtraven.

                  As for “calibrated” vs “tailor,” sure, in a technical sense they are distinct, but I was using them in a colloquial sense as equivalent, that is, I didn’t calibrate my reaction to the individual’s requirements. That seems colloquially okay to my ears.

                  In any case, my original post to which Erik linked was about the HE labor issues of the Coursera initiative in the context of shifts in HE political economy, about which I’ve written quite a bit at New APPS. I’m in agreement with mtraven and you that we are seeing the routinization of humanities teaching. I just think this is based on bad psychology and bad political economy and deny it is techno-inevitabilist.

                  While “de-skilling” and “proletarianization of academic labor” are exaggerations here, they are just barely exaggerations in the case of teaching English Comp. The trouble here is in good old-fashioned political economy and work conditions: there’s no inherent drudgery in grading English Comp (or French lit, American History, Intro to Philosophy, etc.) essays, it’s just exhausting to grade too many of them. And that has to do with class size and course load, not with the type of work. So what calibrated peer review does is treat the bad working conditions of English Comp (etc) teachers as a given and try to alleviate the “drudgery.” But again, grading essays is not drudgery; grading too many of them is. So what we see here is a techno-fix to a politico-economic-labor issue.

                  In any case the mention of Dreyfus was a nod in the direction of the embodiment critique of distance learning. Part of that critique is the collective nature of getting a reasonably small number of people together in the same space vs isolating a huge number of them in their separate rooms somewhere. To really discuss that we’d have to get into all sorts of issues in the affective sciences, some of which I tackle in this class.

                • Bijan Parsia says:

                  Not to be too defensive here,

                  I certainly don’t mind you defending. If you start feeling defensive or otherwise uncomfotable, I’ll happily let it drop.

                  but the original comment at 19 April, 10:57 am, did not have any misreadings,

                  Hmm. I’ll try to be more careful. I might have been looking at the wrong message.

                  Let’s see:

                  Welcome to the future.

                  The essence of closing off critical thought is naturalizing / inevitabilizing a social process.

                  I would argue that this is a misreading (or, let us say, a less charitable reading) in that stating a standard course of events does not, itself, mean that you are precluding alternatives. The “welcome to capitalism” itself at least suggests that there are possible alternatives (inevitable within capitalism is not the same as inevitable across capitalism). Indeed, I read it as “This is a standar/common mode” not “THIS IS INEVITABLE SUCK IT UP LOZER”.

                  I would also quarrel with the equating of “naturalizing” and “inevitabilizing” (and the word “inevitabilizing”) and whether this is the essence of closing off critical thought. Clearly you can close off critical though in other ways, and not every naturalization destroys critique. (My favorite example is Firestone: Sex differences being natural exactly doesn’t exempt them from critique.

                  and contains what I would maintain would be hard to capture in a rubric given to UG peer graders, that is, detecting and critiquing the techno-inevitabilism tropes of “welcome to the future” and “welcome to capitalism,” etc.

                  This is only true if the detection is correct. Which seems at least debatable.

                  Without the slightest bit of Mr Chipsism, educating students =/= producing chairs (table, rugs, cars, TVs …), newspapers, insurance claims, and law citations.

                  This seems to be a straight up misreading. mtraven isn’t equating educating students with manufacturing from raw goods, per se. He’s at worst analogizing them and more reasonable saying that the labor of education is primarily a kind of labor and thus heir to all the sins that labor is heir too. Which is exactly what Erik said earlier in the thread. And is uncontroversial, I think. Esp. wrt to the vulnerability to technological advance.

                  And your instantiating of examples was weird. Lawyers, arguably are the nears to teachers in the examples mentioned and the key product of lawyers is legal services not law citations. And getting a will done by your family lawyer is probably a better experience in a lot of ways than filling out a web form. But the latter is hugely cheaper and for a wide range of people (esp. those who never had or had to struggle to have such services before) a big step up. But as the ability to automate lawyerly services improves, it leaves lawyers with less to do or splits lawyers into luxuries and drudges.

                  This happens with teachers as well. Standardization of the form of labor as you try to scale it out leads to imposition of drudgery which leads people to try to automate that drudgery. As technology gets better, you can get radical shifts or slow erosions. Consider surgeons. It’s quite plausible that in 20 years they’ll be largely replace with robots (or related automation), at least for routine operations (say, wart removal). And this might make health care cheaper and perhaps better or at least not hugely crappier. Which would increase overall wealth at the expense of the surgeons. I don’t see how there’s any confusion of people with raw materials here. I’m not even sure what that was supposed to show. There might be negative consequences to reducing human interaction in lots of cases, but the question is whether it’s worth it. I actually love self checkout lines, for example, but I lose the interaction with the check out person. But arguably, that’s not hugely different than not being in a small town with a stable population where I’ve known and been friends with the checker all my life.

                  Academic institutions, like unions, provide some insulation from these market pressures thanks to the lingering pre-capitalist values that keep them going, but not a lot.

                  You say that as if it’s a bad thing.

                  This is clearly wrong on your part as is shown by the immediately following sentence:

                  BTW, my point is not so much that the Luddites were wrong; it’s that they failed.

                  I’ll stop here for the moment because I’d rather respond to the substantive part of your comment separately.

                  The misreading occurred in the subsequent discussion of Dreyfus, which became something of a red herring in distracting us from the original critique of mtraven.

                • Bijan Parsia says:

                  In any case, my original post to which Erik linked was about the HE labor issues of the Coursera initiative in the context of shifts in HE political economy, about which I’ve written quite a bit at New APPS. I’m in agreement with mtraven and you that we are seeing the routinization of humanities teaching. I just think this is based on bad psychology and bad political economy and deny it is techno-inevitabilist.

                  I don’t think either of us are quite arguing that. I’m certainly not.

                  I think it is “inevitable” that we’ll be able to build better programs and have much better understanding of how people learn and this will almost certainly set up strong affordances for radical change in universities. This is esp true to the fact that big power centres and forces are working hard to destroy universities (see my earlier comments on this). It’s easy to imagine both the relieving of drudgery and good employment for PhDs. What’s harder to see is how to get there.

                  If you have a post wherein you make the psychology/political economy (and pedagogic? surely that’s central) case, I’ll welcome it. Right now, I’m starting to explore bringing calibrated peer review to my dept. (I already incorporate a fair bit of writing in my courses but CPR seems like it could save me some effort AND improve the outcomes.)

                • John Protevi says:

                  If you have a post wherein you make the psychology/political economy (and pedagogic? surely that’s central) case, I’ll welcome it.

                  I try to make that point here in the next graf of that comment, where I deny that grading compositions is inherently drudgery:

                  While “de-skilling” and “proletarianization of academic labor” are exaggerations here, they are just barely exaggerations in the case of teaching English Comp. The trouble here is in good old-fashioned political economy and work conditions: there’s no inherent drudgery in grading English Comp (or French lit, American History, Intro to Philosophy, etc.) essays, it’s just exhausting to grade too many of them. And that has to do with class size and course load, not with the type of work. So what calibrated peer review does is treat the bad working conditions of English Comp (etc) teachers as a given and try to alleviate the “drudgery.” But again, grading essays is not drudgery; grading too many of them is. So what we see here is a techno-fix to a politico-economic-labor issue.

                  As for the pedagogy, that’s in the graf after that, on collective bodily co-presence in relation to the evolution and biology of affective cognition.

                  The other stuff I think is safely w/in the realm of debate, i.e., whether “welcome to the future” is techno-inevitabilist (of course that’s an ugly neologism!).

                  Anyway, for some needed humor, I thought the best part of my New APPS post was the line about badges, but no one has picked up on that yet!

      • John Protevi says:

        Last point for now. One way the inevitablist trope works is by naturalizing / physicalizing political economy, as in “insulation from market pressures.” Mirowski’s More Heat Than Light is a good start here. See also, Natural Images in Economic Thought: “Markets Read in Tooth and Claw”

      • mtraven says:

        The essence of closing off critical thought is naturalizing / inevitabilizing a social process.

        I think you are misreading my comment if that’s the main point you got out of it. I thought my slightly favorible mention of Luddism and disfavorable mention of scabbing would have gotten the point across, but maybe not.

        Anyway: I don’t know what’s inevitable, but I try to predict the future as much as anyone. And tech progress looks pretty inevitable, barring some kind of social collapse. The consequences to laboring people due to tech progress are not inevitable.

        Without the slightest bit of Mr Chipsism, educating students =/= producing chairs (table, rugs, cars, TVs â¦), newspapers, insurance claims, and law citations. Thatâs because students are human beings, and not lumps of raw material.

        It’s true — and as I said, the kind of mass-scale, partly automated teaching provided by Coursera and others is probably not as good as teaching in a small-class, personal seminar style. But it’s a lot cheaper.

        You say that as if itâs a bad thing.

        Uh, no.

        In simpler terms, there has always been conflict between solidarity and exploitation; some social systems do a better job at supporting one side or the other…

        We are in agreement there.

        • Dana says:

          Even though mtraven’s walked back most of what ze said, I still feel the need to pile on. I think a big problem with our public dialogue, and perhaps in society generally, is an obsession with predicting the future because the desire to be correct turns the prediction into self-fulfilling prophecy. I’d like to see a lot less predicting of what will happen, and a lot more predicting of what can happen, or what should happen, or predicting what the best possible future could be.

          We’ve lived in a capitalist world for quite some time now, but we’ve never before lived in a world that was so market-oriented. Learning is a long-term, costly, difficult-to-assess process that is valuable to the survival and prosperity of capitalist societies. It is giving way to credentialization because new technologies make possible substantial market rewards for those cheap, easily identifiable social expedients. These technological innovations (I’ll not call them progress) could be used to enhance learning, but first we have to recognize that learning should not be measured in dollars earned or saved by the “educators” but in skills acquired by the students.

          • mtraven says:

            Even though mtraven’s walked back most of what ze said

            Did not.

            Either I’m writing unclearly or there’s some reading comprehension problems going on here (likely both).

              • Dana says:

                I’ll have to disagree, since my reaction to mtraven’s comments was precisely the same as John Protevi’s. In the future you may want to temper blanket declarations about the hollowing out of academic labor (or anything else).

                And this misunderstanding, if you can call it that, demonstrates the limits of our “technological progress.” Perhaps mtraven’s perspective would have been more clear if we’d had the benefit of hearing hir tone of voice, seeing facial expressions, and body language, or having the opportunity to ask for immediate clarification. Until technology can duplicate the discomfort caused by my interrogating an undergraduate who hasn’t done the reading, we’re pretty far away from technology superseding current pedagogical methodologies.

                • Bijan Parsia says:

                  I’ll have to disagree, since my reaction to mtraven’s comments was precisely the same as John Protevi’s.

                  It’s perfectly possible for several people to make the same misreading without it being, per se, an problem of the writing. This is especially true when the readings aren’t independent.

                  (OIf course, it’s reasonable to say, “Misreadings are inherently a problem with the writing…write not just to be understood, but not to be misunderstood.” But that’s more of a regulative ideal. For example, I don’t see any ambiguity in what they wrote that’s the source of themisreading.)

                  In the future you may want to temper blanket declarations about the hollowing out of academic labor (or anything else).

                  Did you hit the wrong reply button? I never made any such declaration.

                  And the original comment:

                  Welcome to the future. Academic labor surely is labor, and just as surely, it is going to get hollowed-out by technology just like other fields. It happened to manufacturing jobs (spawning the original Luddites and continuing on today), it has done a job on newspapers, and it is poised to happen to a variety of white-collar jobs, from insurance claims processors to lawyers and teachers.

                  is rather banal. Thus far neither you nor John have engaged with this observation, afaict.

                  Indeed, what’s to temper? Either he’s wrong or right about the vulnerability of instruction to technological advance. Prima facie, he’s obviously right. Your response didn’t seem to have anything to do with this. Indeed, while mtraven obvious was making a prediction, there’s no evidence that they were “obsessed” with predicting the future or were thus trying to fulfil their predictions. Sorry, but that clearly wasn’t a problem in the writing, but a problem in your analysis.

                  And this misunderstanding, if you can call it that, demonstrates the limits of our “technological progress.”

                  I can and it doesn’t.

                  Perhaps mtraven’s perspective would have been more clear if we’d had the benefit of hearing hir tone of voice, seeing facial expressions, and body language, or having the opportunity to ask for immediate clarification.

                  This is silly. When one grades an essay, you don’t have the tone of voice or facial expressions or the opportunity to ask for “immediate” clarification. The technology of writing precludes all that. But it doesn’t matter if the writing is offline or online.

                  You could have easily asked for clarification instead of launching an offtopic broadside. Similarly, John could have done a more careful reading before commenting. Of course, this wasn’t a real grading situation, so polemical rather than pedagogic motivations are more salient. But we shouldn’t misattribute the source of the errors.

                  Until technology can duplicate the discomfort caused by my interrogating an undergraduate who hasn’t done the reading, we’re pretty far away from technology superseding current pedagogical methodologies.

                  I see no evidence that one couldn’t make someone uncomfortable by online comments. I’m also rather skeptical about the pedagogic benefits of discomfort.

                  Really, this is a total non sequitur. Assessment usually isn’t face to face. Big lectures don’t usually involve a lot of individualize interaction. etc. These are obviously the cases most vulnerable to technological advance. But these things don’t need technology to cause labor issues.

        • Bijan Parsia says:

          It’s true — and as I said, the kind of mass-scale, partly automated teaching provided by Coursera and others is probably not as good as teaching in a small-class, personal seminar style. But it’s a lot cheaper.

          Even this isn’t clear, alas. As I wrote above, I’m perfectly prepared to believe that the best small-class, personal seminar style teaching (including having a good fit between the instructor and students) would be much better than the best of mass-scale, partly automated teaching. But it’s not clear that these aren’t reversed at the median. And expensive crappy teaching is much worse than cheap crappy teaching. One of my worst, absolute worst, classes ever was a small seminar. Horrific. At the same university I had an amazing 60 person lecture class.

          (I have the ridiculous title of “eLearning Champion” in my dept (it’s a university thing). One of my bonneted bees is finding techniques that raise the floor for comparatively little effort rather than things that work superdupergreat when done by the best, most dedicated teachers. This is one reason I’m often skeptical about enthusiasm for non-lecture modes of teaching. When they work, they can work rather well (usually trading depth for breath). But they can fail hard for various classes of students or in the hands of a lot of teachers. They tend to be the classes I’d have wanted to take and gotten a lot of our, but I wasn’t a typical student.)

  23. Matt says:

    Sounds reasonable to me – we already use basically the same system to score every hand-written response on standardized tests. Same methodology, except it requires “a college degree”. Just “a”, mind you – not anything related in the *slightest* to the material being scored.

    Seriously – there are kids out there right now having their futures decided on high-stakes science testing scored by 50ish homemakers with a 30-year-old BA in basketweaving and 4 hours of computer-based training…

  24. Bijan Parsia says:

    So CPR is not an automated grading technique, nor is it a “future” technology, nor is it Koller’s development.

    OTOH, there is plenty of work on automated grading techniques, and apparently, they do rather well:

    Eight of the nine automated essay scoring engines that were evaluated in the demonstration represented commercial entities and captured over 97% of the current automated scoring market in the United States….By and large, the scoring engines did a good of replicating the mean scores for all of the data sets. Figure 1 illustrates these in graphic form. Table 5 shows the deviation scores (deltas) from the means of each data set. Accuracy was most likely influenced by the size of the scale. For example, all vendor engines generated predicted means within 0.10 of the human mean for Data Set #3 which had a rubric range of 0-3. However, even when the range was much larger as in Data Set #8 (range of 0-60), mean estimates were generally within 1 point, and usually smaller.

    (The writing forms were quite varied.)

    The study seems quite well done and read the paper for all the appropriate caveats. (Two quick ones: The engines were trained on a *large* number of essays which helps them. The total corpus was in the tens of thousands. I have no idea how the engines would perform on more idiosyncratic essays. However, some of the sub-data sets had to be transcribed which potentially hurt the automated engines more than the human graders.)

    Yes yes, data in a dying thread…my speciality.

Leave a Reply

You must be logged in to post a comment.