Sunday, April 3, 2011

When are the exam?

Of course it's impossible to design an exam that encourages exactly the right type of learning. One big problem is that we want people to understand things over a wide interval of time, but when we tell them they will be measured in time t, they can get away with just understanding in time t. But even if we can only give 1 exam, we can make progress by randomizing when it actually occurs, effectively spreading it over the interval.

[Note: This post is not about the unexpected hanging paradox, although you may want to read about it!]

The problem is that cramming for an exam does not seem to promote retention. Knowledge gained through cramming follows a sharp peak that drops off steeply on both sides. If that's bad, then why don't we see more exams that are given at some random time? Instead of "the exam will be given on day t," why not "the exam will be given randomly on one of these 3 dates"? If retention really is better accomplished by not cramming, then in fact spreading the possibility of testing over a few weeks may actually make it worth learning the material more deeply, rather than just cramming repeatedly.

There are surely logistical issues to consider. But if you have a problem with random exams in theory, why? In considering this, keep in mind that the twin goals of testing are to incentivize good learning and to measure that learning. Here are the important questions: Are we motivating better learning? Are we moving toward a less biased estimator of learning? Are we moving toward a less noisy estimator of learning? (If it comes at the cost of significantly higher variance, it may not be worth it).

Why are random ("pop") quizzes rare rather than standard, and why do they so frequently only cover yesterday's material?

7 comments:

  1. Interesting. My guesses:

    People, including students, hate to lose a feeling of control, and teachers don’t want to be too disliked, in part because of the course evals but also because it’s just plain human to want to be liked.

    It could also be an issue of priorities or an issue of image. Maybe they just aren’t as interested in incentivizing the best kind of learning for their students as they are in other things. Maybe they’d rather be seen as innovative researchers than innovative teachers.

    ReplyDelete
  2. I'd like the minimum MSE estimator of learning. :) You make a good point on bias versus noise.

    Upon reading your post, I had a thought. The cramming issue may also infect long homework assignments (not just quizzes or exams). For this reason, a long two-week assignment might be more burdensome than two one-week assignments (even if these shorter assignments merely partition the longer one).

    ReplyDelete
  3. Exogeneous CombustionApril 5, 2011 at 12:09 PM

    I have a minor explanation, part of which syncs up with the first part of Mr. Wehr's comment.

    I can't think of a teacher who is properly incentivized outside of (perhaps) a Business School context or Charter School context. Public Schools and most Universities don't have a clear connection between reward and student learning. Insofar as they do (in Business Schools, for instance), they're based off student recommendations.

    A world in which one has to prepare for endless spontaneous evaluated learning experiences is stressful: I imagine poor reviews are given to such teachers. Insofar as these teachers care about their reviews, they will shy away from these pop quizzes.

    Further, Insofar as these teachers don't care about teaching effectively, they won't give grading-intensive tasks like pop quizzes.

    In most of the worlds and incentive systems we observe, then, pop quizzes seem like they'd be avoided, even if they teach more.

    The question might then turn to: "but why haven't we set up proper incentives?" My answer is trade-union monopoly and the horrors of political economy.

    ReplyDelete
  4. Guys,

    Good reasons why we might not be surprised to see the relative lack of randomized exams. I don't find this shortage of randomization particularly surprising, either. But fwiw, here's a counterpoint.

    We should be quite mindful of what actually motivates teacher evals and student opinions of teachers more generally. Students really, really like teachers who give them good grades. And that itself is not changed here: whatever the format of the exam, the grades will still be relative and can be scaled however the teacher desires.

    Furthermore, if randomizing motivated students to learn more, there's a considerable subset of students who *appreciate* being pushed to learn, especially after the fact, i.e. at the time of writing their teacher evaluations. Many of us appreciate having gone through a class that made us learn a lot.

    Before you roll your eyes too hard, that was true in my (admittedly particular) high school environment, and it was true for me in many higher-level classes in college. So we would have to say why there is no attempt at randomized testing in *those* environments.

    ReplyDelete
  5. Tony,

    Yes, partitioning homework would doubtless have an effect on their learning.

    When to do one's HW? When to study for an exam? These issues are actually not the same because you can do your homework at any point and it will get the same score, whereas your score on the exam is a function of how recently you studied the material.

    If partitioning the homework causes a significant change to people's behavior (causing them to, say, not do it at the last minute), it's probably being driven by dynamic inconsistency, procrastination. On the other hand, dynamic inconsistency is actually not needed to drive the optimality of cramming. (you might say cramming simply minimizes the amount of studying needed to obtain a given score). Yet, whatever the underlying reason, the effect on *learning* is the same: get people to space out their learning, and they will probably learn more, hour for hour. That's true for homeworks and exams both, so it's worth considering when setting the deadlines.

    One final note: Giving your students more time to do their homework *relaxes* the constraint on how they budget their time, but it actually *tightens* the constraint on your grader! Each week they are allowed to hold onto their homework is a week it can't be graded! I would say this is a negligible effect, except that I am currently the grader.

    ReplyDelete
  6. Good point on the timing of learning and exams. Your comment made me think of homework (again) and whether you *really* need dynamic inconsistency to get people to backload the homework load.

    It's possible that students think they will be more productive starting their homework late because (by then) they will have covered all of the necessary material in lecture (and the instructor will have given all of the useful hints they will get).

    We usually assume that students backload time spent on homework because of procrastination, but -- depending on the helpfulness of the hints -- it may be optimal to delay working on homework.

    ReplyDelete
  7. Tony,

    True! So...don't give helpful hints? :)

    ReplyDelete