This week I've been reviewing the solutions of assignments submitted by student of our program transformation course. One of the things that strikes me again and again is how hard it is for students to get a grasp of the problem that they have to solve in an assignment.
In the last few installments of our program transformation course, the students had to develop a program instrumentation that traces the number of calls for every callee/caller pair for one of the assignments (we usually have about 10 assignments). Last year, we just described the problem in the assignment. The solutions of the students were ok, but not really exciting. They forgot to handle all kinds of cases, and some solutions even didn't terminate for some input programs.
This year, I included a set of tests in the assignment, which illustrate most (but not all) of the problems in this program transformation. Surprise, surprise: the students suddenly were able to handle all the issues illustrated by the testsuite that I provided. However, obviously I did not give the students all tests (evil grin). Indeed, several solutions could not handle the tests that I did not provide.
Most students don't write test. Worse, if they do test, then they create a single file and modify the test to check a new situation that they might have discovered. In this way they don't build up a nice testsuite. The important part of testing is that they can be repeated automatically, not that you run a test once! I'm trying very hard to convince students to test their code properly, but they don't seem to understand the need for it, so you typically get questions like "Is 10 tests enough?". I'm afraid that this is not a problem specific to students.
This might not be very surprising to you, but I am surprised how clear the results of this small 'experiment' are (I did not do this experiment on purpose). I wonder what the implications of this should be for education. Clearly, having a grasp of a problem is the most important part of the solution.