I know I’m not the first person to blog about course evaluations (ProfHacker has at least three posts alone), and recently I’ve seen a few articles discussing campuses considering moving to online course evaluations and potential pitfalls, particularly in the area of response rates. (See for instance, Wired Campus writing on the topic.) This semester I had tremendous success in getting my students to complete the online Georgia Tech Course/Instructor Opinion Survey (CIOS), and this post shares some of my thoughts on the matter.

First, let’s talk history. Georgia Tech has been using some version of CIOS since 1986. The survey was moved online in 1999 and shortened to its present 10-question Likert scale format in 2002. Around 2005 or 2006 students were given more options for free-form comments with the addition of comment boxes after each question. (The standard overall free-response area remained.) Thanks to some research compiled by Tris Utschig of our Center for the Enhancement of Teaching and Learning, we know that our response rate since moving online is consistent with the response rate to paper surveys and in the middle of the pack nationally. The Institute response rate since moving online has ranged from a low of 34.52% to 46.55%, with most terms right around 40%. (Data through Spring 2007.) Some faculty have complained about a decline in response rates, and in some sense that’s true. What’s happened is a redistribution of student responses. The liberal arts faculty who routinely had students complete the paper CIOS have seen their response rates drop. However, the engineering and science faculty who teach large classes often failed to administer the paper CIOS, and thus their response rates are now up. My personal history with CIOS response rates prior to this term is shown in the table below.

TermCourseEnrollmentResponse Rate

(Two response rates are given for Fall 2008 as my course was part of a pilot of new CIOS questions. The first rate is for standard CIOS and the second is for the pilot. We urged students to complete both surveys to analyze correlations between responses on the two surveys.)

Essentially, I’ve managed to beat the Institute average every term, but I’ve rarely been as high as I’d like. (Nearing 80% as I did in Fall 2008 was pretty good, but that was my smallest class, too.) This year, the Student Government Association partnered with the Office of the Provost to try to get students to complete CIOS. In addition to the standard reminder emails, the Provost sent an email to all students calling on them to complete CIOS. Additionally, every instructor on campus received a report every third day during the three-week survey period giving the response rates for their courses. (It’s an absurdly high number of clicks to check your own response rate, so for many faculty members, this probably helped.) This email also included the following tips (my commentary in brackets):

• Talk to the students in your class about CIOS. [Been doing that forever.]
• Tell your students that you want them all to complete the survey. [And that.]
• Tell your students that you will read the results and take them into consideration as you prepare to teach this and other classes in the future. [Yup. This term I emphasized that even though this was my last term at Tech, I would use the feedback in future teaching.]
• If possible, tell your students about ways that you have changed the way you teach due to past results. [Don't have specific examples, but they have seen me change in response to midterm surveys.]
• Tell the class how the survey results are used — by you to improve your teaching, by your chair/dean to measure teaching effectiveness when reviewing your performance, by their peers when choosing what sections to register for. [Harder here as a grad student.]
• Have your students bring their laptops (or web-enabled phones) to class some time during the last two weeks of class and give them 10-15 minutes to complete the CIOS survey. The survey is quick enough that they can even share laptops if they do not each have one and still complete the survey for your class in the time allotted. [Tried this term for the first time.]
• Have a “competition” with an instructor of another section of the same course or a similar course and tell the students that you want your course to have a higher response rate than your colleague gets. [Really? Does this work?]
• Go to http://www.cetl.gatech.edu/cios/index.htm (and encourage your students to go there) for two short PowerPoint presentations (one for faculty and one for students) about CIOS. [Yeah, students need another PowerPoint at the end of the term.]

From the list of suggestions, there was one that I had wanted to try. (I’d heard about it several times before from CETL staff, so it wasn’t the email that got it on my radar screen.) I decided I would give up 15 minutes of class time on Wednesday of the last week of class for students to take CIOS on their laptops or smartphones. I informed them of this several days in advance and proceeded to leave the room 15 minutes early on the designated day. Before class, 11 of my 59 students had completed CIOS. After class, that was up to a whopping 20 students (33.9%). I’m betting most of those nine students would have completed the survey anyway, so I’m not sure that giving them class time had any impact. (The course in question is the same junior-level applied combinatorics course I taught in Fall 2008.) After seeing this experiment fail, I decided I needed to try something else.

I decided that I would try to provide my students with some incentive for completing CIOS. Some instructors will give across-the-board extra credit if the response rate breaks a certain threshold. (The survey is anonymous, so you can’t give it only to those who complete it.) This seems intellectually dishonest in a math course, since uniform extra credit doesn’t really change anything. One colleague here in the School of Mathematics did something I might consider in future large service courses. She gave them the option of dropping their lowest test if a certain percentage of students took the survey. This is a pretty nice option, especially if it’s something you were considering doing anyway. The way my course is structured, however, dropping a test would not be a good idea. Instead, I settled on the simple act of telling them their course grades early. CIOS is open until midnight on the Sunday after final examinations, and I’ve always been wary about releasing course grades via our learning management system when CIOS is open. I don’t want my evaluations to be impacted positively or negatively by students knowing their grades when they take the survey. However, I figured that if 85% of my class of 59 completed the survey, any impact of the other students knowing their grades when completing CIOS would be tiny.

To support my little experiment, I decided that I should make sure that the students knew where the response rate stood on a regular basis. This way, those who really wanted their grades on Thursday (instead of Tuesday) could help exert peer pressure. I also learned quickly that I needed to include a direct link to the survey site in every announcement I posted to the course management system. The few I posted without a link triggered negligible increases in responses. However, those with links generally created noticeable upswings. For fun, let’s track the response rate over time:

DateResponse rate

Yes, that’s right, of my 59 students, 93.2% or 55 of them completed CIOS this term. (I’m guessing that the auditing student and the one who quit coming but didn’t drop account for half of those who didn’t complete it.) I took an NPR pledge drive approach to the updates I gave them (that’s where I pulled the percentages above from), ensuring that they knew how many more responses we needed to reach the goal and pleading with them to take the survey. I really need to check with my friends in CETL to see if I set any sort of record for completion percentage in courses with enrollments of 50 or more.

What’s the take-away here? Well, first of all, I want to know if the overall Institute response rate went up this term. I know it didn’t go as high as my rate did, but my success may be partially attributable to the increased efforts of SGA and the Provost. I won’t write off giving students time in class for online evaluations yet, but another flop in that area will cause me to re-evaluate if I want to give up 15 minutes of class time for the survey. I really do think that the combination of continual reminders and giving the students an incentive of some sort to meet a class response rate goal was helpful this time around. In the future, I’ll either use the incentive of releasing grades early or allowing a drop test (or drop homework or quiz or something like that) to help drum up student responses. Now I’ll just have to wait until next week to see what my students had to say about my teaching this term.

Anyone have success stories about response rates for online course evaluations they’d like to share?

[Creative Commons licensed photo by Flickr user mformarcus]