First, let’s talk history. Georgia Tech has been using some version of CIOS since 1986. The survey was moved online in 1999 and shortened to its present 10-question Likert scale format in 2002. Around 2005 or 2006 students were given more options for free-form comments with the addition of comment boxes after each question. (The standard overall free-response area remained.) Thanks to some research compiled by Tris Utschig of our Center for the Enhancement of Teaching and Learning, we know that our response rate since moving online is consistent with the response rate to paper surveys and in the middle of the pack nationally. The Institute response rate since moving online has ranged from a low of 34.52% to 46.55%, with most terms right around 40%. (Data through Spring 2007.) Some faculty have complained about a decline in response rates, and in some sense that’s true. What’s happened is a redistribution of student responses. The liberal arts faculty who routinely had students complete the paper CIOS have seen their response rates drop. However, the engineering and science faculty who teach large classes often failed to administer the paper CIOS, and thus their response rates are now up. My personal history with CIOS response rates prior to this term is shown in the table below.

(Two response rates are given for Fall 2008 as my course was part of a pilot of new CIOS questions. The first rate is for standard CIOS and the second is for the pilot. We urged students to complete both surveys to analyze correlations between responses on the two surveys.)

Essentially, I’ve managed to beat the Institute average every term, but I’ve rarely been as high as I’d like. (Nearing 80% as I did in Fall 2008 was pretty good, but that was my smallest class, too.) This year, the Student Government Association partnered with the Office of the Provost to try to get students to complete CIOS. In addition to the standard reminder emails, the Provost sent an email to all students calling on them to complete CIOS. Additionally, every instructor on campus received a report every third day during the three-week survey period giving the response rates for their courses. (It’s an absurdly high number of clicks to check your own response rate, so for many faculty members, this probably helped.) This email also included the following tips (my commentary in brackets):

• Talk to the students in your class about CIOS. [Been doing that forever.]
• Tell your students that you want them all to complete the survey. [And that.]
• Tell your students that you will read the results and take them into consideration as you prepare to teach this and other classes in the future. [Yup. This term I emphasized that even though this was my last term at Tech, I would use the feedback in future teaching.]
• If possible, tell your students about ways that you have changed the way you teach due to past results. [Don't have specific examples, but they have seen me change in response to midterm surveys.]
• Tell the class how the survey results are used – by you to improve your teaching, by your chair/dean to measure teaching effectiveness when reviewing your performance, by their peers when choosing what sections to register for. [Harder here as a grad student.]
• Have your students bring their laptops (or web-enabled phones) to class some time during the last two weeks of class and give them 10-15 minutes to complete the CIOS survey. The survey is quick enough that they can even share laptops if they do not each have one and still complete the survey for your class in the time allotted. [Tried this term for the first time.]
• Have a “competition” with an instructor of another section of the same course or a similar course and tell the students that you want your course to have a higher response rate than your colleague gets. [Really? Does this work?]
• Go to http://www.cetl.gatech.edu/cios/index.htm (and encourage your students to go there) for two short PowerPoint presentations (one for faculty and one for students) about CIOS. [Yeah, students need another PowerPoint at the end of the term.]

From the list of suggestions, there was one that I had wanted to try. (I’d heard about it several times before from CETL staff, so it wasn’t the email that got it on my radar screen.) I decided I would give up 15 minutes of class time on Wednesday of the last week of class for students to take CIOS on their laptops or smartphones. I informed them of this several days in advance and proceeded to leave the room 15 minutes early on the designated day. Before class, 11 of my 59 students had completed CIOS. After class, that was up to a whopping 20 students (33.9%). I’m betting most of those nine students would have completed the survey anyway, so I’m not sure that giving them class time had any impact. (The course in question is the same junior-level applied combinatorics course I taught in Fall 2008.) After seeing this experiment fail, I decided I needed to try something else.

I decided that I would try to provide my students with some incentive for completing CIOS. Some instructors will give across-the-board extra credit if the response rate breaks a certain threshold. (The survey is anonymous, so you can’t give it only to those who complete it.) This seems intellectually dishonest in a math course, since uniform extra credit doesn’t really change anything. One colleague here in the School of Mathematics did something I might consider in future large service courses. She gave them the option of dropping their lowest test if a certain percentage of students took the survey. This is a pretty nice option, especially if it’s something you were considering doing anyway. The way my course is structured, however, dropping a test would not be a good idea. Instead, I settled on the simple act of telling them their course grades early. CIOS is open until midnight on the Sunday after final examinations, and I’ve always been wary about releasing course grades via our learning management system when CIOS is open. I don’t want my evaluations to be impacted positively or negatively by students knowing their grades when they take the survey. However, I figured that if 85% of my class of 59 completed the survey, any impact of the other students knowing their grades when completing CIOS would be tiny.

To support my little experiment, I decided that I should make sure that the students knew where the response rate stood on a regular basis. This way, those who really wanted their grades on Thursday (instead of Tuesday) could help exert peer pressure. I also learned quickly that I needed to include a direct link to the survey site in every announcement I posted to the course management system. The few I posted without a link triggered negligible increases in responses. However, those with links generally created noticeable upswings. For fun, let’s track the response rate over time:

Yes, that’s right, of my 59 students, 93.2% or 55 of them completed CIOS this term. (I’m guessing that the auditing student and the one who quit coming but didn’t drop account for half of those who didn’t complete it.) I took an NPR pledge drive approach to the updates I gave them (that’s where I pulled the percentages above from), ensuring that they knew how many more responses we needed to reach the goal and pleading with them to take the survey. I really need to check with my friends in CETL to see if I set any sort of record for completion percentage in courses with enrollments of 50 or more.

What’s the take-away here? Well, first of all, I want to know if the overall Institute response rate went up this term. I know it didn’t go as high as my rate did, but my success may be partially attributable to the increased efforts of SGA and the Provost. I won’t write off giving students time in class for online evaluations yet, but another flop in that area will cause me to re-evaluate if I want to give up 15 minutes of class time for the survey. I really do think that the combination of continual reminders and giving the students an incentive of some sort to meet a class response rate goal was helpful this time around. In the future, I’ll either use the incentive of releasing grades early or allowing a drop test (or drop homework or quiz or something like that) to help drum up student responses. Now I’ll just have to wait until next week to see what my students had to say about my teaching this term.

Anyone have success stories about response rates for online course evaluations they’d like to share?

[Creative Commons licensed photo by Flickr user mformarcus]

Helen E. Grenga attended Shorter College as an undergraduate, earning a bachelor’s degree in chemistry in 1960. From there, she moved to the University of Virginia, earning her doctorate in physical chemistry in 1967. She came to Georgia Tech as a postdoc in chemistry. In 1968, she was appointed Assistant Professor of Metallurgy (now Materials Science & Engineering). This appointment made Dr. Grenga the first woman to hold a tenure-track appointment in Georgia Tech’s College of Engineering. (The first woman faculty member at Tech was Dr. Mary K. Cabell, who joined the School of Mathematics in 1960.) Dr. Grenga’s appointment came 16 years after the Board of Regents repealed its exclusionary policy that prohibited women from enrolling at the state’s technological university and 12 years after Shirley Clements Mewborn was the first woman to earn a degree (EE) from Georgia Tech.

Helen E. Grenga (Photo: GT Communications & Marketing)

Dr. Grenga’s career at Georgia Tech extended over 30 years. During that time, she was served as the faculty advisor to the student chapter of the Society of Women Engineers, which now annually presents an award named after her. Additionally, she served as secretary, vice president, and president of the national SWE. Her most visible lasting legacy, however, has to be in her efforts to create the graduate cooperative program. Georgia Tech’s cooperative education program for undergraduate students was established in 1912, just 24 years after the Institute enrolled its first students. Despite regularly being the nation’s largest voluntary co-op program, it was restricted to undergraduates for over 70 years. Dr. Grenga worked to secure a grant to create the graduate co-op program, which was established in 1983. It is presently the largest co-op program for science and engineering students in the United States.

In 1985, Dr. Grenga was appointed Associate Dean of the Graduate Division. She spent the remainder of her career at Georgia Tech working on behalf of graduate students. When she retired, her title was Associate Vice President and Dean of Graduate Studies and Research. During this time, Dr. Grenga served as advisor to the Graduate Student Government Association. Although I was never privileged to meet Dr. Grenga, it is clear that my predecessors in SGA greatly valued her advice. She twice received the Graduate SGA Administrator of the Year Award. She was the first to repeat as a recipient of this award. Those who were privileged to work with her as an advisor speak fondly of her and her dedication to Georgia Tech’s graduate students.

Upon Dr. Grenga’s retirement, Graduate SGA permanently honored her by annually presenting the Helen E. Grenga Friend of the Graduate Student Award to a member of the community who embodies her dedication to  graduate students on campus. Recent recipients of this award have included a campus minister, the first graduate student ombudsman, the Vice Provost for Research and Dean of Graduate Studies, the Assistant Dean of Graduate Studies (retired), and a mathematics graduate student. Yes, that graduate student is me. I received the honor from my SGA colleagues in 2006 for my efforts on behalf of health care insurance for graduate students. It was somewhat of a bittersweet occasion, as the award was presented just weeks after Dr. Grenga passed away on 14 April 2006. Last week, I had the occasion to discuss Dr. Grenga’s legacy with my colleagues in the Graduate Student Senate as we discussed who should receive the 2010 Helen E. Grenga Friend of the Graduate Student Award. This year’s award recipient will be announced next month, but I am confident we have selected a recipient who is building on Dr. Grenga’s legacy.

Helen E. Grenga came to Georgia Tech as a pioneering woman in engineering. She retired after an outstanding career as a faculty member and administrator who created opportunities for thousands of students (women and men alike) in science, technology, engineering, and mathematics. Best of all, that career left a legacy that lives on today and will continue for many years in the future.

In the first part of the class, we did some group discussion to build up understanding. It seemed to have gone well, so I had a clicker question to make sure they understood the concepts. The question I posed was as follows (with percentage of students giving that response in parentheses):

If a fruit basket generating function contains a factor that is a polynomial, you know

1. a fruit is constrained by a nonzero minimum. (24.44%)
2. a fruit is contained [sic] by a maximum (37.78%).
3. the quantity of a fruit must be a multiple of some integer. (20%)
4. none of the above. (17.78%)

I intended that 2 be the correct answer. (We cleared up that “contained” should have been “constrained” before I shut down voting on the question.) Thus, I was a bit confused why no answer even had 40% of the students behind it. I used a standard peer instruction technique and had them discuss in their groups and then called on a group. The group that I called on had the answer that I wanted. I then decided we should discuss the other answers and why they were incorrect. This is usually valuable, but it was even more valuable than I imagined on Friday. The students really wanted to argue that their answers were correct, and I generally couldn’t argue with them (other than #4).

For answer #3, my question had the deficiency that it didn’t say the integer should be greater than 1. I gave them a hard time because such a response wasn’t in the spirit of the question, but it was technically correct. For #1, the issue is much more subtle. One student asked about “negative fruit”, which I quickly shot down as silly. Another student had an idea that I didn’t fully grasp at first, but that became valid with more discussion. The issue arose because I didn’t specify that the factor in the generating function was the entire factor associated to a fruit. For example, if you need at least two grapefruit, you’ll introduce as a factor in your generating function. That’s clearly not a polynomial, but the overall generating function would have a factor that turned out to be a polynomial, namely . I had to concede that even #1, which I’d tried to dismiss quickly, was a viable choice.

How did I react to the question in class? I’m pleased to say that I handled it pretty well. A natural first reaction to a slip-up in a clicker question is almost always “Drats! I thought I’d done that perfectly.” However, it became a teachable moment. In reality, we were able to discuss far more aspects of generating functions than I intended with the question. I was also able to determine that my students had a quite solid grasp on the topic. I think the fact that they were so eager to discuss the various answer choices shows that we’ve created a comfortable environment in class, too. (I had 45 of 59 students respond via clicker to this question, which is pretty good for the day before Spring Break.) I’d probably use more of these ambiguous clicker questions, but my software (TurningPoint AnyWhere Mac) doesn’t support multiple mark questions. I do the poor man’s version (add choices like 1 and 2, 1 and 3, 2 and 3, etc.) sometimes, but I don’t like to do it too often. Without multiple mark, the better students will often ask “Is there more than one right answer?”, which gives it away earlier than I’d like.

One of the requirements to get a Tier 5 visa is that you prove you’ve got enough money to not be a burden on the British taxpayers. There are two general ways to do this: documentation from your sponsor that they’ll provide funds to maintain you and documentation or documentation that you’ve had at least £800 on deposit with your bank for the past three months. I thought that I’d be using the former criterion. However, my funding is coming from the Marshall Sherfield Fellowship Foundation (administered through the Marshall Aid Commemoration Commission) and not the LSE. This means that the fine folks at LSE cannot embed information about funding in my Certificate of Sponsorship, and the UKBA will only accept maintenance documentation of this form from the visa sponsor. It took most of a day to figure out that no one at the LSE or MACC really knew how this would all work (UKBA just changed the immigration rules in the fall, so this is not surprising). I got referred to the British Consulate in Chicago, through which I will apply for my visa. I contacted the contractor who handles visa applications there, and they basically said “we can’t help you figure out your documentation.” At that point, I decided I’d just prove my own maintenance via bank records.

I’m a big fan of online bank statements, but they are a royal pain when dealing with immigration issues. The UKBA needs original bank statements (thankfully they will accept them denominated in US dollars and not pounds), but I don’t have them. As an alternative, their website says that I can submit a letter from my bank with account details that certifies I’ve had at least £800 in the account for the entirety of the past three months. That sounded easy enough, so I sent Wachovia a secure email through online banking. They referred me to a page about immigration letters, which says that I can get such a letter for $25! I figured that would be tolerable, since I could get the MACC to reimburse me (most likely). I read more carefully, however, and found a red flag. Wachovia’s immigration letter provides:

• Account Number
• Type of Account
• Date Account Opened
• Current Account Balance
• 12 Month Average Balance

Notice that there’s nothing on there about my balance over the past three months, which is a requirement of the UKBA. Another secure email to Wachovia, and they replied to tell me they can’t help. They did give me the phone number to directly call the balance verification office this time, however. This is when the fun began.

After navigating the lengthy set of menus, I got a human being at Wachovia. It sounded like she was ready for the week to be over, and this made it hard to get anything useful done with her. After discussing my situation, she put me on hold. I don’t know why, but it was probably to talk to a supervisor to learn more about what she could do. She eventually came back on the line and told me they could list my average balance for each of the last three months. I tried to explain that I need confirmation that I had at least £800 in my account every day for at least three months, and she just got confused since they do things in dollars. I explained that was fine, as the UKBA would do the currency conversion, so that wasn’t the important thing. I emphasized that an average balance wasn’t going to verify my minimum balance. I even told her that I’m a mathematician, so she should trust me on this. Back on hold for a couple minutes. When she came back on the line, I was offered the same option that would not help me out. She still couldn’t acknowledge that what I needed for was different than what she was offering. I pulled out an example: Suppose I have $850 (I gave up on pounds) in my account for the first half of December (exactly half!) and then withdraw $100 and leave $750 in there for the second half. Wachovia will report my average balance as $800, but I would not meet the UKBA criterion (supposing it were stated in dollars). There was just some awkward silence, followed by telling me they don’t provide daily balances. I tried one more time to tell her that I only needed the minimum balance for the last three months, which shouldn’t be too hard. Again, no dice.

Tired of arguing, I decided to try another option. The UKBA will accept printed online statements, provided they are accompanied by a letter verifying their authenticity. Of course Wachovia can’t provide one of those either. However, for the price of $5 per statement, they could provide me with copies of my statements. We then got into a discussion of whether I would still need a letter, because this impacted who would process the statement request. This resulted in questioning if the statements would be on gray statement paper with color logo or if they’d be like what I could print from online banking. For the former, no letter would be required, but the latter would require the letter. Turns out they would be like the old paper statements I used to get, which means I would not need a letter. Then we went round and round about if I needed the statements stamped. If so, she’d provide them; if not, customer service would do it. Again, I had to ask what sort of statement customer service would provide. Since it met the UKBA criteria, I was happy to move on to customer service. I think she was happy to get rid of me and promptly transferred me.

The main Wachovia customer service call center is staffed by much nicer people, it seems. I got a gentleman who seemed very happy to be helping me on a Friday afternoon. He was very conscientious about doing everything he could to ensure I would not be charged for the statements and looked into getting a consolidated statement for my two accounts. Even though I wound up on hold with him for some time, I genuinely felt that it was useful. The only scary moment was when he wanted to put me on hold so he could talk to balance verification after I said “immigration”. I quickly interjected that I’d just been on the phone with that department and they weren’t helpful to me. He thanked me for letting him know and saving him time and we went forward. It looks like I’m now going to get the statements I need to prove to the UKBA that I have enough money. However, I still can’t get over the fact that a bank employee could not understand the difference between an average balance and a minimum balance. If I didn’t need months worth of banking history in the immediate future, this experience would have me  seriously considering finding a smaller bank where I could have worked with someone local and gotten exactly the letter I need.