Tag Archives: technology

Surveys Sins

I feel like I’ve completed a lot of surveys lately.

Surveys can be a really effective way to gather information easily, especially if they are online. Unfortunately, it seems that the easy-to-use technology allows people to forget that they are asking a lot of their respondents. In this post, I am going to look at some examples of surveys I’ve been asked to complete lately, in order to understand what is bad.

Screen Shot 2015-08-01 at 1.46.42 AM

Example A: This survey was sent out to hundreds of physics teachers on a listserv, asking them to help with doctoral research.

Sin 1: Incomprehensibility. Here, the first question is misplaced (it should come after the fourth question, I think).

Sin 2: Question types not matching the information being requested. Note above the use of small text fields for questions asking for numbers (which is fine), but then the use of the same small text field for a question asking the respondent to provide an elaborated, “please describe”, response. A similar sin is confusing radio buttons (circles – choose one) and checkboxes (squares – choose zero or more).

Sin 3: Asking irrelevant questions. The question about gender, above, doesn’t seem to be relevant to the analysis (there are no further demographic questions in the study).

Sin 4: Making questions required (especially demographics). It is a cornerstone of modern research that human studies need to be voluntary, and that participants have the right to withhold information and skip questions, especially about demographics. Making questions “required” violates this expectation.

Sin 5: Surveys that are too long. The survey above has 64 Lickert-style questions (see here). Honest, thoughtful answers will require much more than the 15 minutes suggested in the introduction. Too-long surveys inconvenience the respondent, and risk invalidating the study by causing respondents to quit, or stop taking the survey seriously.

Screen Shot 2015-08-01 at 1.39.15 AM

Example B: This survey was sent to teachers and seems to be designed to assess how teachers’ understanding of student misconceptions evolves as a result of professional development.

Sin 6: Poorly-designed questions. The above question seems to have been written following a misconception (that superconductivity means faster electric flow), but the question doesn’t work here. High school students are unlikely to study superconductivity in meaningful depth, if at all and there is no way for respondents to indicate this (the “sturdier” option can be easily discarded via testwiseness). Thus, there are really only two possible solutions. Options D and E don’t work for the “Wrong Answer” row. This question type worked for other items, but it doesn’t belong here.

Screen Shot 2015-08-01 at 1.35.29 AM

Example C: From a survey nominally requesting feedback about the process of grading an exam.

Sin 7: Questions that don’t follow. Above, second question implies a certain response from the preceding multiple-choice question. I think the intention is “if you answered in such a way, then why?” but this is not the prompt that is given.

Sin 8: Technical errors. The second text box appears for a second subject, but if no second subject is entered, the box remains. This can be confusing and distracting.

Screen Shot 2015-08-01 at 2.31.54 AM

Example D: From a survey requesting information for college counselling.

Sin 9: Redundant questions. The above survey effectively asks the same question three times: first, it collects my username; second, it asks for my name; third, it asks for the title of my course. Example B, above, required me to enter personal demographic information twice. Surveys should not be asking for information that can be obtained more reliably elsewhere.

Proofing Report Cards

At my school, we write report cards quarterly. Each student gets a unique paragraph from each teacher. Of course, such lengthy writing needs to be edited, and so we have “buddies” who read through our work. And these reports are lengthy! My editing buddy wrote about 7500 words this quarter — for comparison, that’s several times longer than Swift’s Modest Proposal, and more than a quarter the length of Dickens’ Christmas Carol.

Carefully reading through so many reports (especially while sleep-deprived from staying up late to finish your own the previous night) is a challenge. Here are three tricks I use to help catch errors:

1. Use Find/Replace to remove repeated text. Sometimes we need to add the same note to each report, but there’s no point reading the exact same thing 100 times.

2. Use a screen reader (ie: VoiceOver on Mac, Narrator on Windows), and crank up the reading speed. Since my editing buddy is a pretty good writer, the sporatic mistakes are typos, missed words, or grammatical errors like verb tense agreement. I can hear grammatical errors much better than I can see them, and with a pair of headphones on, I can do the washing-up at the same time.

3. Copy/paste the whole shebang into a text editor (ie: Word). The grammar check on my word processor is better than the spellcheck on my browser, or on Google Docs.

Open-Ended Questions on Twitter

Screen Shot 2014-05-03 at 3.36.57 PM

Twitter is a great way to push a quick message to a broad audience or, in other words, to get a quick shout in the internet hubbub. Over the past year, I have been trying different ways to use Twitter as a novel educational tool. I’m not interested in having students write tweets to report their work, as that can be done just as well with paper, pen, and the ability to count to 140.

Screen Shot 2014-05-03 at 3.16.26 PM

Instead, I think the power of Twitter is that it is always available. I can sneakily get my students to think about educational topics while they’re out with friends, hanging at home, or on the bus. I have been tweeting open-ended questions, usually accompanied by an illustrative photo, that might provoke students to think deeply about non-trivial science questions.

Screen Shot 2014-05-03 at 3.36.10 PM

I try to tweet a combination of questions that can be answered with a calculation, questions that have generally-accepted answers, and questions that are truly mysterious for deep reasons. I personally love these last most of all, but I know many students find them frustrating.

Screen Shot 2014-05-03 at 3.36.31 PM

“Who Owns the Learning?”

A little less than a year ago, I attended the CEESA regional schools conference and met education technology savant Alan November. For years, November has been at the forefront of the movement to use technology to deliver improved education. He’s taken on the role of provocateur: he challenges the education community to use technology in new ways, in better ways, and in the right ways.

November’s latest book, Who Owns the Learning, is clearly designed to be a rebuttal to those who would claim that the provocateur-in-chief isn’t leading from the front. The core of the book is a presentation of four uniquely 21st-century learning activities. They are:

  1. Students use screencasting software plus a microphone to record a video tutorial.
  2. Note-taking duty is assigned, on a rotating basis, to one student in the classroom; notes are then shared electronically.
  3. Students should learn how to use search engines (Google, etc) optimally to do research online.
  4. Classrooms should collaborate (via Skype, etc) with authors, experts, and other classes around the world.

None of these ideas is revolutionary but, on reflection, none are prevalent in my school. I decided to try out each idea, and see whether they offer the benefits November claims.

Video Tutorials

I would describe the stumbling-block here as a low bar to mediocrity. Most students are reluctant to record their work, especially concepts they’ve only recently mastered. On the other hand, once a student feels comfortable doing a recording, there’s very little to ensure the recording is of useful quality.

I tried this concept with a 12th-grade mathematics class. I set up a multimedia station (including a BLUE snowball microphone, a Bamboo tablet, and a HD webcam) in the classroom. However, after I explained the process, there was very little interest from students. Ultimately, only three students completed the assignment. Most felt too shy to record their work and, since I had only one station, I couldn’t require that they do the recording during class-time.

The tutorials that did get recorded and shared, however, were not of stellar quality. The student spoke too slowly, explained simple arithmetic processes instead of the complex problem-solving techniques that were required to reason to a solution, and didn’t edit out long pauses while they fumbled for their calculators.

Reflecting now on the utility of that process, I doubt that much was learned: the process of solving the problem was completed in the early stages, there was little impetus for reflection on the process, and most of the time was spent on technical aspects such as setting up the lighting and recording multiple takes. Using a pen+tablet has a steep learning curve, too, so the students felt they’d wasted time with that approach, before resorting to filming a notebook from above.

I think the classical approach — having students master different skills, and then teach each other — is a much better use of time. The end-products were barely watchable, and certainly wouldn’t hold the interest of an uninvested viewer. Unfortunately, video tutorials end up being an application of technology for technology’s sake.

Here are a few of the video tutorials (“vodcasts”) we created.


I tried the idea of scribes with my 11th-grade and 12th-grade math classes. We randomly chose a scribe for every class when I’d do some lecturing, worked examples, etc (but not, for example, classes where we just had a quiz, or where we solely did group work). It was not effective.

An unstated truth about note-taking during class is that it forces the students to stay focused on the content matter. Without the requirement to take notes, the students should have been squinting and pointing at the board, asking more questions, and trying to understand the logic in every small step. Unfortunately, that’s not how mathematics is learned (not even in an ideal world). We introduce new ideas in math one at a time, with the majority of the work being familiar. For example, the power rule of derivatives is brought in, but when we use it we also employ multiplication, exponent rules, and so forth. In other words, there isn’t all that much that is new to puzzle over!

Worse than the fact that the scribes idea encouraged my students to drift off during class was that the Scriptorium (where I uploaded and shared the class notes recorded by the scribes) was hardly ever accessed. Most of the class notes, in fact, have never been read. I could require that the students access them — for example, by writing the homework on those pages — but that wouldn’t change the fact that teenagers do poorly enough reading their own notes, never mind those taken by a presumably-untrustworthy peer and accessible only electronically.

Here are the Scriptoria of my Grade 11 and Grade 12 classes.


This student learning concept is of a different nature from the other three. It is not a particular activity, but rather a suggestion that we educators do a better job of teaching students how to conduct research. The point is well-taken (by me, at least): the Dewey Decimal System has had its time in the limelight, and the research skills we need to teach should reflect the fact that professionals are turning to the internet, and not the library, for their information needs.

But I think November’s focus here is too narrow: he seems to have fixated on particular ideas (using TLDs to filter quality of web sites, social bookmarking sites like Diigo, customizing search engines) that are likely to be irrelevant in a decade. Instead, we ought to be teaching students higher-level approaches: how can you use contextual information to determine the validity of information, what are academic research journals (and how trustworthy are they!?), and what constitutes an authoritative source, anyway?

The nature of knowledge itself is undergoing a revolution, and our students desperately need to be prepared.

Global Collaboration

November starts the description of his final learning approach by describing how the online fan fiction community is giving young writers a chance to share their work with a global audience. The characters, settings, and tropes of popular works (Harry Potter, Star Trek, etc) give youth a comfortable playground within which to write, and the global interest in these franchises gives them an instant audience. The point is that students are no longer isolated creators, so classrooms shouldn’t be isolated either.

November suggests using Skype to videoconference with experts (including authors, which I think is a fascinating idea — if a bit exhausting for those who making their living writing YA lit!) and with other schools. Although I am a bit doubtful of the curricular benefits for a mathematics classroom — who are we going to call, Hipparchus? — I can see how it would be a great way to build some interest and spark some interesting why-does-this-matter discussions.

Last autumn, I took a class of physics students to visit the Large Hadron Collider at CERN. Hearing from the men and women who work there was exciting, but the overwhelming majority of the buzz came from the locale: there’s nothing quite like standing beside a billion-Euro piece of equipment, touching mock-ups, and wandering around a museum filled with the devices that discovered the particles we talked about in class.

Four years ago, I was coaching Mongolia’s national debate team as we prepared for the world championships. One of the motions was about Guantanamo Bay. After a couple weeks of research, we videoconferenced with a friend who does some pro bono legal work on behalf of the detainees. In that circumstance, her more-nuanced, professional understanding was appreciated, understandable, and educational. We were able to prompt lines of discussion that didn’t make the newspapers, ask questions about technicalities that were of interest to us, and to get some inside information that wasn’t widely known.

In some situations, videoconferencing will be informative and useful. But in others — mathematics and sciences, for example — there’s not a lot they can contribute.

The most difficult part with this last aspect is the problem of forming connections. It takes work to track down the right expert through social channels. And, as committed as they are to social outreach, many researchers simply don’t have time to chat with individual classes.

Over the past couple weeks, I’ve been doing science demonstrations for primary school classes at my school. These are (comparatively) simple things, such as launching rockets, playing with dry ice, and whatnot. Given the choice between the virtual and the physical, I think I’ll stick with the physical.

At least, until OS One is released.