Emergent Curriculum in Big Classes

Mylène diPenta’s tremendous insight into the role of power in our learning relationships has been resonating in my head for the past five years, and serves as inspiration for the work I describe in this post.

This semester, I sought to develop and implement a pedagogy in a large-enrollment introductory physics class that was (a) rooted in rich demonstrations, (b) relied on students’ own development of key concepts (like Modeling Instruction), and (c) empowered the students to construct their own curriculum. Here’s what I came up with as a learning cycle, which would last about 2 weeks:

1. I demonstrate an interesting phenomenon in class
2. Students engage in sense-making and model construction in small groups
3. In real-time, student groups will share back their scientific theories and emergent questions related to the phenomenon
4. I do some chalk talk to pool together the students’ ideas and also to standardize notation
5. I also work through a couple examples of ways that the ideas could be transferred to different contexts
6. Students go back to their groups and work through a couple practice problems
7. Students work in groups to propose (a) a statement of a learning objective related to this phenomenon, and (b) a couple examples of ways student understanding could be assessed
8. A consensus learning objective is recorded, shared, and added to the syllabus; also, I use the assessment proposals to compose a homework assignment for individual practice
9. Students generate a list of questions based on their curiosity about extensions to the phenomenon, which gives me some ideas to prepare for the next interesting phenomenon

I tried this out for a couple months. We made it through units on simple harmonic motion, electric charge and forces (Coulomb’s Law), and electric fields (including integrating continuous charge distributions, but not Gauss’s Law). However, it has become apparent that this learning cycle wasn’t effective in my context. Here are some reasons why:

A. We lost four days of instruction to inclement weather, which meant I needed to rush a bit to address all the concepts that are required in this course for the engineering students’s professional accreditation. This learning cycle is very slow. I’m okay going slower and covering a bit less ground if it means students will learn better, but the learning gains weren’t apparent, and this was just dragging out too much.

B. The students weren’t engaging in the depth of reflection that I was hoping for when they wrote their learning objective proposals and examples.

C. I ran into a bit of a wall trying to come up with rich, multifaceted demos for the next few concepts, such as Gauss’s Law and electric potential. I think that a demo-led pedagogy is possible, but it’s gotta have some really good demos. Additionally, the lecture hall is some distance from my office (which limits how much, and what, I can lug over), and it is sometimes challenging to pull together rich demos as an interloper in the department, here.

I think that portions of what I attempted here will be useful for the future, but overall I’m going to call this one unsuccessful. Fortunately, there’s plenty of time left in the semester to pivot toward something that will work better for the students. I’m going back to lectures that include solving problems and some time for students to turn and talk about concepts and practice solving problems, in class. This course also has a 2-hour tutorial session on alternating weeks, so I get some time for the students to dig into some small-group problem-solving: that has been tremendously valuable for helping that all of the students are getting some practice with the key skills and ideas, regardless of what happens in the lectures.

Diversity Falters

As part of an analysis of some things, we happened upon an interesting sort of paradox involving the benefits of classroom and small group diversity. Let me explain the assumptions (most of which I think are reasonable for a college physics class, for example, although simplistic):

1. We’re looking at gender diversity, and assuming a binary, for this model. Our students are boys and girls. At the classroom level, there is a certain level of diversity which we can parameterize by, for example, the fraction of students who are girls.
2. Students work in groups of 4.
3. Because of stereotypes associated with the class, some of the girls (but not the boys) prefer to work only with other girls and will form all-girl groups. Let’s call this self-segregation, and we can parameterize it by the fraction of girls who choose to work only with other girls.
4. Aside from the girls who self-segregate, all other students join groups randomly, without regard to gender or other factors (I use binary probability distributions).
5. Compared with single-gender groups, groups that have 1 girl and 3 boys are less effective in terms of learning outcomes and experiences (I’ll call them “toxic”), perhaps because of in-group/out-group effects within the culture of a masculinized discipline or classroom. Groups with 2 or 3 girls are about as effective as single-gender groups.
6. Because of an intervention, we hope to change (ie: decrease) the fraction of girls who choose to self-segregate.
7. We are interested in increasing the fraction of girls in a classroom in which they are traditionally a minority.

Then, for some values of the segregation fraction and the fraction of girls in the class, decreasing segregation and/or increasing the fraction of girls in the class will increase the fraction of toxic groups, which could have a net detrimental impact on outcomes/experiences. Or, in other words, sometimes, even though women benefit from working together, increasing the fraction of girls in the classroom will result in girls being isolated, and thus performing worse and having worse experiences. Likewise, an intervention that makes students feel more comfortable in the classroom may, in some cases, be net harmful.

Here’s a graph. Imagine self-segregation is 0.5 and your class is 0.25 women (blue line). If you increase the fraction of women in the class to 0.35 (orange line), the fraction of groups with 1 woman & 3 men increases. If you introduce an intervention that reduces the self-segregation fraction to 0.4, the fraction of groups with 1 woman & 3 men increases.

This isn’t a novel idea. Girls in a boy-dominated class have their self-defense mechanisms in place. They know that being the only girl in a group of all boys is hazardous. They prefer to work with other girls as a way to avoid that experience. Well-intended changes that disrupt that equilibrium without addressing the core inequity may have negative consequences.

Anyway, I thought that little calculation was sort of interesting. Thanks for reading!

Labs: What Works?

Here’s a collection of some experimental results about different curricular approaches to introductory college physics labs:

Curricular Approach	Conceptual physics understanding (FCI, FMCE, CSEM, etc)	Attitudes toward experimental physics (E-CLASS, etc)
Traditional (highly-structured, concept-focused)	no change	negative
Integrated lab+lecture (eg: ISLE, SCALE-UP, Studio Physics, Modeling, TEAL)	positive*	no change
Conceptual inquiry-based (eg: RealTime Physics)	positive	unknown**
Skills-based teaching experimentation (eg: Thinking Critically)	no change	positive
Open-ended (eg: project-based labs)	unknown	no change
Integrated lab+tutorials (eg: Labatorials) with reflective writing	positive*	no change (DFEBQ)

* integrated labs don’t provide an apples-to-apples comparison
** I’m working on a paper that addresses this

This table suggests a few possible research agendas about introductory physics labs:

1. Fill in the gaps (eg: E-CLASS in RealTime Physics? FCI/FMCE in project-based labs?)
2. Developing/identifying/studying additional curricular approaches
3. Identifying/studying additional metrics (eg: agency, critical thinking, scientific reasoning, uncertainty)

Whither HS PER?

I’m trying to put together a few thoughts about high school (HS) physics education research (PER): why it’s needed, why it’s difficult, and what kind of support it needs to take off. I come to the discussion after teaching high school for 10 years and near the end of a physics Ph.D. focused on PER, with research at the university level.

Why High School PER?

High school physics teachers have always been the most innovative, creative, and responsive educators teaching physics. They’ve been the first to work with standards-based grading, the first to adopt new pedagogies like Modeling, and so on. Over the past generation, in my opinion, dedicated high school physics teachers have developed a remarkable arsenal of instructional and assessment strategies, approaches to lab-work and improving classroom equity, and other pedagogical techniques. Blogs, twitter, and PLCs allow us to document some of this work, but the short life-span of these forms of records may make it difficult to share with the next generation of teachers. Research publications would allow expert teachers to provide durable and significant insight to future teachers, whether they are in teacher preparation programs or are reading about physics pedagogy on their own time.

While teaching is a rewarding vocation, I think that many of us feel the itch to take the “next step” — into a school leadership, as a department head, as a coach, contributing to curriculum review, or seeking to innovate instruction or assessment. For some educators, research can be a fulfilling part of reflective practice that helps to round out the job.

What are the barriers?

For physics teachers interested in getting involved with physics education research, there are some significant and substantial roadblocks. The first of these is time. PER scholars at the university level often include research in their 9-5, whether they are grad students or professors. For some high school teachers, PER can be a hobby, but I don’t know of any who have been able to negotiate research time into their employment contracts.

A second barrier is that PER exists at a sort of continental divide. I will use some organizations in the USA to make this point, but this issue is pervasive. K-12 science education research (such as the NARST community) rarely focuses on specific disciplines in high school, and tends to be centred on Schools of Education at big universities. The NSTA is more teacher-focused, but misses out on research, and high school disciplines don’t really have a home base there. The AAPT welcomes high school physics teachers, but there is a gap between many researchers (who tend toward PERC, anyway) and high school teachers. In short, high school PER scholars don’t have a home base.

A few years ago, while teaching high school physics and math, I tried to write an article for The Physics Teacher. I can see now that it was embarrassingly bad, and it was (rightly) rejected. In the past four years studying PER at grad school, I’ve learned how to write papers about PER, how to do a literature search, and a million other little things that help me to be a (hopefully successful) physics education researcher. On top of that, I also learned a ton about research methods, from statistics to qualitative methods. Most high school teachers can’t spend half a decade in grad school to learn how to do PER. But, more importantly than the expertise gap, there is a confidence gap. It was tough enough for me to build up the courage to write an article for a teaching-focused journal, there was no way I’d have the confidence to submit a technical article!

Another issue facing high school teachers is ethnical research practices. Some schools and school boards say a flat “no” to requests to conduct and publish research on students. Parental consent may be required in many other circumstances. At universities, ethics are codified into Institutional Review Boards which, while not easy, at least make it possible to conduct human subjects research. Negotiating permission to study and publish is another skills that high school teachers need to develop in order to practice PER in their classes.

Research follows money, and in the USA (and many other places) the current funding climate is set up to support college-level education research, graduate student training, and programs that increase the number of high school physics teachers. Even when small grants are available, it can be difficult for high school PER scholars to find out about them, find time to apply, and even build up the confidence to go for it. Another funding issue is that there are baseline costs associated with PER that are insignificant for many university researchers, but stumbling-blocks for high school teachers, including the cost of attending conferences, article processing charges for publications, and membership fees for professional organizations.

Finally, it is easy to overlook the importance of having access to a university library. Too often, as a high school teacher trying to do research, I would stumble across an amazing-sounding paper that I couldn’t access because of paywalls. Sci-hub and open-access publications (like PR:PER and the PERC proceedings) may be lessening this burden.

A Way Forward?

High school PER scholars need a home, community, support, and respect. I wonder if it would be possible to apply for some medium-sized grants (eg, $5k from PERLOC) to set up a community that would provide:

• A series of online night classes on PER-relevant methods
• Mentoring from experienced PER scholars on designing experiments and publishing results
• Connections with PER scholars who can consult on specific topics or methods
• Space for collaboration between researchers seeking to work on a single project
• Opportunities to talk about your research and get feedback from fellow high school PER scholars
• Financial support to attend conferences and publish papers
• A regular colloquium talk series to introduce current broad areas of interest in PER

Faculty at two-year colleges experience a lot of the same expertise and constraints as what I described above, and so I think they would be a valuable addition to this little proposal. Personally, I’m not sure where I’ll be in 6 months, but no matter in what context I end up teaching, I’d like to continue to support work like this. I love high school physics.

If you have opinions, are interested in making this happen, or want to poke me to ask why I haven’t done anything about this even though it’s been X months since I wrote this post, send me an email: danny.doucette@gmail.com.

Managing Cheating

To my friend M,

Next week, you’ll be teaching as an instructor-of-record for the first time. I am so excited for you, and for your students who will get to learn from a kind and insightful instructor. There’s been a lot of talk about cheating lately and, as someone who has been “around the block” a few times, I wanted to put together a few thoughts that might be helpful for you this semester.

Why Do Students Cheat?

In our large-enrolment introductory physics courses, few of the students are there because they are interested in the subject. Mostly, they’re enrolled because they need Physics 1 & 2 for med school (or another health track) or engineering. So while we see the beauty of physics, our students view the course as a hurdle to overcome — either a checkmark for their med school applications and (maybe) some preparation for the physics portion of the MCAT, or a “weed-out course” required by the school of engineering.

Physics has a reputation as a difficult subject: esoteric, highly mathematical, nearly impossible to master. So when students who would rather see the course in the rearview mirror struggle with physics problems, they have little motivation to seek out help through proper channels: going to office hours, asking their TA for help, checking in at the physics resource room, or collaborating with classmates. After all, why should they struggle to learn something they don’t care about? Then, later on, on high-pressure final exams for which they haven’t properly prepared, students cheat because they need a certain minimum grade in the class and can’t imagine earning that grade without the outside assistance they’ve grown to rely on.

At the same time, resources that feed students answers under the guise of helping them learn (ie: Chegg) are now available, and growing fast in the student collective consciousness. Worse still, there is a negative spiral: students who use Chegg feel defensive about it and tell stories about how bad their instructor is, how difficult physics is, how modern work requires resourcefulness, and how everyone is doing it. These stories create feedback when other students hear them: I thought the instructor was okay, but maybe he really isn’t very good? I thought Chegg was cheating but if everyone is doing it then maybe I have to as well, in order to keep up?

Here are some student comments from our university’s subreddit about cheating this past semester in the introductory physics courses:

I’m not outraged nor am I surprised. Cheating happens all the time.

People say “don’t cheat it’s wrong” but I guarantee each and every one of us have done something academically unethical.

It’s college and in these times your GPA is now apart of who you are… So why wouldn’t students use ALL their resources to better themselves?

In summary: students cheat because they see their classes as hurdles to overcome, feel pressure to succeed, and are given permission from their peers since cheating is commonplace and acceptable. So what can we do about it?

 

Preventing Students From Cheating

Everything in education that is done well needs to start from a place of compassion. How can we help students avoid the allure of cheating? The equation below suggests three starting-points:

1. Help students to see physics as something more than just a hurdle to overcome. Share your joy and enthusiasm with the class. Invest time in developing lectures and homework problems that are genuinely interesting. Go deeper, too: understanding physics fundamentally changes how we view the world. Compare a neophyte’s reaction to the bowling ball pendulum to that of a physicist, for example. Help students understand how their studies in physics are actually vital to their future careers in medicine or engineering by bringing in relevant, real-life examples from those fields.
2. Reduce the pressure of high-stakes exams by instead having multiple tests. Which do you think students would find more stressful: two 20% midterms and a 30% final, or a 10% test every week for seven weeks? The two approaches do equally well at assessing student learning. If you have the support to do standards-based grading (sadly, we don’t in our department) then invest the time and energy in doing that. Decrease the pressure by giving students plenty of useful, quick feedback about their performance by using formative assessments like quizzes (I like quizzes that have a multiple-choice component that can be graded instantly and a problem component that can graded overnight and returned 1-2 days later with detailed commentary).
3. Remove students’ permission to cheat by establishing unambiguous rules about what is, and what isn’t, academic misconduct in your class. Do this in the syllabus, spend 15 minutes in the first class talking about it and giving examples, and explain how you will respond to first and further offences. At our university, further offences are out of your hands, and go straight to the Dean. Don’t establish a rule you can’t or won’t enforce. Have students sign an honor pledge and make sure they understand the university’s Academic Integrity Code (and know it, yourself). Enforce your rules strictly: if you give them an inch, they’ll take a mile.

 

Catching Students who Cheat

Our university has a subscription to GradeScope, a platform that allows student work submissions and streamlines grading. I suggest using it because it makes it easy to store and compare student work. TurnItIn is integrated with Gradescope, and it is a good idea to use TurnItIn if your students are submitting any longer written work.

Don’t use textbook (or Sapling or LON-CAPA) problems directly as they can be easily found online. If you’re not sure about writing your own problems, re-write textbook problems. Be sure to change all names and places, and rephrase as fully as you can. You might as well be clear about this with students so they don’t waste their time searching.

Where possible, seek ways to give each student a customized problem so they can’t copy someone else’s answers directly (and you’ll be able to notice if they do). Here are some ways to do this:

• Have students use their student ID number (or the Nth digit in their student ID number) as one of the parameters in a problem
• Create multiple different versions of the same assignment/test by (a) rearranging the order of questions on the assessment, (b) reordering the answers in multiple-choice problems, (c) changing the values of some key parameters, and/or (d) using similar but non-identical problems
• Some of the biology instructors have been using scripting in PDF forms to customize assignments. Students type their name / ID number at the top of the page, and the values of key parameters on the page change as a result. You can create such forms using Adobe Acrobat.
• Ask questions that require students to use their knowledge of physics in a context that requires a written response. Context-rich problems are good for this, if you require that students write a couple sentences explaining their reasoning and results.

Even with all the preventative measures in place, you will likely still find student submissions that duplicate the work of others, either online or other students in your class. If you’ve been thorough and deliberate in establishing and explaining your response to academic integrity issues, and given students plenty of timely feedback on their learning, the next steps will be clear and unambiguous, and you will have a good response to the inevitable sob stories.

If you find your original problems on Chegg, you can submit a DMCA takedown request, which typically takes a couple days to process. Chegg also stores the IP addresses of its users. If you suspect students were cheating on a test or exam using Chegg, the Dean can request records as part of their investigation. More information here.

Hope this helps! Your friend, -D.

Taking Labs Online

I want to briefly summarize what we did at the University of Pittsburgh to take our introductory college physics labs online in March and April, 2020, when classes for the last 6 weeks of the semester were interrupted due to Covid-19.

The Lab

We had a bit of 400 students enrolled this semester. In order to better serve health-science track students, we run the introductory physics lab as a separate 2-credit, one-semester, offering from the Physics 1 and 2 lectures. Each section has a maximum enrolment of 24 students, and is led by a graduate student teaching assistant (TA). Typically, students attend a one-hour lab lecture and a 3-hour lab session each week. They get graded based on attendance at the lab lecture, a pre-lab assignment, a digital lab report that is completed in pairs during the lab (and usually finished up and submitted afterward), and a short lab homework assignment. We use Gradescope to streamline the submission of work and the grading.

New Policies

The University of Pittsburgh announced the switch to online learning mid-way through the spring break, and immediately extended the break by a week. That meant we had nearly two weeks to figure everything out and communicate with TAs and students. The lab manager and I met and made a few decisions:

• We would reduce the number of labs from 12 to 11
• We would allow students to drop the grades from their two lowest-scoring labs (originally the policy was just one lab)
• We would develop online versions of the lab reports that could be completed individually
• The pre-lab and lab homework would stay the same
• The lab lecture would be dropped, but the slides would be posted online for students to read
• We would ensure that all deadlines would be flexible for students
• TAs would continue to play an important role in delivering these courses

Gradescope turned out to be a good technology: by this point, students were comfortable with uploading their own work and assigning pages to questions. Presumably those whose partner always took care of it at least knew someone to turn to for help. The TAs were able to continue their grading responsibilities from home.

New Labs

Our labs are loosely based on Real Time Physics, and have been modified so that there are 20 “chunks” of work (either a prediction, an activity, an in-depth question). Each “chunk” is described on a separate page of a Word document, and students write, insert graphs and pictures, fill in tables, write formulas, etc to complete their lab report. Since these online labs were being done solo, rather than in pairs, we reduced the number of “chunks” from 20 to 10. That felt like a good amount of work for students to do in 3 hours at home by themselves.

We relied heavily on some of the excellent physics simulations that have been developed over the last few years. We chose only simulations that run either in HTML5 or Javascript, making them functional across nearly all devices. We used simulations from PhET, Physlet Physics, the Physics Aviary, and the excellent Falstad circuit simulator. Each lab used 2-3 simulations, and asked students to collect and analyze data, make and check predictions, develop conceptual understanding, and reflect on some of the bigger questions (eg: should it really be called Snell’s Law?).

Implementation

I was worried that some of our students would be unable to continue their studies after the university went online, but that seems not to have been the case. While 400 students submitted the lab report before the shift to online instruction, 405 submitted the first online lab report, and 393 submitted the second online lab report. I reached out to the 20-or-so students enrolled in the class who had not submitted the first online lab report, and heard back from about half. Most asked for a short extension to finish the work as they moved or adapted to a new schedule. Two students indicated that they were struggling with online learning and expected not to be able to complete their classes (I wrote back with some resources, and heard nothing further).

Although the lab reports and at least one email asked students to send any complaints or criticism about the lab reports to me, I heard from only two students. Both wrote to say thanks for developing the materials and for making the online transition a smooth one.

We asked the TAs to be available to students during the first half of their regularly-schedule lab sessions via chat on Blackboard Collaborate, and also to aim to respond quickly to emails at this time. The TAs received very few questions via either medium, but I think it was valuable that they made the effort to be available to their students. One downside of the transition to online learning is that we have fallen behind on the grading. I expect that, unfortunately, the delay has resulted in some uncertainty for the students, who can usually see grading as soon as it is done on Gradescope. Most of the TAs for the labs are graduate students in their first or second year of studies, so the switch to online learning was significant for them because it also affected their own classes. Many are also international students (like myself), and that has made the closing of the university extra challenging.

Moving Forward

We will be able to evaluate the effectiveness of our approach to online learning through concept inventories and an attitudinal assessment that we regularly do with this introductory lab. However, looking toward online learning during the summer semester, we’re not sure that these simulation-based online labs are the best approach. We are trying out the IOLab system right now, and might adopt it for the July-August lab offering.

Finally, a few take-away messages based on what worked in our lab and what we’ve seen to be less effective in our class and others:

1. It’s important to scale back expectations, accommodate lateness, and design work that can be done individually by students without their regular support networks (office hours, after-school help, peers, class)
2. Early, frequent planning and communication are the key to getting a viable system in place and to getting students to buy-in to your approach to online learning
3. People don’t like passively watching videos. Simulations are better, and there’s lots of them out there.

Lab Reports

Here are the lab reports our students completed:

Report 08 (Online Version)

Report 09 (Online Version)

Report 10 (Online Version)

Report 11 (Online Version)

Doing Interviews

I’m working on a paper that is based on a series of interviews my supervisor and I conducted over the past year. The interesting part of that research effort is the content of those interviews, and so that’s what the paper is about. However, I’ve been thinking a fair bit about how to conduct ethnographic research in introductory physics labs, including interviews, and want to share the model we’ve come up with. Here, then, are some considerations for doing interviews in contexts like my own.

Finding Participants

It’s tough to get students to agree to do interviews. It takes time during a busy semester, it’s something completely optional for them, and the financial incentive I can offer ($25, in my case) are a drop in the bucket compared with the cost of college. More significant, though, is fear of the unknown: this “interview” could be all sorts of unpleasant.

Thus, I combine my interviewee selection with organic discussions I have with students in their labs. By the end of the semester, they’ve gotten to know me as a friendly person who regularly visits their room, occasionally offers help with their work, and respects them as learners. Thus, it’s pretty easy for me to say, “I think this is the dial you’re looking for. By the way, would you be willing to sit down with me for an hour and talk about your experiences in the physics lab?”

When I approach students who have gotten to know me, at least a little bit, the response rate for interview invitations is about 75%. But when we approach students out of the blue, less than 50% of our invitees responded to email invitations. And even this response rate is impressive compared with the meagre return we get from poster advertisements. I think this shows the tremendous importance of establishing a foundational level of trust and respect before even beginning the interview.

Pseudonyms

When we report quotations in our publications, we use pseudonyms to protect the anonymity of our interview participants (note: participants, not subjects). As part of the introduction sequence for my interviews, which covers the ground rules set out by our IRB among other things, I ask my participants to choose their own pseudonym.

The result is that our pseudonyms are sometimes gender-neutral (and so I need to be explicit about the respondent’s gender if it is applicable in my analysis), and they usually don’t meaningfully reflect the ethnic or racial background of the participant. But they do give the participant a little bit more say in how their ideas are being represented, and I think that’s important.

Who Benefits?

I worry a lot about how interviews are a form of resource extraction: I, as a white man, am transcribing and using the ideas and thoughts of the participants (many of whom, by design, are underrepresented minorities) in a way that will benefit me directly, by helping me to achieve a degree and maybe a job. So I make it a point to frame our discussion in terms of how they can help future students by helping to redesign the class.

This practical, outcome-guided, framing is important for any research in which there are extra incentives for the researcher. Then, too, I think it is important to be clear with the participants about the goals — that this interview will help us improve instruction directly, and might also lead to an academic publication.

Making Meaning Together

It is a mistake to think that the meaning of an interview is created when the research and/or their collaborators pore over transcriptions and debate frameworks. Rather, the interview comes with meaning built into it, because the participant has their own frameworks and ways of understanding. Often those ways of understanding are better than the researcher’s.

This stance helps me ask better questions, and to participate better in dialogue. Our interviews are semi-structured: we compiled a list of 30 “questions”. Actually, these are things that we want to ask about. Usually, we will seek to weave these questions into the discussion.

Transcribing and Writing

I’m not very good at it, and I don’t know a good solution (let me know if you do!), but I transcribe all our interviews after we finish them. We also write post-interview reflections, in which we seek to identify key issues. After the interviews, we meet to discuss what we heard. This is when the coherent narratives start to emerge.

The paper I am working on now seeks to allow the participants to tell their own story, by copious use of long quotations. I’m also using a framework to help draw conclusions from what is being said.

 

Points & Sets: Lab Reasoning

As I pursue my PhD in physics education research, I have found time to explore a number of interesting questions. In this post, I’ll explore an approach to thinking about student reasoning in the introductory physics lab that we decided not to pursue at this time.

The Physics Measurement Questionnaire

Building on more than 20 years of development, use, and research, the Physics Measurement Questionnaire (PMQ) is a staple in introductory physics lab research. Its home is the group of Saalih Allie. This 10-question assessment is based on the following experiment:

Students are asked a series of questions that seek to determine the extent to which they agree with a set of principles about how lab-work is done (at least, in the abstract). The principles don’t appear explicitly in the relevant literature, but seem to be:

1. More data is better
2. When collecting data, it is good to get the largest range possible for the independent variable
3. The average represents a set of repeated trials
4. Less spread in the data is better
5. Two sets of measurements agree if the averages are similar compared with the spread in their data

These axioms feel true and, as a teacher, I see value in getting my students to understand fundamentals about the nature of scientific work. However, while they are broadly applicable, the reality of science is that none of these axioms are exactly true. There is the question of whether science has a universal “method” — Paul Feyerabend argues it doesn’t. When I sat down with a distinguished professor to look at the PMQ, the professor could identify the “right” answer, but often didn’t agree with it.

This reminds me of the “Nature of Science” that was brought into the International Baccalaureate (IB) physics curriculum in 2014. It appears in the course guide as a 6-page list of statements about how science works, culminating in this mind-bending image:

So maybe attempting to define how science works isn’t a productive approach. Fortunately, the PMQ isn’t just a test of whether students agree with certain axioms of experimental physics.

Question-Level Analysis

In addition to evaluating students on their agreement with the above axioms, the PMQ also asks students to justify their reasoning. An example question follows:

After “Explain your choice”, the PMQ includes several blank lines for the student response. The instructions suggest that students should spend “between 5 and 10 minutes to answer each question”.

A thorough analysis of student responses is possible by using the comprehensive categorization scheme in Trevor Volkwyn’s MSc thesis. Volkwyn (and the PMQ authors) view the PMQ as an assessment that aims to distinguish two types of student reasoning about experimental data collection:

Point-Like Reasoning is that in which a student makes a single measurement, and presumes that the single (“point”) measurement represents the true value of the parameter that is being measured

Set-Like Reasoning is that in which a student makes multiple measurements, and presumes that the set of these measurements represents the true value of the parameter in question

Alternatively, we could view the point-like paradigm as that in which students don’t account for measurement uncertainty or random error.

Examples of responses that conform to the point-like reasoning paradigm, for the example above, include:

• Repeating the experiment will give the same result if a very accurate measuring system is used
• Two measurements are enough, because the second one confirms the first measurement
• It is important to practice taking data to get a more accurate measurement

Examples of responses that match the set-like paradigm include:

• Taking multiple measurements is necessary to get an average value
• Multiple measurements allows you to measure the spread of the data

Thus, it is possible to conduct a pre/post analysis on a lab course to see whether students’ reasoning shifts from point-like to set-like over the course. For example, Lewandowski et al at UC Boulder have done exactly this, and see small but significant shifts.

My Data

I subjected a class of 20 students to the PMQ, and coded the responses to two of the prompts using Volkwyn’s scheme. The categorization was fairly straightforward and unambiguous.

For question 5, which asks students which of two sets of data is better (one has larger spread than the other), 17 students provided a set-like response and 3 gave the point-like response.

For question 6, which asks students whether two results agree (they have similar averages and comparatively-large spreads), only 1 student gave a correct set-like response, and the other 19 provided point-like reasoning.

I think this indicates two things:

1. Different types of prompts may have very different response levels. Similar results are found in the literature, such as Lewandowski’s paper, above. This suggests that the set-like reasoning construct is complex, either with multiple steps to mastery or with multiple facets. Thus, it might not make sense to talk about it as a single entity with multiple probes, but rather as a collection of beliefs, skills and understandings.
2. Some of the reasoning on question 6 seemed shallow. This suggests, for me, a bigger take-away message: my students aren’t being provoked to think critically about their data collection and analysis.

Going forward, we’ve decided not to use the PMQ as part of our introductory lab reform efforts. However, by trying it out, I was able to see clearly that our new curriculum will need to include time dedicated to getting students to think about the relevant questions (not axioms) for the experiments they conduct:

1. How much data should I collect?
2. What range of data should I collect?
3. How will I represent the results of my data collection?
4. How will I parameterize the spread in my data, and how will I reduce measurement uncertainties and random error?
5. For different types of data, how can we know if two results agree?

Interestingly, some of Natasha Holmes’ work on introductory labs starts with the 5th question by asking students to develop a modified t-test, and then uses that tool to motivate cycles of reflection in the lab. That’s another approach we’ve agreed doesn’t quite work for us, but that is likewise a huge source of inspiration.

 

A Trilemma in Progressive Education

 

This is a quick note about something I keep encountering in practice, but haven’t seen described in The Literature. If you have an references, please send them my way!

Consider a situation in which you are making some changes to educational materials or structures in order to address a bias in the status quo. This could mean re-framing classroom rules, editing materials, or rethinking university admissions. In such a situation, there seem to be three approaches you could take:

1. (Representational) Equality
2. Neutrality
3. Responsivity

Example One: Gender in Physics Problems

This week on Twitter, I mentioned that I was re-writing some physics problems that had been coded (a) exclusively male, and sometimes (b) using inaccurate vocabulary (ie: “spaceman” instead of “astronaut”). An Equality approach would be to make the subjects of the questions 50% male and 50% female. Kate Wilson explains why this matters:

I think it's important to have gender equality or neutrality to avoid leaving girls feeling excluded, even subconsciously. Performance is more subtle – small changes in wording can make big changes in performance gaps. Girls are put off by answers that sound unscientific.

— Kate Wilson (@unswkwilson) June 2, 2018

Summer C emphasizes that gender isn’t a 50/50 split, and that assuming a binary nature of gender is problematic, including for students who aren’t cisgender.

Anything to promote inclusion of ALL students. It means so much to them to be validated in our classrooms.

— Summer C (@TeachFORSCIENCE) June 1, 2018

One such response, then is to attempt to use gender-neutral language (ie: Neutrality). In some cases this works easily, as Jenn Broekman explains:

https://twitter.com/jsb16/status/1002548348191199232

In other cases, though, it takes a bit more work:

@TeachFORSCIENCE and I were just talking about using Zis/Zheir pronouns on documents which would be the next step.

— Bree BarnettDreyfuss (@BarnettDreyfuss) May 31, 2018

I like gender-neutral pronouns in principle, but we’re probably a generation away from them being practical. An alternative is to adopt the 2nd-person perspective:

I read somewhere (and of course I can't find it now), that problems should use "you" as the person. (E.g., "You are pushing a box.") The thinking is that students relate to it better when worded that way. Unintended consequence is avoiding gender/cultural bias.

— Frank Noschese (@fnoschese) May 31, 2018

While “you” seems like a good solution, it’s not always going to be valid — for example, a problem in which there are two independent agents. There’s also the issue that this isn’t solving the bigger problem: even if my problems are gender neutral, students are still getting cultural signals about who is and who isn’t a physicist. This brings us to the idea of Responsivity: using these cultural artifacts to push back against counterproductive messaging and norms students are bringing to the classroom:

And, while we're on the topic, queer the hell out of those scenarios! No offense, but it's past time for us to abandon the binary approach to gender, screw striving for so-called "parity," and write problems that challenge hetero/homo normative assumptions. 2/2

— Chris Gosling (@duckduckphysics) June 1, 2018

This discussion was enlightening for me — thanks to those who contributed!

Example Two: Latent Colonialism

I’ve been slowly putting together some ideas around the idea of “latent colonialism” in physics, the idea that a large portion of the canon of physics culture is a remnant of the colonial era and thus carries some problematic attributes with it. For example, textbooks around the world feature the same group of ~20 physicists (Newton, Franklin, Ampere, Maxwell, Einstein, etc) — all white, male, and Western European. Or consider that the names of the famous physics equations and principles, although developed over a couple millennia, have Western European names. This has been shown to negatively impact students who aren’t white and male.

So, how do we respond? We could:

1. Seek Neutrality. Simply remove the names from the canon. Instead of Snell’s Law, we would call it the Law of Refraction. This approach has been adopted in several curricula, but I see two problems with it: first, instead of giving an advantage to white boys, the curriculum gives an advantage to no-one; and second, culture is complicated and it’s not clear we’d be successful at expunging all the biased artifacts.
2. Instead, we could seek out cases in which white women and people of colour contributed to the history of science and highlight them as well. This would aim to present a canon that is representationally Equal. I’m currently reading up on Émilie du Châtelet, for example. This would allow us to re-cast the history of physics as an endeavour that involved white men and women in Western Europe, and built on the mathematics and early work done in the Middle East, and to a lesser extent India and China. Putting aside the cries of “revisionist history”, this isn’t clearly going fix everything, either. For one thing, the truth is that a lot of contemporary physics knowledge, in the form in which we currently teach it, was assembled by white men who refused to allow white women or people of colour in on the proceedings (with some rare exceptions).
3. Which bring us to the Responsive approach, which is to acknowledge the deliberate exclusion, intellectual theft and, well, colonialism that took place during the construction of the set of ideas we think of today as the canon of physics knowledge. This would be hard: it would involve physics teachers talking about history and racism in their classes.

Likely, the best response is to do all of these, in the contexts in which they work best, to the best of our abilities. For example, I taught for some time in a country in which it was constitutionally prohibited for schoolteachers to talk about gay marriage or other gay rights issues in their classes: that type of thing severely restricted my ability to talk with students about some of these issues. It is my hope that this classification is useful in deciding which response to take.

Intro Physics Labs: Why?

Some of my research these days is focusing on the introductory physics lab. In this post, I will seek to outline some qualitative findings about introductory physics labs at universities and colleges in the USA. I am planning to follow-up these conclusions by doing a survey of a representative sample of physics departments — but that will have to wait until late August.

1. Parallel or Separate?

In most programs, the physics lab runs parallel with the physics course. At Lincoln University, for example, students take Phys 105 and 106 as their two-semester introductory physics sequence, and also enroll in Phys 105L and 106L at the same times. Here, the goal of the lab tends to be focused on reinforcing core ideas. At Penn State, labs “are designed to provide you with hands on experience with the material being investigated in class”. The key concern here is that recent research suggests that labs provide “no added value to learning course content” (Holmes et al).

Less common is a lab that is run as a separate course, sometimes requiring the first course in the introductory physics sequence as a pre-requisite. At Drexel, for example, students take Phys 128 as a separate course. These separate courses tend to focus more on the development of experimental skills and mindsets. At Carnegie Mellon, the purpose of intro lab course 33-104 is “to become skilled at acquiring, recording, and analyzing data, and drawing conclusions from experimental work”.

2. What is the purpose of labs, anyway?

The AAPT lab guidelines focus on the process of “constructing knowledge” and scientific skills, rather than core ideas. Ideally, I think, labs would meet both aims: helping students to enrich their understanding of core physics ideas, while also learning how scientific knowledge is generated experimentally.

In constructing a lab experience, core ideas and scientific skills will need to be interwoven. Could we say that a lab is successful if students only practice core ideas? Could we say it is successful if students only learn scientific skills? I would say that the former is not acceptable, but the latter might be.

3. What labs are being done?

Cookbook-style lab manuals are nearly ubiquitous. Most of these seem to have been written locally, but have the same format: an overview of the relevant physics (which students rarely read), then step-by-step instructions for what students should do in the lab, and finally some questions.

There is variation in how students are assessed. Often, there is a pre-lab quiz. Written lab reports are common (often 1 per group of 1-3 students), as are worksheets that need to be filled in.

Most labs are 3-hour sessions, with a new experiment each week. In some places, a 1-hour lecture precedes the lab session. In others, experiments stretch over two weeks.

Some manuals seem to try to scaffold students from this mode of highly-structured inquiry (where direct instructions are given) toward guided inquiry (where, instead, students are given goals and broad guidance). I wasn’t able to find any lab programs that aim for open inquiry.

4. How do AP, IB, and Cambridge A-Level lab expectations compare?

For all three of these programs, labs are expected to occupy about 20% of the instructional time. In the AP, IB and A-Level classes, labs are explicitly expected to build toward independent student work (ie: open inquiry). AP labwork is not assessed directly, while IB students submit a lab report for external assessment, and A-Level exams include a substantial practical component.

It seems to me that colleges and universities have substantially lower expectations for student performance in labs than is found in the AP, IB, and Cambridge A-Level courses.