This is SOOOO cool!

Thanks Sarah!

Thank you for the questions! We are in the first year of the project and just starting to figure out the answers, while thinking a lot about possibilities like those you mention. By the end of the project we hope to be at that point of informing math teaching and learning, but at the moment we're tackling technical and definitional issues that will highly influence the types of conclusions we can draw. For example, defining what constitutes a unit of analysis (especially in terms of time) for the computational approaches we're developing will likely impact how well we can align to and expand on qualitative analyses.

One thing we're planning to analyze with respect to body position is the classroom activity type, as well as the transitions between activity types over time. Students in our data work individually, in groups, interact with the teacher, and do other activities that might be related to both body position and movement (like gestures). That might align closely to what you're curious about with students up and moving around, which we're very curious about as well!

Certainly! Our approach will complement live observation by avoiding this problem of missing things due to divided attention, though not necessarily providing the same kinds of insights that an observer might. We are also interested in advancing retrospective analysis of videos, even without live observers. Sometimes these datasets are so large that annotation is intractable, while automated analyses can perhaps find patterns across hundreds of hours of video/audio quickly, and direct annotators' attention to the most interesting areas of a large dataset.

We will be analyzing verbal interaction in a manner of speaking, though we don't anticipate it will be practical to automatically transcribe the text of what students are saying. Classroom audio is quite messy, in that there are often multiple speakers, imperfect audio data, and lots of other noises. However, we are analyzing aspects of speech that do not necessarily require transcription. For example, we are attempting to detect silence, voice activity of any kind, overlapping speech with multiple speakers, and then aligning those kinds of events with events in the video data. The audio and video can complement each other by providing context (e.g., how connected are speech and gestures in different topics or classroom activities?).

You're quite right; it is challenging! That is one of the technical problems we're working on, because currently there is no automatic distinction between teachers and students, nor a way to track the identity of a person (either teacher or student) over time. However, we think it is at least possible to somewhat solve this problem by measuring the amount and type of movement of individuals to see if there are clear distinctions between teachers and students.

Thanks! We're really excited to see where the project goes as well. We think that it could be a really good tool for education researchers to help them analyze more data and see additional patterns. 

Thanks Eric! Your project looks really interesting too. I'm very familiar with ENA and am interested to see how you will apply in your project.

I think collaboration looks a lot different via remote tools versus in person and it would be a really interesting extension of the work on our project to think about what would need to change in our analysis techniques to take that into account. 

Thanks! Yes, I'm guessing we would be able to detect the gorilla. :)

Our project is set up so that these tools and methods could be used by other researchers to investigate the same types of research questions that they would typically look at when analyzing video data qualitatively. This process of analysis usually takes a very long time and is very laborious and our hope is that our tools and methods would enable researchers to investigate a larger amount of classroom video data with the same types of lenses they normally would. It might also enable them to find additional patterns in their data that wouldn't have been accessible to them with the smaller amount they would typically analyze.

Thanks! We're encouraged to see that people could potentially see our work as useful in a variety of ways.

In our project, we are narrowly focused on developing tools for researchers, which will result in specific tools that may not be useful pedagogically. So tools that could be used for feedback to teachers (either in real time or offline) are not something we will develop in the project. We have recognized that since our tools will be publicly available and future work could be done to building on them to create pedagogical tools, we want to carefully consider the potential ethical issues beyond researchers.

In my research, I focus a lot on teacher learning. I think there could be potential in the tools being a foundation for similar tools specifically designed to support teacher learning, but extremely careful design work would be necessary to create tools that empower and support teachers in their own sensemaking rather than something that acts as surveillance or evaluation of teachers and students. We truly believe that the power in our tools comes from integrating human and computer analysis, and I think the same principle would apply to teacher learning. So it is less about tools themselves being able to provide feedback, and instead, supporting teachers to make sense of information coming from both computer and human analysis to develop feedback. For example, maybe tools could be developed that help a teacher keep track of how much time they spend interacting with each group in the classroom and what types of interactions they have. I know teachers who are curious about this and find it tedious to track. I wouldn't necessarily consider this feedback, but instead, data or information. The teacher could use that information, which could help them identify a specific interaction with a group they want to analyze more deeply, perhaps with colleagues or an instructional coach. I think this approach aligns well with other projects that are carefully designing tools for teachers around classroom data, and they've found it incredibly important to design ways to support teacher sensemaking, not just the feedback or data given to teachers. Two that come to mind are: Practical Measures, Routines, and Representations (https://www.pmr2.org/); and the EQUIP classroom observation tool (https://www.equip.ninja/).

I'm curious what kind of feedback or information you think would be useful for instructors to have that is difficult to get with existing tools?

Hi Michael, and thank you for your message.

Our expert on computer vision is Nigel Bosch. I know from some recent updates that identifying students who did not consent to participate is one practical/ethical challenge that we are working to address. Another concerns how to discern between the teacher and students when, for example, the teacher is behaving more "like a student" (such as when providing feedback in a small group) or when a student is behaving "like a teacher" (like when presenting to class!). Some other challenges are more organizational. My understanding is that the OpenPose tool generates a ton of data. But Nigel and his team are certainly up to that task. I will reach out to Nigel (or invite him to share more) for any additional details.

As an aside, I loved your video on music and coding - I'm excited to learn more about the project.

Thank you again for posting,

Josh (on behalf of the TCA^2 team)

To reiterate and agree with what Josh said, the most challenging part of the computer vision methods so far is figuring out who is who. That applies both over time (which person in one frame of video corresponds to a person in the previous frame?) and within each moment (who is a teacher, student, or person who we should not analyze?). Occasional tracking problems, or people moving in and out of frame, or behind other people, makes these all very tricky!

(Just popping in to say hi Mike! :))

Dear Isabel, thank you for asking this - cultural, racial, and gender dynamics are especially important when it comes to computational methods (for us and others carrying out similar kinds of research). We have started to think about how to incorporate these factors our project in at least two ways. First, we've made a commitment to not shy away from conversations about race, ethnicity, and gender - our own and that of the teachers and students who appear in the video we're using - and we're just beginning to have these conversations as we begin to analyze the data. Second, and related to the part of your question regarding the human researchers, the human-based analysis can provide chances for us to bring our priorities and goals to the (qualitative) analyses of these videos. But, that also means that we can bring biases to these analyses (and conversations about them), too, and so I think offering training about (or, at minimum, structured opportunities to discuss) race, ethnicity, and gender and how they apply to this work is a really great idea. 

Thank you for asking this. I for one would welcome hearing about any suggestions about what kinds of training would potentially be the most impactful - especially at this early stage in this work.

Thank you again!

Thanks so much for your thoughtful response! This all makes complete sense. This is definitely not my area of expertise (just something I'm interested in), so I'm not sure I have any wisdom to share.  I agree, though, that training would be a great opportunity to get people talking about how bias can be transferred to technology, and how to be aware of that.  I'm excited about the work you're doing and your care on all of this.

Hi Raffaella! Thanks for your comment. At this stage of the project, we are primarily focused on producing tools that will help researchers. I think it would be great to figure out ways to eventually use this to support STEM teacher education as well. 

Hi Ed, thanks so much for your comment. I agree that there are a lot of potential applications for this work and similar types of approaches. We really hope that this is just the beginning of a line of work that can have positive implications for many areas of educational research. 

Thanks Jeremy!

Thanks for your comments! Camera placement is indeed a super important element. We've just started comparisons between angles, so there are still a lot of unknowns. However, preliminary results are very encouraging. We're able to detect the position and pose of students/teachers with quite similar success rates across two camera angles in our tests so far. The biggest question is whether our findings will change notably with different camera angles, which is something we're planning to test.

I think we could look at student position as an indicator of attention and engagement. Similar projects have done so with some success. However, I don't think that will be a high priority for this project because we're more focused on starting with unsupervised discovery, where we find themes emergent from the data, which may or may not include student attention/engagement.

Hi Michael,

There is indeed some strong similarities between our lines of work. In fact, I think both projects use the same underlying computer vision technology, just for different purposes. It will be interesting to see how much our results align by the end of project as well.