Designing a remote professional development course to support teacher customization in science

Abstract

This study reports on the design, implementation, and impact of a remote professional development (PD) course for secondary school teachers who were transitioning to remote instruction during the COVID-19 pandemic. We designed technology innovations to strengthen the previously successful in-person course. The innovations support teachers to customize an instructional unit by setting and revising goals based on evidence from their students’ prior work on the unit. A Curriculum Visualizer makes the pedagogy of the unit visible and guides planning for customization. Carefully curated small group activities using Zoom breakout rooms ensure that each teacher could share their thoughts, ideas, and impressions with other teachers. Participants were 23 science teachers from 12 different schools in a western U.S. state. We developed rubrics to code customization goals, plans, and moves using bottom-up methods and iterative refinement. Reflections on student work and use of the Curriculum Visualizer enabled teachers to set and refine customization goals and make evidence-based and pedagogy-aligned customization decisions that enhanced the interactive learning opportunities for their students. Our results reinforce the C-b model proposed by Sailer et al. (this issue) by illustrating the value of using technology to support collaborative, interactive PD activities.

During the COVID-19 pandemic teachers around the world customized instruction for students learning at home. This study reports on the design, implementation, and impact of a remote professional development (PD) course for secondary school teachers who were transitioning to remote instruction. Informed by prior research and an in-person course that used digital resources, the remote PD featured a personalized, interactive, collaborative sequence of activities to engage teachers in designing evidence-based and pedagogically informed curriculum customizations. We investigate the impact of the remote PD course designed to take advantage of technology, including review of student work and use of a Curriculum Visualizer, to support participants to customize a web-based curriculum unit, and the reactions of the participants to the design of the remote PD course.

1. Rationale

1.1. Customization of instruction

Many factors motivate teachers to customize their curriculum. They often align new materials with their existing classroom practice, even when the new materials implement a different pedagogical approach (Matuk, Linn, & Eylon, 2015; Remillard, 2005; Schneider, Krajcik, & Blumenfeld, 2005). They often increase the relevance of materials for their specific student population (Squire, MaKinster, Barnett, Luehmann, & Barab, 2003). In the US, the increasingly diverse K-12 population (Digital Promise Global, 2016) has motivated teachers to make curriculum adaptations that welcome students with unique prior knowledge and language ability. Moreover, new curriculum standards such as the Next Generation Science Standards (NGSS Lead States, 2013) in the US, have necessitated adaptations to meet new performance expectations. Starting in 2020, the COVID-19 pandemic challenged instructors worldwide and across education levels to quickly shift to all online instruction, requiring thoughtful curriculum adaptation.

1.2. Professional development for curriculum customization

Successful PD programs support teachers to engage in cycles of enacting a lesson, reflecting on student work, customizing the lesson, and teaching the refined version again (Fallik, Eylon, & Rosenfeld, 2008; Marx, Blumenfeld, Krajcik, & Soloway, 1998; Voogt et al., 2015). These courses typically start with a research-based curriculum (as opposed to creating from scratch), examine student work to determine where curriculum customizations are needed, and use an instructional framework such as Knowledge Integration to distinguish which customization designs to pursue (Gerard, Varma, Corliss, & Linn, 2011; Penuel & Gallagher, 2009). Without these supports, curriculum customization may undermine rather than strengthen students’ opportunity for inquiry (Davis, Palincsar, Smith, Arias, & Kademian, 2017; Drake & Sherin, 2006; Kerr, Marsh, Ikemoto, Darilek, & Barney, 2006).

A customization cycle is a central mechanism in research-based PD programs such as lesson study (Lewis, 2016) and teacher action research (Power & Hubbard, 1999). Customization using a lesson study approach for example has improved teachers’ ability to learn from student ideas (Bruce, Flynn, & Bennett, 2016; Lewis & Perry, 2015), monitor student learning (Van Zee & Minstrell, 1997), and select student work for class discussion that emphasizes reasoning over correctness (Pang, 2016). Further, cycles of enactment, reflection, and reenactment are typical in video-based PD models where teachers use video recordings of themselves teaching a lesson to reflect on their practice, to analyze their teaching effectiveness and to plan new and more effective teaching strategies (Chen, Chan, Chan, Clarke, & Resnick, 2020).

Teachers possess localized knowledge of their students and understand the constraints of their context, which enables them to productively innovate while benefiting from supportive PD (Randi & Corno, 1997; Squire et al., 2003). Using evidence from practice to prompt teacher reflection has shown to focus attention on students’ ways of reasoning about the discipline, and the specific instructional design strategies that impact student thinking (Lewis, 2016; Little, 2003; Roth et al., 2011). Engaging teachers in curriculum customization enables a flexibly adaptive approach to PD (Trautmann & MaKinster, 2010) that supports teachers to modify existing materials to meet the needs of their students and suit their classroom context.

1.3. The customization PD model

The PD model used in this research features a cycle involving both customization and visualization of the curriculum. It has been refined over ten years, is grounded in the Knowledge Integration (KI) framework, and incorporates web-based affordances (Gerard et al., 2010, 2011). The PD model supports teachers to customize web-based science units to improve support for developing coherent understanding and fostering learner agency. Teachers analyze the impact of their prior teaching and curriculum design decisions using logged student work; use the KI framework to plan revisions; teach revised units in their classrooms and reflect on progress. Specifically, teachers examine logged student responses to selected embedded assessments to determine student ideas to further build on or connect to, and gaps or inaccuracies in student understanding to address (Gerard et al., 2011; Tissenbaum et al., 2016). The KI framework helps learners build on their often fragmented and even contradictory prior ideas to develop a coherent understanding of a science topic. Instruction engages learners in (a) generating or eliciting their initial ideas as a basis for KI; (b) using interactive experiments, models, or other activities to discover new ideas; (c) distinguishing among initial and newly discovered ideas to sort out coherent insights; and (d) reflecting on these distinctions to form a coherent explanation or solution (Linn & Eylon, 2011). Curriculum units that teachers customize during the customization cycle are designed to support students’ knowledge integration.

The PD course is designed to illustrate for teachers how KI principles guide curriculum design. The teachers use the KI framework to negotiate which customization designs to pursue and how to refine them by fore-fronting consideration of their teaching and learning goals (Penuel & Gallagher, 2009). Teachers reflect on their success in PD workshops following teaching customized versions of KI curriculum.

1.4. Role of technology in PD

Using technology to engage teachers in customizing instruction enables participants to develop digital skills and experience learning activities that can inspire designs for student activities. Research shows that instructors in higher education benefit from learning how instructional technology works and how to use it in transformative rather than transmissive ways (Chien, Chang, & Chang, 2016; Reich, 2020; Sailer, Schultz-Pernice, & Fischer, this issue; Tamim, Bernard, Borokhovski, Abrami, & Schmid, 2011; Wekerle, Daumiller, & Kollar, 2020). The cycle of customization, implementation, and reflection has successfully supported instructors to adopt technology-enhanced inquiry curriculum (Kafyulilo, Fisser, & Voogt, 2015).

Designing PD to integrate technology and pedagogy involves learning how to teach with the technology as opposed to just learning how to use the technology (Lawless & Pellegrino, 2007). PD that features promising uses of technology, can empower teachers to articulate their prior ideas about technology and identify their strengths; they can then build on their strengths while also envisioning how the technology plays out in their classrooms (Wilkerson, Andrews, Shaban, Laina, & Gravel, 2016).

A small body of research has developed and tested tools that enable teachers to design and adapt web-based curriculum. The goal of these tools includes broadening teacher knowledge about the uses of technology for instruction (Laurillard et al., 2013; Laurillard & Ljubojevic, 2011), supporting teachers to articulate and refine their often tacit design knowledge, and improving curriculum design by incorporating teacher knowledge of their students and the topic.

For example, edCrumble is an online learning design platform that allows teachers to create web-based learning designs (LDs) with the support of data analytics. It features a visualization that displays, in real-time, information about the learning design, such as the type of activities in the existing design, amount of time allocated to different activities, and sequencing of activities, so that users can improve their design based on evidence from users (Albó & Hernández-Leo, 2018). edCrumble was tested in a higher education programming course, where teaching assistants design labs and homework activities on a weekly basis. When comparing users who designed with the visualization available versus those who used the system without the visualization, the authors found that the final lesson designs were less likely to include concepts that had not yet been taught, when users had the visualization. In effect, the visualization helped the users to sequence the content. Further, all instructors said they preferred the design interface with the visualization (Albó, Barria-Pineda, Brusilovsky, & Hernández-Leo, 2019).

Similarly, the Web-based Inquiry Science Environment (WISE, https://wise.berkeley.edu/) has been used for both student facing curriculum and professional development (Gerard et al., 2011; Gerard, Bradford, Lim-Breitbart, Wiley, & Linn, 2019). For students it has been used to design and deliver research-based and interactive online science curriculum (Linn, Clark, & Slotta, 2003). It offers a broad and accessible set of authoring tools teachers can utilize to design or customize these online units. Past research demonstrated that over time, teachers unfamiliar with the technology at the beginning, become more literate and, importantly, more confident authoring online curriculum themselves (Bradford et al., in press). WISE offers an authoring view that makes it easy to delete or add entire lessons or specific activities. Within one lesson, teachers can choose to add components (pre-designed activities) such as multiple-choice question, open response, text editor, or more complex activities such as a drawing tool, a concept-map tool, an annotation tool, a discussion board, as well as add or link videos, animation, or models. Each component can be customized. For example, teachers can edit the instructions given, import student responses from prior activities in the unit, show student work in the annotation tool, or have students share their concept maps to be critiqued by a peer. As a result, WISE is an ideal platform for both student learning and as a support for PD.

In summary, fledgling efforts to incorporate technology into professional development have yielded promising results. We report on efforts to capitalize on this progress in the design of a PD course for remote learning.

1.5. Research questions

With the transition to remote instruction, we designed a remote version of an in-person course and tested it to answer the following questions: (a) What is the impact of a remote PD course featuring newly designed technologies for review of student work and for a Curriculum Visualizer, to support participants to customize a web-based curriculum unit? and (b) How do the participants respond to the design of the remote instruction?

2. Design of the remote PD course

The remote PD course is based on previously successful in-person workshops, features the customization cycle, and is designed using the WISE technology. We sought to exploit technology to amplify promising aspects of the in-person course and add insights about effective PD in general.

The design of the PD course follows KI by eliciting teachers' customization (or instructional) goals based on joint review of the unit and a group brainstorm; guiding teachers to discover new ideas by interacting with multiple representations of their students’ ideas from their unit, and exchanging teaching experiences with colleagues from other schools who have taught the same unit; encouraging teachers to distinguish among what worked and what was challenging for students; and using a Curriculum Visualizer that captures the activity structures in the curriculum to support teachers to reflect on the relationship between teacher goals and student work to plan customization steps.

The remote PD course integrates all activities on the WISE platform so teachers can link their students' ideas to the pedagogies underlying the curriculum design, develop curricular strategies for attending to students’ ideas, and take ownership of the curriculum while learning remotely. The remote PD takes advantage of branching, automated scoring, interactive activity structures, importing of work from one step to another, and online planning tools to create a personally relevant user experience that fosters teacher noticing and evidence-based and pedagogy-grounded customization decisions.

In-person workshops of the past provided ample opportunity for teacher collaboration within as well as across school, grade level, and science domain collaboration (Gerard et al., 2011). Leveraging teacher knowledge about their school's or district's specific requirements and expectations, about the student population they teach, and their experience teaching specific topics in the past is a common feature in other PD models that are designed to support teachers as curriculum designers (Judson & Lawson, 2007; Lang, Drake, & Olson, 2006). Teachers sharing and discussing their customization ideas elicits the reasoning behind those ideas (Remillard, 2005), which can guide decision making and lead to reflection about why and to what end or purpose one customizes. A challenge for the remote PD course design was to maintain this key feature of PD and to offer a similar collaborative experience to teachers as the in-person workshop did in the past. In our design of the remote course, we designed breakout rooms to ensure teachers could exchange views with all course participants, have focused discussions with fellow teachers from the same school or teachers who were customizing the same instructional unit, and exchange ideas about teaching dilemmas (e.g., hands-on experiments during distance teaching) in general with a small group of other teachers, preferably teachers from other schools, who taught different units or grade levels.

Another essential feature of the customization cycle PD model is the collaboration between teachers and researchers. Researchers, as experts in instructional design and particularly in the technology used to customize, can act as mentors during the PD experience. As teachers customize and enact curriculum that is developed by researchers, it seems advantageous for researchers to share their intentions behind certain curriculum designs (Ball & Cohen, 1996; Davis & Varma, 2008). Further, when teachers are aware of design intentions, they can provide more detailed feedback as to why certain activities, in spite of their theoretically effective design, might not work in a classroom and both parties can work together to redesign. Such collaborations may lead to increased use of learning principles in classrooms. Moreover, as experts in the technology, collaboration between researchers and teachers is crucial to get new-to-technology teachers interested, help teachers overcome fears related to using technology, and guide their increasingly independent use of technology. Past research from in-person PD workshops has shown that especially when teachers author curriculum, availability of researchers and technology designers motivates and maintains teachers authoring and increases their confidence to be able to author (Bradford et al., in press). A challenge for the remote PD course design was to maintain the exchange between teachers, researchers, and technology developers. In our design of the remote course, we ensured that each teacher or teacher team was paired with the researcher who is lead researcher for the unit they were customizing. In add In addition, technology developers moved between breakout rooms to respond to questions.

In sum, we designed a remote PD course building on successful aspects of the in-person workshops using the customization cycle PD model (Gerard et al., 2019). In addition, we designed the remote PD to maintain success factors that were valuable in other PD models focused on supporting curriculum design (Ball & Cohen, 1996; Davis & Varma, 2008; Judson & Lawson, 2007; Lang et al., 2006; Penuel & Gallagher, 2009). For example, we provided opportunities for teachers to interact with fellow teachers as well as researchers and technology experts. We also exploited technology to strengthen the visualization tools supporting pedagogy informed customization decisions.

2.1. Eliciting initial customization goals

Teachers were asked to record their initial customization goal and plan at the beginning of Day 1 of the PD course. Initial customization goals are motivated by teachers’ past experience of teaching the unit as well as their general goals for science instruction such as making content more relevant or providing opportunities for self-directed learning. Given the special circumstances of teaching during shelter-in-place orders, we surveyed teachers prior to the workshop to identify common challenges and successes of distance teaching and started with a whole course discussion about distance teaching. After a whole course exchange about the results from the survey, we split into breakout rooms and teachers shared their strategies for addressing distance teaching challenges.

Next, teachers moved on to the individualized steps in the PD course unit. The first of the individualized steps was to record the initial customization goal and plan. To guide teachers setting their goals, we provided links to the curriculum unit they chose to customize, to the milestone item for the student work they would review, and the NGSS performance expectations the unit covers. Based on the distance teaching discussion and the review of the curriculum, teachers then recorded their initial goal (“Describe your customization goal” and “Describe your initial plan how to address this goal through customization of the chosen unit.“).

2.2. Discovering evidence for customization: reviewing student work

At in-person workshops, teachers sit with other teachers who are focused on customizing the same curriculum unit, open their WISE Teacher Tools and the logged student work for a selected embedded assessment, review the student work using a KI rubric provided by the researchers, and jointly discuss their insights aloud. The remote PD supported review of student work using the WISE branching technology: For each participating teacher we designed an individualized set of activities that included their own students’ work from a past classroom use of WISE. For teachers who did not use the unit they were customizing in the past, we selected student work from a school with comparable demographics.

2.3. Rationale for selecting student work

Each WISE unit integrates so-called milestone items that provide insight into how students think about target phenomena. These embedded assessments are placed after students interact with and explore the materials and can only be fully answered by connecting multiple ideas students gathered from their interaction with the dynamic models or explorations in the unit. Hence, these items provide insight into how students express single ideas as well as how well they connect relevant ideas. Most items we selected are also aligned with NGSS performance expectations (NGSS Lead States, 2013), providing teachers with insight into whether students meet these expectations. In addition, when possible, we selected an assessment that included automated knowledge integration scoring, so that teachers could compare their assessment of student thinking with the computer's Natural Language Processing (NLP) algorithm (Liu, Rios, Heilman, Gerard, & Linn, 2016; Riordan, Bichler, Bradford, et al., 2020; Riordan, Wiley, King-Chen, et al., 2020).

We designed two activities for each teacher to engage with student work to discover evidence for customization. In the first activity, teachers sorted individual students' responses according to the KI level the teacher assigned to each response. In the second activity, teachers reviewed a report that aggregated their class’ level of understanding.

2.4. Reviewing individual students’ work

In past in-person workshops, teachers logged in to their WISE accounts and opened the Teacher Tools from a past use of the unit they wanted to customize. They were guided to a milestone item or chose an item themselves. Next, teachers browsed students' logged responses for that item. Often, they discussed with teachers sitting next to them while discovering evidence. For the remote version, we selected 10–12 responses individual students submitted in a past use of the unit and copied them into one of the three individualized steps in the remote PD course unit. This way, teachers did not have to switch between the PD course unit and their WISE teacher account to review student work. Also, this design enabled us to choose responses that covered a broad range of ideas, giving the teacher insight into inaccurate, incomplete, or elaborated expression of target ideas. The remote activity was designed so that teachers first reviewed the assessment item and the associated KI rubric for the item. Next, the teacher reviewed student responses that illustrated the range of student understanding across the KI rubric, and common ideas at each score level. Teachers evaluated the responses, sorting them into buckets, each of which represented a score level on the KI rubric. After sorting, the teacher compared their scores to scores assigned by an expert researcher for that item. To support comparison, we showed the student response, its KI score and provided a text box with the reasoning for why this KI score applied. With this remote version of the reviewing individual students' work activity, we guided teachers' analysis of student work, supporting them to distinguish among students’ ideas, selecting those that students could build on, and identifying ideas needing further attention in instruction. We designed the comparison activity so that teachers could contrast their interpretation of student work with that of an expert (for that item) - something that in the in-person workshop would occur naturally when researchers sat with teachers or circled the room and spontaneously started to discuss student responses. This activity had the added benefit of introducing the KI rubric along with justifications for assigning score levels.

2.5. Reviewing class-level understanding

In the second activity to discover evidence for customizing, teachers reviewed a report that aggregated the level of understanding for their class. For certain milestone items, WISE produces an automated report to help teachers interpret their students' understanding (Wiley et al., 2020). The report consists of a histogram of student scores to capture the distribution of levels of student understanding across their class (Fig. 1 ). In past PD workshops, this feature was in development and only introduced to teachers. For the remote course, many of the units teachers chose to customize already included a milestone item with an associated report. We used a screenshot from the report that was generated for the teacher's past run to guide the teachers' reflections on the ideas that students most likely have and the ideas that students most likely need further help developing. We then guided this reflection so that teachers reflected on how they could customize the unit for the next classroom use to build on the ideas students hold while also guiding them to develop additional productive ideas.

Fig. 1.

Note. Teachers engaged in two different activities to reflect on their students' work. First, they sorted individual students' responses into KI levels (A) and then they reviewed a report that gave them an overview of their students’ average understanding (B).

2.6. Revising customization goals

After each reflection on student work in the PD course, teachers were asked to reflect on what they learned about their students' ideas. After the first activity (sorting individual student's responses into KI levels and comparing the results with a researcher version), teachers were asked to revise their goal (“How might you adjust or add to your customization goal based on this information?“) and plan (“Any new ideas to add to your approach to customize the unit to meet your goal?“). After the second activity (reviewing an aggregate summary of their class' level of understanding), they were again asked to revise their goal (“Refine your customization goal based on this new information”) and plan (“Refine your customization plan based on this new information“).

As kick-off activity for Day 2, teachers revisited the last version of the customization goal they recorded on Day 1 before moving on to the Curriculum Visualizer to plan customizations.

2.7. The curriculum visualizer: Connecting evidence and pedagogical theory

During the PD course we supported collaboration between researchers who facilitated the design of the curriculum unit and the teachers who were customizing the unit for their students and would later enact it. This allowed the curriculum developers to share their design intentions, which are not always obvious (Davis & Varma, 2008). Further, collaboration with a researcher who has participated in the development of the unit helps teachers new to curriculum design to design pedagogically informed activities (Lin & Fishman, 2004).

For past in-person workshops, a paper-based Curriculum Visualizer tool was used to make the curriculum developers' design intentions visible. Each lesson in the unit was represented by a notecard and the sequence of activities in each lesson were represented by the sequence of post-it notes on that notecard. The color of the post-it note corresponded to the pedagogical intentions the designer had for that activity (e.g., a pink post-it note indicated the activity was intended to elicit students’ prior ideas). By making visible the pedagogy underlying the curriculum design teachers were supported to connect the evidence discovered in student work with pedagogical and instructional design theory. Additionally, this Curriculum Visualizer tool allows researchers and teachers to jointly reflect on the purpose and effectiveness of each activity in the unit, identify gaps and redundancies, and make deliberate pedagogically driven customization decisions.

To replace the paper-based tool used in in-person workshops, we designed an online Curriculum Visualizer tool to make the KI framework underlying the design of each activity in the WISE science unit visible (Fig. 2 ). The online version of the Curriculum Visualizer allowed teachers to move activities within the unit, move entire lessons, remove or add activities and lessons. To support teachers' pedagogical decision making when adding activities, colored slides were added to the slide template. Thus, when adding a new activity (by inserting a new slide), teachers chose which KI process they wanted to support with the new activity. Teachers edited the slides to record customization decisions or make notes of changes they planned to make. While working on specific steps, teachers used the one slide view, they were however able to zoom out and see an overview of their unit in slide form (Fig. 2). This overview revealed whether a KI process was over or underrepresented and showed whether the design supported all processes equally. Both the paper-based and online version of the Visualizer guide teachers’ customization planning, supporting them similarly in designing coherent curriculum when building on the evidence discovered from their classroom.

Fig. 2.

Curriculum Visualizer example: Activity view. Note. The activity view shows the customization plan for one activity. Related steps are shown as slides in a column. The Grid view shows all the activities as color-coded slides.

2.8. Course sequence

The course lasted 3 half days and each day office hours were offered in the afternoon for teachers who wanted to continue working on their customizations. The remote web-based PD course was available before, during, and after online sessions. The synchronous activities involving the whole group, small groups, and individual work time were facilitated using Zoom.

3. Method

3.1. Participants

Science teachers from 12 different schools across a western US state registered and participated in the course (23 total, 19 middle school and four high school). All teachers were either actively collaborating with the research team or had used the open educational platform WISE in the past. Most teachers (18) had participated in professional development with the research team in the past and five were participating for the first time.

3.2. Data sources

The WISE platform logged all teacher interactions and responses in the online PD unit. We analyzed the logged teacher responses to prompts in the unit about their customization goals and plans, their reflections on student work, and their reflections on remote PD. We also analyzed the teacher customization decisions which were logged in the online Curriculum Visualizer.

3.3. Analytic approach: Coding and rubrics

3.3.1. Coding initial customization goal and plan

We used emergent coding (Chi, Bassok, Lewis, Reimann, & Glaser, 1989) to develop a bottom-up rubric for the customization goals and plans. One researcher read through all the responses and noted the emerging themes, the themes were discussed and refined, resulting in the customization goals rubric (Table 1 ). We identified 6 distinct goals across teachers and coded the presence or absence for each goal for each teacher.

Table 1.

Customization goals rubric.

Code	Description	Category
1	Usability	Customization goals pertaining to • fixing broken links, • videos that do not work, • automated feedback that is incorrect,
2	Increasing Accessibility	Customization goals pertaining to • increasing accessibility for students with special needs (language) • adding word banks • rephrasing, replacing terms/expressions and/or prompts
3	Refining Content	Customization goals pertaining to • changing, correcting, adding content/resources • domain-specific goals
4	Monitoring Progress	Customization goals pertaining to • teacher monitoring of student learning progress, • teacher monitoring of student progress in the unit, • or student monitoring their own learning
5	Distance Teaching	Customization goals pertaining to • making the unit work for distance teaching focus on content changes • making the unit work for distance teaching • focus on administration/orchestration
6	Guidance for Understanding	Customization goals pertaining to • supporting learners to investigate, process, and understand the content better • adding scaffolding activities

Notes. Rubric used to code teachers' customization goals in workshop unit, each teacher could mention multiple goals.

Teachers’ customizations plans were coded as: (1) plan outlines detailed steps (activities) that will be taken to achieve goal (e.g., add discussion board, add drawing activity); (2) plan outlines general steps that will be taken to achieve goal/steps the teacher will take to identify the actual customization steps needed to achieve goal (e.g., go through the unit to find the steps to rework); (3) plan indicates the teacher is not sure how to achieve these goals yet; and (4) plan indicates that the teacher will learn in the workshop how to achieve these goals.

Using these two rubrics, two researchers coded the responses to the initial customization goal and plan prompts individually, compared the coding and then discussed the disagreement. No disagreement led to any changes of the categories. We report the frequency for each customization goal and plan.

3.3.2. Coding revision of customization goal and plan

We further coded whether teachers revised their initial goal or plan. We compared the initial response to the responses submitted after discovering evidence from reviewing individual student's work and to the response submitted after discovering evidence from reviewing class-level understanding. We compared the revision to the customization goal and plan version submitted on Day 2 when teachers revisited their goals.

3.3.3. Coding customization decisions

We analyzed the customization decisions teachers recorded using the online Curriculum Visualizer tool. We first segmented the data in the Curriculum Visualizer to isolate each specific customization recorded in the Visualizer. Two researchers segmented all data from one teacher together to define segmentation rules and defined a customization instance as:

• 1.Customization instances are descriptions of changes in the slide body or comment on a slide
• 2.A customization instance can be indicated by an added slide, a removed slide, or a slide that has been moved to a different place in the unit (refers to single slides and entire lessons respectively)
• 3.Semantics within a slide may indicate different instances of a customization (new paragraph, space, etc.)
• 4.Each customization instance refers to one specific goal from the teacher's perspective (i.e., the description refers to one action)
• 5.Description of an activity and a revised activity prompt are considered one customization instance (e.g., comment describes adding discussion forum and slide body outlines the exact prompt that is used for the discussion forum)

To test the reliability of the segmentation method, two researchers consequently segmented data for another teacher independently. Upper and lower bound proportional agreement was 93% and 88% (Strijbos, Martens, Prins, & Jochems, 2006). We resolved the observed disagreement for this teacher and subsequently one researcher segmented all slide sets for all teachers or teacher groups.

The customization decision rubric (Table 2 ) was developed bottom-up by investigating themes in the customization instances. One researcher read all the customizations teachers noted in the Visualizer and categorized them into themes. These themes were discussed and refined to come up with distinct categories with another researcher. Next, two researchers coded each segment for one teacher together to test interpretation and application of the customization decision rubric.

Table 2.

Customization decision rubric.

Code		Description	Category
0/1	Navigation guides	• Adding stop signs • Adding constraints • Locking lesson
0/1	Instructional videos	Videos explaining content
0/1	How To videos	• Videos explaining navigation • Videos explaining what to do (group work, etc.)
0/1	Adding outside resources/Additional evidence\2	• Linking outside OERs (quizzes, simulations, etc.) • Connecting classroom practice/resources to WISE • Increasing personal relevance with adding resources ! Excludes instructional videos
0/1	Adding new WISE activities	Adding activities already existing in WISE; refers to new activities within steps or new steps (excludes adding a whole lesson or activity sequence) • Adding discussion • Adding drawing • Using WISE activity structure • Etc.
0/1	(Re)moving content	• Moving activities to different place in unit • Taking out activities or lesson • Taking out text or other materials
0/1	Adding knowledge checks	• Adding MC items • Adding reflection questions • Adding auto-scored items
0/1	Creating challenges/adding SDL explorations	• Adding additional activities for specific students • Adding self-directed learning opportunities • Adding Bonus Zone or entire lessons or activity sequences
0/1	Revising prompts/adding sentence starters	• Revision of prompts • Adding sentence starters • Replacing words/simplifying language
0/1	Chunking/sequencing	• Breaking longer activities into smaller steps • Combining separate activities into one
0/1	Modifying existing activities	Making changes to an existing activity in WISE; includes multiple changes such as revising prompt, changing picture, modifying feedback, shorten text, etc.
0/1	Clarifying instructional goals	Outlining goal for the unit, activity, or step (can be teacher externalization of what their instructional goal is or can be revised/added goal description for students in unit)

Note. Each customization instance was coded with the one category that applied (1), the same customization instance received 0 for all other categories.

The rubric was slightly refined to clarify categories and then both researchers coded all segments for another teacher independently. Cohen's κ was calculated for each category in the customization decision rubric to test the rubrics reliability. Since Cohens' κ was either 1 or 0 for most codes it was not a useful indicator of reliability (e.g., Cicchetti & Feinstein, 1990), but showed that most categories were clear in their meaning. For three categories Cohen's κ was 0.16, 0.57, 0.73. All disagreements involved a category that was missing. The problem was resolved by adding a category fitting these uncategorizable customization instances. Subsequently, one researcher coded all customizations instances for all teachers or teacher teams. Then, a second researcher looked at the coded version of all customization instances for all teachers and marked where they disagreed. The disagreements were resolved in discussion. In total (across all teachers), there were 142 customization instances and disagreements on seven customization instances.

3.3.4. Coding remote PD reflections

Teachers' reflections on the online facilitation were assessed using logged teacher work in the course unit. At the end of Day 1 and Day 2, teachers responded to these three questions: (1) Suggestions for tomorrow's agenda or ways to improve remote facilitation, (2) This is the first workshop we have facilitated at distance. How did each of these facilitation tools work for you? Zoom whole group, Zoom breakout rooms, Interacting with the WISE workshop unit, and (3) Describe your engagement in the remote workshop. Were you, and in what ways, able to contribute your own ideas and build on colleagues' ideas? The end-of-day remote learning reflections were investigated after Day 1 and 2 to inform remote strategies of the following day as well as to rapidly refine the remote workshop design based on participants' needs. At the end of Day 3 teachers were asked, “Please share any other reflections or feedback you have from the workshop”.

We analyzed responses to the remote PD reflection items by first reading all responses and identifying which aspects of the workshop teachers mentioned and whether these aspects were described as an effective feature or a challenge to remote PD. Based on the identified themes, we developed a coding rubric to quantify the aspects that were most commonly mentioned as effective or as a challenge (Table 3 ). One researcher coded the data by checking whether any of the identified features was mentioned as effective or as a challenge in any of the responses to all seven questions for one teacher. For example, the same category could be coded for one teacher as a result of this teacher's response to a question on Day one and for another teacher as a result of a response to a question on Day 2. A second researcher independently coded all data in the same way. Disagreements were resolved in discussion. The discussion revealed that a category was missing, and it was added.

Table 3.

Online PD reflections rubric.

Code	Description	Category
0/1	Zoom breakout rooms	Response describes Zoom breakout rooms as effective or as challenge
0/1	Whole group session	Response describes whole group sessions as effective or as challenge
0/1	Balance	Response describes the balance of whole group, small group, individual work as effective or as challenge
0/1	WISE workshop unit	Response described the WISE workshop unit as effective feature or challenge
0/1	Remote – In-Person comparison	Response mentions a comparison of in-person and remote workshop including positive or negative aspect of either format (as effective feature or challenge)
0/1	Teacher-Researcher Collaboration	Response describes the teacher-researcher collaboration (in small group) as an effective feature/challenge
0/1	Interactive nature	Response indicates that teacher contributed their own ideas or built on ideas from others/found it challenging to do so
0/1	Individual work time	Response indicates that teacher appreciated time allocated to working individually or on customizing/authoring/felt there was not enough time to work individually or on customizing/authoring

Note. We coded whether any of these features were mentioned as effective or as a challenge in a teacher's response to one of the seven remote PD reflection questions. We coded 1 if feature was mentioned as effective or as a challenge and 0 if it was absent.

4. Results and discussion

We designed a remote PD course featuring newly designed technologies to guide each teacher to use evidence from their students’ work to customize a web-based science unit that they planned to teach in the upcoming school year. We aligned the online activities and tools with the knowledge integration framework and promising features of PD models (Ball & Cohen, 1996; Davis & Varma, 2008; Gerard et al., 2010, 2011; Judson & Lawson, 2007; Lang et al., 2006; Penuel & Gallagher, 2009; Remillard, 2005). We report on (a) the impact of a remote PD course featuring newly designed technologies for review of student work and for a Curriculum Visualizer, to support participants to customize a web-based curriculum unit, and (b) how the participants respond to the design of the remote instruction.

4.1. Impact of the remote PD

We analyzed the goals teachers generated and the ways they revised them while reviewing student work. We studied the kinds of customizations teachers planned and how they used the Customizer to integrate their customizations to align with the pedagogy of the unit.

4.1.1. Initial customization goals

Most teachers (21 of 23) generated a customization goal. The most common goal was increasing accessibility and refining content, followed closely by distance teaching and monitoring progress (see Fig. 3 ).

Fig. 3.

Frequency of initial customization goals for N = 23 teachers. Note. Six distinct customization goals were identified in the data. Teachers could mention multiple goals. Twenty-one out of 23 mentioned at least one goal.

Most teachers (21 of 23) recorded an initial plan to reach the customization goal they generated. Almost half of the teachers planned to modify the WISE technology to reach their customization goal, while the other half were eager to gather ideas in the workshop.

4.1.2. Review of student work and revision of customization goals

Almost half (11) of the teachers revised their original customization goal and plan after discovering evidence from student work. Out of the 11 teachers who revised their original customization goal and plan (Fig. 4 ), five revised after the first activity (reviewing individual student work), five after the second activity (reviewing aggregated student work), and one revised on Day 2 when teachers revisited their goals. Eight teachers who revised their goal also revised their plan. Three teachers did not revise their goal but revised their plan. Thus, 14 of 21 teachers revised their goal or plan based on review of student work.

Fig. 4.

Customization goals and moves for N = 23 teachers. Note. ¹Teachers worked in teams of two. ²Teachers worked in teams of three. Three teachers did not use Curriculum Visualizer. Each dark grey shaded cell indicates that a teacher/teacher team expressed the respective customization goal/move. The frequency of each Customization Move is shown in the light grey shaded cell on the right side. The total number of Customization Moves per teacher/team in the light grey shaded cells on the bottom. Teachers participting for the first time in this PD: One teacher in team 4, two teachers in team 7, teacher 8 and 17.

Reviewing student work enabled teachers to notice student ideas and think about how to respond to them. For example, one teacher initially recorded their goal as: “For my customization, I might want to make some multiple choice or short answer questions that force students to identify 2–3 ideas and then prompt them to write their explanation to incorporate those responses into a scientific explanation.” After sorting student responses into KI levels, the teacher added the following to their goal:

“2.3 [step in the respective science unit] asks students to explain what insulators and conductors do - maybe that is a good place to have students draw a model that would show the molecular view of the material??? ^∗∗ thought: For customization, possibly include a question you now have or a flag button for help - especially for quiet kids or asynchronous work.”

Thus, after reviewing individual student work, this teacher noticed opportunities for students to deepen their understanding. Specifically, opportunities to elaborate on how insulators and conductors work. The teacher identified a drawing activity that could help students distinguish the observable from the molecular view of insulation. Initially, during the whole group discussion, this teacher planned to support students to engage with the concepts during distance learning by modifying multiple choice answer options and eliciting students’ thoughts in open responses. After the reflection on individual student work, the teacher was able to make the goal more specific and to emphasize distinguishing ideas rather than only recalling them. The review of student work led to a customization decision to include a drawing activity that helped students distinguish ideas, consistent with the KI pedagogical theory.

When reviewing individual student work, many teachers noticed specific student ideas. For example, one teacher said “A lot have the idea of equilibrium. Most have the idea of glass feeling colder. Some have ideas of rate of transfer affecting how something feels.” The same teacher said, “They need to further develop connections between their ideas.” after reviewing the report that included histograms displaying the average KI scores their class received. These examples show that both student work activities seem to be supporting the teacher to diagnose student needs. The review of aggregate student work is likely to reveal information on a general level: The ideas students seem to understand or struggle with and whether they do or do not yet link these ideas when explaining science phenomena. The individual work review is likely to reveal specific ways in which students express the ideas and what they are confused about.

4.1.3. Customization decisions recorded in the visualizer

Most teachers customizing the same unit worked in a group but on their own unit. Some teachers from the same school formed a team and worked on the same unit (two teams with three and two teams with two teachers). Two teachers who participated for the first time in the PD worked in a group but on their individual unit and three teachers who participated for the first time worked in a team. We analyzed the customization decisions teachers recorded in the online Curriculum Visualizer. For each teacher or teacher team, we recorded the number of customizations, and the frequency of each customization decision. We aligned these with the goals teachers expressed in the workshop unit (Fig. 4). The teams recorded between 2 and 21 planned customizations with a median of 10 and a mean of 10.1 customizations.

The Visualizer enabled most teachers to make multiple distinct customization decisions. Teachers made customizations throughout the unit - rather than just adding a new feature at the end - and used different customization strategies to address their goals. Both teachers who were new to using WISE, and those who had previously attended professional development, were able to formulate robust customizations.

Most teams took advantage of the technology to add new WISE activities, add outside resources, or modify existing activities. For example, teachers chose to add the “discussion forum”, a WISE activity that allows students to post their ideas and respond to ideas of their classmates in real time or asynchronously. Teachers also made use of auto-scored items either to provide students with feedback on their understanding or to monitor their progress.

Adding outside resources, teachers made use of the technology to augment and integrate previously offline activities. Using WISE authoring, the teachers broadened content to respond to their students' ideas by linking classroom resources or digitizing classroom practices. WISE supported a wide variety of interactive activities, unlike typical learning systems that only allow instructors to follow a predefined linear structure (Reich, 2020). For example, one teacher team modified an Annotator activity in which students critique a fictional peer's explanation by changing the featured student response to one from a student from their school. Further, the teacher team modified the critique activity to have two parts: first students were asked to critique an elaborate (high KI) response and then one that would receive a lower KI score. Aligned with the KI pedagogy, this customization supports students to distinguish between ideas expressed in the example response and their own ideas.

4.2. Teacher reactions to remote PD

Teachers responded to three reflection questions at the end of Day 1 and Day 2 and one reflection question at the end of Day 3. We analyzed responses to identify how teachers reacted to the remote PD and why (Fig. 5, Fig. 6 ).

Fig. 5.

Effective Features of the Remote PD Course (frequency for 23 teachers).

Fig. 6.

Challenges of the Remote PD Course (frequency for 23 teachers).

4.2.1. Zoom breakout rooms

Zoom breakout rooms were viewed positively by 16 of 23 teachers, most reporting that they enabled teachers to interact and be heard. Teachers reported: “The breakout rooms were really helpful today. I found it so much easier to do this with another person. I liked learning about the new tools in the WISE workshop unit”; “Yes, we shared ideas in the breakout sessions. It is always good to hear other teachers' perspectives”; “Great job on the facilitation. I had not done the breakout rooms before, nice feature”; “Great job with the remote workshop! I liked how small the groups were so everyone had a chance to share their voices”; and “All of these were great. I especially like seeing how break out rooms can be used in different ways. I haven't used them for students to work on individual work, so this was interesting.”

One teacher mentioned a Zoom breakout room challenge faced during remote instruction, saying, “The breakout rooms were chill, although it can be awkward when the groups are small and most don't have their camera on.” This suggests the need to structure the discussion in a breakout room. It also emphasizes the importance of video in remote instruction. Being able to read facial expressions is a crucial part of interacting in classrooms and especially important with remote instruction.

4.2.2. Whole group session

Some teachers (5 of 23) mentioned the whole group session as a positive and effective feature of the workshop. Most teachers shared that they were able to contribute in all formats including whole group sessions. One commented, “I contributed both during break-out and whole group discussion.” One teacher raised an equity concern, noting that white males dominated whole group discussions, remarking:

“I do want to mention the lack of equity of voice in certain conversations during this workshop. In some of the smaller breakout rooms it worked well to have people be able to speak up when they had questions as they worked on customizing units, but in other breakout rooms and in the whole group there was definitely a dominance of white male voices.”

This reveals the importance of establishing turn taking and monitoring practices for remote instruction just as for in-person discussion.

Another equity issue concerned access to sufficient bandwidth, technological equipment such as computers, tablets, or phones, as well as skill in using technology. While participants in this workshop had adequate technology, there may have been individuals who did not enroll due to lack of access. One weakness of the MOOC movement has been inability to reach learners who lack access to technology or connectivity (Reich, 2020). Instructors may focus on access and neglect design for e-participation of each learner.

Sailer et al., ‘s (this issue) model of technology-enhanced/online learning emphasizes the importance of shifting focus from equipment and technology skills to support for interactions between instructors and learners. They review literature suggesting that in higher education, teachers are challenged to design interactive learning experiences. Our results suggest the importance of designing interactive learning experiences that are effective for each learner. For example, ensuring that each voice is heard when there are group discussions.

4.2.3. Balance of whole group, small group, and individual work

Some teachers (4 of 23) mentioned that the whole group, Zoom breakout rooms, and individual work or time to work on customizations was well balanced. For example, one teacher said, “This was a well-orchestrated use of the tools available to us. I thought the amount of time we spent together versus in groups was good. I thought the types of activities you chose for each setting were totally appropriate.” Another teacher said, “The zoom format with breakout rooms worked very well. Time was well spent and balanced, pacing good, (…).” The 3 teachers who mentioned a challenge related to the balance all expressed needing more breaks, being physically challenged to sit for long and in front of a computer. For example, one teacher suggested, “Taking more short breaks rather than a longer break.” Another teacher said:

“We do need to take a 5 min stretch break every hour. I don't like to leave and miss anything, but cannot sit in a chair that long. We are teachers! Sitting in a chair on a screen that long was one of the hardest parts of distance teaching/learning.”

During in-class instruction, students change rooms between lessons and thus move for a few minutes every 45–90 min. Remote instruction needs to feature time for participants to get up, have snacks, or move around. Teachers reported that they lost attention when sitting for long periods of time. This is consistent with reports that asking students to sit for long periods of time may not be ideal (Cowgill et al., 2019; Norris et al., 2020; Patel et al., 2018).

4.2.4. Workshop unit

Five teachers volunteered that the workshop met their needs and provided space to reflect. For example, one teacher mentioned:

“I really like the WISE workshop unit. Being that I have only used WISE once, it was nice to go through as a student and experience that side. I was better able to understand the struggles my students were going through as they were navigating WISE for the first time.”

Such comments underscore the value of using the same system for teacher learning that the students use in the classroom. Teachers experience guidance for PD and also appreciate how students experience guidance in the online curriculum. This experience can help teachers support their students when using the unit in the classroom.

However, teachers noted that some questions felt repetitive. For example, a teacher commented, “I found the WISE workshop unit very helpful to give more specific prompts and directions throughout - some of the questions felt a bit repetitive though.” This repetition occurs because the workshop and the units emphasize revision. Since almost every teacher recorded their initial customization goal and only half of the teachers revised their goal, we conclude that careful consideration of the timing and amount of revision opportunities is essential to guide teachers, or students, through revisiting initial ideas. Too many opportunities may lead to revision fatigue, a counterproductive outcome as teachers, or students, may not recognize the value of revision when it feels like a nuisance.

4.2.5. Remote versus in-person PD

The remote PD was viewed favorably by most teachers. One teacher mentioned, “It was almost as good as being in-person. And if it was in-person I probably wouldn't be there.” This aligns with the enrollment figures. We expected about 12 teachers to participate, based on past experience and ended up with 23. The remote format allowed teachers with other commitments to flexibly join sessions, leave early or join later without interrupting the overall workshop flow. Nevertheless, two teachers preferred the in-person course over the remote version because of the long periods of sitting or working on the computer. One remarked, “This was a good substitute for the real thing, but definitely not a replacement. I had access to the researchers for questions and help but sitting so long in front of a screen is so tiring on the body and eyes.”

4.2.6. Teacher-researcher collaboration

Teachers (5 of 23) mentioned that researcher availability as a positive feature of the remote workshop. Teachers were thankful to have researchers available to quickly fix technology glitches, provide support using the authoring system, or find technological solutions for their goals. For example, teachers often had ideas of what they wanted to do but did not know how that would be possible using the available features in WISE. Teachers’ comments included: “I was pleased with the application/functionality of the facilitation tools. It was beneficial to have tech support/developers on hand when glitches arose”; “I enjoyed interacting with the knowledgeable researcher about the unit in question”; or “Nice to have a WISE rep in each group as we worked, it made the customizations easier.” Supporting teachers to customize with technology enabled them to participate in the design of interactive and student-centered learning activities (see Sailer et al., this issue).

4.2.7. Interactive nature

Overall, 19 out of 23 teachers reported being able to share their ideas and build on the ideas of their peers. It was our goal to design a remote course that maintained the in-person interactive nature of our typical courses and our results showed that the Zoom breakout rooms were valuable for this outcome. Teachers commented, “I liked that it was different people each time we broke out into groups. Allowed me to hear new voices and insight”; “I feel like I was easily able to share my own ideas and that my ideas were heard. It was very easy to engage in the small groups”; “I did contribute my own ideas. I found colleagues' thoughts about instruction for English learners to be very helpful, and something I did and will continue to build on. (…)“; or “I liked how we were put into groups and couldn't naturally gravitate to our coworkers. I heard from people I might not usually talk to at the workshop.”

We designed Zoom breakout rooms to build the community, pairing teachers who were working on the same unit for some sessions, intentionally pairing teachers from different schools for discussion sessions (e.g., on how distance teaching went for them). Compared to in-person collaboration, the assigned peer collaboration in breakouts built a broader community. We ensured that each small group was structured either by a course facilitator. Our results suggest that this is essential for making group work effective. We were fortunate to have sufficient staff that we could have a representative in each small group. It could be useful to experiment with scripts or roles for participants (e.g., Fischer et al., 2013).

4.2.8. Individual work time

Some teachers (4 of 23) wished for, “a lot of time to actually customize.” Many appreciated the additional office hours for teachers who were interested. Teachers appreciated that the workshop design incorporated time to individually work on planning, reflections, or customization. One remarked, “I needed the extended, quiet work time to make progress on my customization work.” Another commented, “Time is issue. Authoring and programming are not my forte'.” This underlines the challenge of adjusting activities for participants who work at varied rates or have familiarity with the authoring system. Teachers who are familiar with WISE are much faster at implementing and even planning customizations in comparison to teachers who are new to the technology and are getting familiar with its possibilities and functionalities.

In summary, teachers reacted positively to the remote workshop, with some preferring it to the previous in-person approach (primarily for logistical reasons such as traffic and parking). The assigned breakout rooms were effective in building community. Teachers had few complaints beyond the expected Zoom fatigue.

5. Conclusions

We developed and tested a remote PD course in response to the COVID-19 pandemic, adding technologies to strengthen the previous in-person course. The overall course goal remained the same: to guide teachers to use the KI constructivist pedagogy to customize web-based science units based on student work. We added technological innovations including an interactive Curriculum Visualizer to the WISE web-based PD instruction. We supported synchronous whole and small group collaborative activities remotely using Zoom for a three-day PD workshop. We implemented the customization cycle, goal setting, review of student work, and curriculum visualization activities from the in-person course. We designed new online activities where teachers could set goals and revise them after reviewing student work. We created an interactive activity where teachers reviewed student work, sorted their student work by KI categories, and compared their scores to those of an expert. We designed an online Curriculum Visualizer where teams of teachers who had implemented the same unit recorded planned customizations, aligned them with the pedagogy of the unit, and then used the recorded ideas to customize their unit.

Thus, we improved the goal setting activities, guided review of student work, and methods for recording customization plans. These revisions incorporated teacher reactions to the in-person workshop, designers’ observations of limitations of prior workshops, and the constraints imposed by remote learning.

We found that the goal setting activities were effective in motivating teacher teams to identify either specific or abstract customizations to improve the units. Interactive review of student work enabled most teams to notice student ideas, diagnose student difficulties, record possible customizations to improve student outcomes in the Visualizer, and refine their customization goals.

5.1. Teachers valued technology for reflecting on student work

After reflecting on and sorting students’ responses and reviewing reports visualizing the aggregate understanding in their class, teachers revised their goals for customization. Compared to activities using the WISE Teacher Tools during in-person workshops, the activities in the remote PD were more successful. For the remote PD, based on the logistic difficulties of using the teacher tools within the web-based unit, we pre-selected a range of student responses that covered understanding on all levels of KI and designed a sorting activity for each teacher. By seeking a way to reduce the logistic challenges, we also created a process for review of student work that better aligned with the KI pedagogy that the original one. We first asked teachers to predict scores for each student response. We then encouraged teachers to discover the nature of each score level by reviewing all the responses assigned a specific score. Next, we enabled teachers to distinguish their scores from those of an expert. Providing teachers with pre-selected student responses enabled us to focus the teachers on the various ways in which students express their thinking.

Our results show that the reflection on these pre-selected responses helped teachers to notice ideas students hold and ways that students get confused. These findings resonate with a study showing that using a visualization tool when investigating video recordings of their teaching, helped teachers to focus on aspects that supported their learning rather than on operating the recorder to find interesting teaching moments (Chen et al., 2020). Similarly, using a curated data set streamlined the process of discovering information relevant for customization in less time. Combining this with review of aggregated student work, teachers not only noticed which ideas students in their class understood or struggled with, but also how students in their class expressed these ideas. This enabled teachers to identify customization needs (the ideas that need more support) and in addition guided teachers in aligning customizations with the way their students think about these ideas.

5.2. Value of redesigned Curriculum Visualizer

We redesigned the Curriculum Visualizer that makes the pedagogy behind the science activities visible to teachers for remote PD. In comparison to the in-person paper-based version, the online Visualizer is flexible and scalable. It supports remote teams to plan their customizations. Using the Visualizer, teachers generated ideas for improving student learning outcomes and meeting their customization goals. Teams could build on the relevant learning processes they noticed in student work and discuss alternatives with other teachers. Thus, the course prepared teachers to pinpoint missed opportunities and revise the curriculum to strengthen students' engagement. The Visualizer supported teachers to consider the pedagogy when planning their customizations.

5.3. Teacher reactions to PD course

Redesign of the in-person workshop for remote participants resulted in a well-received, web-based unit. There were some drawbacks to the remote PD including the strain of using Zoom. Future remote instruction should increase the number of breaks and opportunities for participants to interact informally. There were also unanticipated benefits such as the flexibility afforded by limiting the whole group activities to 3 h a day and offering office hours each afternoon. For example, one teacher was absent when the Visualizer was introduced and came to office hours to learn about the tool. This teacher was then able to use the tool to reflect on her customization designs on her own and rejoin her team at the next meeting of the workshop. Another benefit of the online version of the Visualizer was that it facilitated peer review of the customization ideas of other teams. Thus, the remote version actually promoted teacher collaboration when designing instruction.

5.4. Overall impact

The format of balancing small groups, whole group discussions and time for individual work on customizations was well received and contributed to teachers’ feeling comfortable sharing their ideas with others. Features such as random allocation to breakout rooms led to collaborations that are unlikely in-person when teachers establish place-based work relationships.

Carefully curated breakout activities ensured that each teacher could share their thoughts, ideas, and impressions with other teachers who had taught the same or a similar unit. In addition, participating teachers could review plans with teachers from other schools and districts, benefitting from new perspectives.

These results reinforce the benefit of empowering teachers to use student work as evidence for design of instruction. Drawing on student work, as implemented here, could be adapted for improving online instruction for higher education students. Our results suggest that making the pedagogical framework underlying the instructional design visible enables teachers to use technology in transformative ways. Our results align with Sailer et al., ‘s model (this issue) which calls for more emphasis on creating interactive learning activities to improve student learning outcomes in remote or technology-enhanced instruction and emphasizes the value of using evidence from student work to do so.

Credit author statement

This is to confirm that all authors have equally contributed to the manuscript entitled “Designing a Remote Professional Development Course to Support Teacher Customization in Science” for the special issue Online Teaching and Learning in Higher Education in Computers in Human Behavior.