The Intersection of Instructional Design and Program Evaluation
Proposal
The Intersection of Instructional Design and Program Evaluation
In elearning classrooms, the role of the teacher is both essential and fundamentally different from the roles they play in face-to-face learning. Teaching presence is the mechanism that bridges the transactional distance between instructor and student in a virtual classroom where direct instruction and facilitation of discourse are achieved through various forms of interaction (Afolabi, 2016; Arbaugh & Hwang, 2006). Ekmekci (2013) and Bowden (2012) presented arguments about the responsibilities instructors hold in setting academic expectations and ensuring that standards of scholarship are upheld. Instructors can create an academic climate that increases connectedness with students and expectations of scholarship by promoting a shared sense of teaching presence (Afolabi, 2016; Ekmekci, 2013). This sense of “being there” or “being together” is experienced in different ways in the online classroom and must be intentionally created for it to be perceived and felt (Lehman & Conceição, 2010). The importance of teaching presence and its influence on the learning process has increased in visibility (Arbaugh, 2013). However, as the online teaching environment has advanced, instructors have not always kept pace, showing a tendency to either rely too heavily on technology to form connections with their students or to revert to conventional practices that are more suitable for physical classrooms (Baran et al., 2013; Cho & Kim, 2013; March & Lee, 2016). Those who are new to online environments are challenged with finding suitable approaches to teaching in virtual classrooms because many did not learn that way themselves (Niess & Gillow-Wiles, 2013).
The Instructional Design Re-Invention Process
In summer 2014, 3 organizations launched Phase 1, an 8-week course for novice teachers that focused heavily on reading research and current publications on language acquisition theory designed to raise their awareness of the issues involved in teaching English Learners. Comments from participating New York and California novices (e.g., “I now teach EL students and do not have resources from the course to help me because I did not get a solid grasp of how to apply the strategies shown in the videos or laid out in the readings.”) raised important questions for program designers. Through the collection of these anecdotal data coupled with a rigorous comprehensive evaluation completed by a team based at New Visions for Public Schools (Trachtman, Koenigsberg, & Zheng, 2014), six key areas were identified in need of improvement. An online education consultant worked with one of the course instructors to design the following changes for implementation in Phase 2, the summer 2015 course.
From Groups to Partner Teams
The Phase 1 course had encouraged group work in an effort to build a sense of community; however, by changing the groups each week and providing relatively little timely feedback to group participants, the novice teachers were not able to build their desired and desirable connections with peers.
In response, in the Phase 2 course, instructors assigned 2-teacher partner teams to work together on a course-long project. Partner Teams remained together for the entire class and were encouraged to meet regularly by phone, Skype, or in person. In their required weekly reflections, novice teacher partners reported on their experiences, sharing confidentially with instructors their frustrations and achievements
From Weekly Assignments to a Single Over-Arching Project
The Phase 1 course required weekly written assignments focused on the readings and research materials assigned that week. The strategy seemed to emphasize learning theory and the acquisition of new knowledge through the preparation of written summaries.
In the revision, the Phase 2 course modeled a project-based learning approach with students assigned a series of tasks that built on each other, culminating in the development of a 3-Day unit or set of lessons that participants could use with their students in the fall.
From Discussion Questions to Real Discussions
The Phase 1 course prompted participants with weekly discussion questions that related to research read and videos watched during the content portion of the week. In their postings they often summarized what they had read; peer responses to posts did not generally extend or deepen the conversation. Further, few discussions moved past the initial response posts.
The Phase 2 course revised the discussion prompts to focus participants on their own personal experiences with the weekly topic. Instructors developed strategic “moves” for deepening discussions to facilitate the development of a dynamic learning community.
From Abstract to Actionable
The Phase 1 course focused on the theory of language acquisition leaving participants to infer how best to implement the theory in their own classrooms. They were invited to watch videos of other teachers teaching English Learners and evaluate the observed instruction. The video watching strategy reflected the course’s implementation during the summer when the novices were not able to observe actual teaching at schools.
The Phase 2 course required participants to dig past the theory by reviewing proven strategies and then deciding which strategies would work best for their specific students. By having novices “create” their own students at the beginning of the course, the project team believed the teachers would feel an important sense of responsibility and commitment to these students, and, vested in their success, would make curricular and instructional decisions based on their needs. While real students continued to remain out of sight, instructors had designed a process to mimic the approach used by experienced teachers to design effective instruction: begin by knowing your students well.
From Anonymity to Collaborative Engagement
The Phase 1 course provided few opportunities for structured contact between students and instructors other than the feedback exchanged on assignments and instructor posts on discussion boards; while some students reached out to their instructors via email, the online delivery of the course appeared to limit interaction and connectedness.
The Phase 2 course featured the same instructors working hard to be perceived as real people attentive to participants and their learning. The instructors posted self-made videos welcoming the participants and orienting them to the week’s course of study. They posted pictures, brief bios, and hosted a Fun Things discussion thread to facilitate connections. Through frequent postings that validated participants’ learning, instructors strove to deepen the discussions by pushing past the initial post, inviting participants to explore extended applications of the concepts emphasized that week.
From Final Evaluation to Formative Data Collection
The Phase 1 course asked instructors to collect weekly survey data but the data were not shared and discussed during the course. Instructors and the project team held 3 reflection conversations during the first summer, providing instructors with an opportunity to gain general insights into the students’ experiences. The final comprehensive evaluation, completed in December 2014, provided actionable feedback but results from this evaluation were not available until after the course had ended.
The Phase 2 course used weekly reflections submitted directly to the instructors as a way to check on participants’ engagement with the online medium, instructors’ pedagogical choices, and the content of the curriculum.
Findings: Curriculum Redesign Outcomes
Participant Engagement, Feelings of Efficacy, and Sense of Preparedness
- Course re-design resulted in more participant engagement, increased feelings of preparation related to engaging with peers, evaluating and reflecting on practice, choosing teaching strategies to meet different student needs.
- Teachers of color had more positive beliefs about their teaching efficacy.
The Intersection of Instructional Design and Program Evaluation
The Intersection of Instructional Design and Program Evaluation
In elearning classrooms, the role of the teacher is both essential and fundamentally different from the roles they play in face-to-face learning. Teaching presence is the mechanism that bridges the transactional distance between instructor and student in a virtual classroom where direct instruction and facilitation of discourse are achieved through various forms of interaction (Afolabi, 2016; Arbaugh & Hwang, 2006). Ekmekci (2013) and Bowden (2012) presented arguments about the responsibilities instructors hold in setting academic expectations and ensuring that standards of scholarship are upheld. Instructors can create an academic climate that increases connectedness with students and expectations of scholarship by promoting a shared sense of teaching presence (Afolabi, 2016; Ekmekci, 2013). This sense of “being there” or “being together” is experienced in different ways in the online classroom and must be intentionally created for it to be perceived and felt (Lehman & Conceição, 2010). The importance of teaching presence and its influence on the learning process has increased in visibility (Arbaugh, 2013). However, as the online teaching environment has advanced, instructors have not always kept pace, showing a tendency to either rely too heavily on technology to form connections with their students or to revert to conventional practices that are more suitable for physical classrooms (Baran et al., 2013; Cho & Kim, 2013; March & Lee, 2016). Those who are new to online environments are challenged with finding suitable approaches to teaching in virtual classrooms because many did not learn that way themselves (Niess & Gillow-Wiles, 2013).
The Instructional Design Re-Invention Process
In summer 2014, 3 organizations launched Phase 1, an 8-week course for novice teachers that focused heavily on reading research and current publications on language acquisition theory designed to raise their awareness of the issues involved in teaching English Learners. Comments from participating New York and California novices (e.g., “I now teach EL students and do not have resources from the course to help me because I did not get a solid grasp of how to apply the strategies shown in the videos or laid out in the readings.”) raised important questions for program designers. Through the collection of these anecdotal data coupled with a rigorous comprehensive evaluation completed by a team based at New Visions for Public Schools (Trachtman, Koenigsberg, & Zheng, 2014), six key areas were identified in need of improvement. An online education consultant worked with one of the course instructors to design the following changes for implementation in Phase 2, the summer 2015 course.
From Groups to Partner Teams
The Phase 1 course had encouraged group work in an effort to build a sense of community; however, by changing the groups each week and providing relatively little timely feedback to group participants, the novice teachers were not able to build their desired and desirable connections with peers.
In response, in the Phase 2 course, instructors assigned 2-teacher partner teams to work together on a course-long project. Partner Teams remained together for the entire class and were encouraged to meet regularly by phone, Skype, or in person. In their required weekly reflections, novice teacher partners reported on their experiences, sharing confidentially with instructors their frustrations and achievements
From Weekly Assignments to a Single Over-Arching Project
The Phase 1 course required weekly written assignments focused on the readings and research materials assigned that week. The strategy seemed to emphasize learning theory and the acquisition of new knowledge through the preparation of written summaries.
In the revision, the Phase 2 course modeled a project-based learning approach with students assigned a series of tasks that built on each other, culminating in the development of a 3-Day unit or set of lessons that participants could use with their students in the fall.
From Discussion Questions to Real Discussions
The Phase 1 course prompted participants with weekly discussion questions that related to research read and videos watched during the content portion of the week. In their postings they often summarized what they had read; peer responses to posts did not generally extend or deepen the conversation. Further, few discussions moved past the initial response posts.
The Phase 2 course revised the discussion prompts to focus participants on their own personal experiences with the weekly topic. Instructors developed strategic “moves” for deepening discussions to facilitate the development of a dynamic learning community.
From Abstract to Actionable
The Phase 1 course focused on the theory of language acquisition leaving participants to infer how best to implement the theory in their own classrooms. They were invited to watch videos of other teachers teaching English Learners and evaluate the observed instruction. The video watching strategy reflected the course’s implementation during the summer when the novices were not able to observe actual teaching at schools.
The Phase 2 course required participants to dig past the theory by reviewing proven strategies and then deciding which strategies would work best for their specific students. By having novices “create” their own students at the beginning of the course, the project team believed the teachers would feel an important sense of responsibility and commitment to these students, and, vested in their success, would make curricular and instructional decisions based on their needs. While real students continued to remain out of sight, instructors had designed a process to mimic the approach used by experienced teachers to design effective instruction: begin by knowing your students well.
From Anonymity to Collaborative Engagement
The Phase 1 course provided few opportunities for structured contact between students and instructors other than the feedback exchanged on assignments and instructor posts on discussion boards; while some students reached out to their instructors via email, the online delivery of the course appeared to limit interaction and connectedness.
The Phase 2 course featured the same instructors working hard to be perceived as real people attentive to participants and their learning. The instructors posted self-made videos welcoming the participants and orienting them to the week’s course of study. They posted pictures, brief bios, and hosted a Fun Things discussion thread to facilitate connections. Through frequent postings that validated participants’ learning, instructors strove to deepen the discussions by pushing past the initial post, inviting participants to explore extended applications of the concepts emphasized that week.
From Final Evaluation to Formative Data Collection
The Phase 1 course asked instructors to collect weekly survey data but the data were not shared and discussed during the course. Instructors and the project team held 3 reflection conversations during the first summer, providing instructors with an opportunity to gain general insights into the students’ experiences. The final comprehensive evaluation, completed in December 2014, provided actionable feedback but results from this evaluation were not available until after the course had ended.
The Phase 2 course used weekly reflections submitted directly to the instructors as a way to check on participants’ engagement with the online medium, instructors’ pedagogical choices, and the content of the curriculum.
Findings: Curriculum Redesign Outcomes
Participant Engagement, Feelings of Efficacy, and Sense of Preparedness
- Course re-design resulted in more participant engagement, increased feelings of preparation related to engaging with peers, evaluating and reflecting on practice, choosing teaching strategies to meet different student needs.
- Teachers of color had more positive beliefs about their teaching efficacy.
