STAAR Analysis Article
Exploros Increases 8th Grade Social Studies STAAR Scores by 17% Across Multiple Districts
Summary
Campuses that systemically implemented Exploros for 3-9 months gained an average of 8.5 points on STAAR in 2017 (17.3% increase) whereas the state average was a decrease of 3 points (5%).
Background
During the 2016-2017 school year, Exploros Social Studies was launched, and 67 Texas ISDs discovered and taught Social Studies using Exploros. Several districts had the opportunity to implement Exploros campus-wide, and for extended periods of time. This document analyzes the outcomes of these campuses.
STAAR Results For Full Year Implementation of Exploros (up to 43% gains)
- Summary of STAAR Results for 2015-16 and 2016-17
- Academic Progress Indicator # 1 - Student Engagement. Characterized by the active participation of every student during 5E model instruction, the indicator is student responses over time. Responses include graphic organizers, brainstorms, predictions, tables, drawings, open responses and more.
- Academic Progress Indicator # 2 - Formative Assessment (STAAR Test Score Indicator) – Exploros end of lesson and unit review formative assessment scores over time were a rolling indicator for STAAR test scores, and played an important role in just-in-time remediation and learning.
STAAR Results For At Least 1/3 Year Implementation of Exploros (Avg. 17% gains)
Implementation Best Practices. In reviewing usage data, teacher and coordinator surveys and interviews, and STAAR campus scores, Exploros defines best practices to achieve academic growth.
Background
In mid-September 2016, Exploros formally launched its 8th Grade U.S. History course and began contacting districts. Despite the school-year having already begun, interest was high – districts were motivated by the level of student engagement, the impressive STAAR results from a prior year pilot (24% increase), and the effective use of devices during 5E model teacher-guided instruction.
Districts began scheduling 90-minute complimentary professional learning seminars to learn how to implement Exploros in the classroom, and by the end of October, more than 20 campuses had committed to piloting for the remainder of the Fall semester. Other districts began organizing for Spring pilots and interest and usage steadily grew through 2016-2017, culminating in 67 Texas districts teaching social studies with Exploros.
STAAR Results For Full Year Implementation of Exploros
As a summative test, the Texas STAAR test assesses students on the full year of 8th grade U.S. History. To review the impact of Exploros on STAAR, we first considered the only district that had Exploros for the entire 2016-17 school year: Weslaco ISD. Beatriz G Garza Middle School had piloted Exploros as a platform for a variety of subject-areas in 2015-16, and the district purchased a subscription to Exploros for the 2016-17 school year. They were especially familiar with Exploros OER Social Studies. It was the most extensive curriculum during their pilot, and it led to significant gains in prior year Middle School Social Studies STAAR results (+24% increase.)
For 2016-17, two campuses elected to systematically implement 8th Grade Social Studies using Exploros. They began teaching lessons facilitated by the platform in August, and continued through the following May, completing a full academic year of instruction.
Summary of STAAR Results for 2015-16 and 2016-17
During the 2015-16 school year, Beatriz G Garza Middle School used Exploros extensively and gained an absolute 11 points, or 24%, on the 2016 Spring STAAR test. During 2016-17, Armando Cuellar and Beatriz G Garza used Exploros extensively, while the other two campuses in Weslaco used Exploros lightly, but did not use Exploros for 8th Grade Social Studies.
After the 2016-17 school year, Beatriz G Garza increased again by 4 points, a year-over-year increase of 7% and a total increase of 33% since Exploros implementation. Armando Cuellar gained 18 points for a 43% increase since using Exploros. The other two campuses were slightly up and slightly down this past year, and in total have dropped mildly over the past two years. On any campus or in any given district, many things change over the course of a year or two. But Adrian Cantu, Social Studies Strategist at Weslaco ISD, says that one thing has been consistent for schools that are showing solid improvement: Exploros.
Full Year Exploros Use (Campus Comparison - Weslaco ISD)
Academic Progress Indicator # 1 - Student Engagement
One measure of instructional fidelity and student engagement is to plot student responses as a function of time. In Exploros, a response is a point of data entry for each student or small group of students participating in a teacher-guided 5E model experience (lesson). Responses take many forms, from open responses with optional image upload, to graphic organizers and tables, to developing diagrams or drawings, to answering embedded assessments or taking quizzes. Each response is mapped to a learning objective, and often to at least one of the TEKS.
The following graph shows responses over time for the cumulative 100K student responses gathered for the two campuses in 8th Grade U.S. History only. With peaks where multiple classes concluded lessons on the same day, the graph demonstrates a consistent use of the platform and curriculum throughout the school year.
Student Responses Over Time
To look more closely at the kinds of activities students engaged in while participating in small group and class-wide lessons, we can look at some summary metrics from each campus.
Exploros Social Studies Activity Summary at Armando Cuellar MS
As depicted in Figure 5, at Armando Cuellar Middle School, three teachers taught 236 unique students 299 experiences, or lessons. With the exception of some unit reviews, the vast majority of these lessons were 5E model lessons. During these lessons, students posted a total of 35,551 responses, where every student was an active participant in the classroom learning experience. This student involvement is critical to impact - in order to progress through a classroom experience, all students post responses. Many of these responses are social, where students share their ideas with the teacher and their peers. Furthermore, students often see what their peers post, broadening the exchange of ideas and concepts. The last metric, Response Shares, is an indication of this idea exchange: the 35,551 responses that students shared, either to the teacher or their peers, were received 271K times.
In the case of Beatriz Garza Middle School, the metrics measured were even higher.
Usage at both campuses was consistent, and the average class went through approximately 25 5E Model Exploros experiences. During these experiences, students posted an average of 150 times each, where posts were often the culmination of included brainstorms on historic concepts and principles, graphic organizers developed around primary texts, and many other higher depth-of-knowledge learning exercises.
A wide variety of research has demonstrated repeatedly that student involvement and a sense of belonging in classroom learning increases enjoyment and academic outcomes. Because Exploros involves all students and leverages students’ social media skills to give every student a voice, engagement in academic learning increases. And predictably, so do STAAR test results.
Academic Progress Indicator # 2 - Formative Assessment
During each classroom lesson facilitated by Exploros, students participate in formative assessment. Some of the assessment types are specifically designed to assess students on the TEKS and prepare students for STAAR testing. Specifically, at the end of every 5E model learning experience, students undergo 3-5 questions to assess understanding, which provides teachers real-time insight into concepts that challenge students. Additionally, at the end of every unit, teachers can assign a unit review with 20-30 questions in it. Some of the questions are licensed from the TEA, in the form of past-year STAAR items, whereas others are not actual past year questions, but are developed to be in the style of STAAR. By using these kinds of questions, students incrementally gain confidence with STAAR-like question content, format, and approach.
Unlike benchmark exams, students answer these items as a function of daily instruction - in small doses, continuously. Items are automatically scored and teachers and administrators can view individual or aggregated results at any time.
A graph of the STAAR-like assessment results can be seen in Figure 7. Showing quiz averages over time, both campuses saw a general uptrend in quiz scores over the course of the school year. This is highlighted by the moving average lines for each campus. Also shown on the graph (in red) is the actual STAAR Social Studies score for each campus for Spring 2017 (both campuses scored 60%.)
Quiz Results Over Time
The summation of the items, scores and respective STAAR test score are shown below for each campus.
Interestingly, the straight average of quiz scores for Cuellar Middle School was 58%, and the average for B. Garza was 55%, which is quite close to the actual STAAR scores for both campuses (60%). Further note that for both campuses, students were on an uptrend in test taking ability as seen in the moving averages, which would imply that a straight average would be slightly low. Exploros has found similar results in other campuses from other districts, even though the data sets are not as complete in other samples. Thus far, comparisons have been straight averages, but in 2017-18, we’ll use additional data and TEKS mapping to regress based on TEKS STAAR weightings to refine predictive analytics.
STAAR Results For At Least 1/3 Year Implementation of Exploros
During 2016-17, nearly all of the Exploros implementations for Texas Social Studies were partial year pilots. Exploros offered formal pilots for the Fall and Spring semesters, but not for both. We had teachers from over 65 Texas ISDs teach students Social Studies, and students posted and shared millions of responses.
While there was a large amount of student 5E model engagement with hundreds of thousands of demonstrations of higher order learning skills (e.g. graphic organizers) and an even greater number of embedded assessments and embedded STAAR practice assessments, it is inherently difficult to infer the effect on a full-year summative test based on partial year use.
Whereas a sample full year use of Exploros, as plotted by student responses over time, is represented above in Figure 4, an example partial year use looks like the sample shown below in Figure 9.
For us to truly quantify the effect on STAAR results for partial year usage, we would need the breakdowns for results for each part of the STAAR test. In still other cases, we had several teachers in a school teaching consistently with Exploros, but a teacher or two taught infrequently. In order to properly analyze such cases, we would need class-by-class STAAR results, which are not publicly available. For the purposes of this report, we used TEA Campus by Campus public STAAR results data.
Over time, we will work with pilot districts to review results more granularly, but as a first pass, we reviewed pilot campuses that met the following two criteria:
- The campus taught with Exploros for at least 1/3 of the year.
- All students who tested for STAAR participated in Exploros learning experiences at a concentration of at least 1.5 experiences per week over the minimum time frame (An average of 13 5E experiences minimum per student for all students on campus.)
When we paired the data in this manner, 5 districts qualified with 6 campuses. All 5 districts and 6 campuses improved on their STAAR results, anywhere from 2 to 18 points, or 5 to 43% improvement. Across the state, STAAR scores dropped, down 3 (5%).
Implementation Best Practices
We’re currently compiling more extensive best practices information, which we’ll make available to our 2017-2018 subscribers as part of their professional learning. But in summary, here are few keys to success.
1. Usage Frequency and Consistency
Teach 5E Model Experiences 1-2x Per Week
Exploros provides ~100 learning experiences per course in Middle School Social Studies (6th, 7th, and 8th grade). They are aligned to the TEKS and broken into units that match most Texas ISD pacing guides. While a teacher could teach with Exploros daily, that is not necessarily the intent. We recommend teaching 1-2 learning experiences per week, which should take about 1-2 class periods. Note that in 2016-2017, feedback indicated that some learning experiences were taking longer than a 45-minute class. To remedy this, Exploros is releasing article summaries which can significantly reduce the length of embedded articles for students. Teachers can assign the full article or summaries on a differentiated student-by-student basis.
Assign the Unit Reviews
The unit reviews have 20-30 questions and are made up mostly of past year STAAR test items. Even if you have not thoroughly taught the experiences in a given unit, assigning the review can provide insights into class and student understanding on important concepts and topics.
2. Guide Classroom Dialog
One of the strengths of Exploros is that it allows all students to actively participate in classroom or small group learning using social media skills – this shortens the time to gather input from everyone. To quote former teacher and public educator Deborah Meier, “teaching is mostly listening. It’s the learners who should be doing most of the ‘telling,’ based on how they grapple with an engaging curriculum.”
Exploros gives every student a voice, and helps each student be a contributor, sharing ideas with peers and in turn gaining a more vested interest in learning objectives. But students need guidance. The learning experience is greatly enhanced when teachers review what students post, and facilitate dialog to emphasize key points or illuminate misconceptions. Built into the lessons is an opportunity to do this between every “E” in the 5E model, and we recommend that teachers take the time to host these discussions. Even if Exploros is being used in a flipped classroom or small groups of students are going through differentiated instruction, guided follow-up with students greatly enhances the impact of each experience.
3. Practice Just-in-time Remediation
Just as guiding the discussion is critical, it’s also important to follow-up on end-of-experience quizzes (the 5th E - Evaluate). In late Spring of 2017, we added a feature for subscribed campuses to facilitate this - the Responses Dashboard. Whereas before this dashboard, a teacher had to end an experience to see student quiz results, they are now visible live in the Responses Dashboard (and you can still see the final results in a heat map). If a large number of students does poorly on a particular question, a teacher can see it as soon as students turn in their quizzes. If possible, it’s best to cover any missed questions the same day, or at the outset of the next class period.
4. Compare Notes and Work Towards Instructional Improvement
One recurring comment from Curriculum Coordinators is that Exploros can help Professional Learning Communities (PLCs). When a cadre of teachers teach the same lesson, can review the depth of knowledge expressed in student responses, and can compare notes on classroom discussions and dialog, the topic shifts from what was taught to how it was taught. As Jennifer Smith, Secondary Social Studies Coordinator at Round Rock ISD put it, “Exploros offers common conversation and focus to PLCs and grade level teams. Instead of focusing on the what they can focus on the how.” And when teachers can spend time discussing how students were taught, and how students demonstrated what they learned, good things happen.