Screener Validity and Reliability

History and Background

The Universal Screeners for Number Sense (USNS) released in three phases during the 2020-2021 school year are the result of more than a decade of development. They are iterations of the Boulder Valley School District’s Math Screeners implemented beginning in 2008. Feedback from dozens of teachers and extensive field testing informed the development and refinement process of the Screeners. The 2020-21 version of the USNS is the result of extensive research, testing, and usage.

Screener Validity

Face Validity: Tasks are cleanly focused on specific skills. Detailed rubrics accompany each task and add additional focus.

Screener Content Validity: The content of the assessments aligns primarily with the standards of the Common Core. Standards alignments are provided for each of the tasks. In particular Jason Zimba’s wireframe graph informed many of the emphases seen throughout the series.

Screener Predictive Validity: Wilkins, Woodward and Norton 2020 confirmed the predictive validity of the USNS. Prior to this research, the BVSD assessment editions demonstrated strong correlations with the Colorado Measures of Academic Progress for predicting results of both the fall Screener to the spring state test and the spring state test to the fall Screeners.

Construct Validity: The construct of “number sense” is an idea that has evolved over the past few decades. The tasks of the USNS seek to reveal sense making and assess the underlying skills that contribute to sense making. Emerging theories of learning trajectories were influential in task selection and calibration. The US Math Recovery Council and the ideas of Realistic Math Education heavily influenced task design.

Reliability

Perhaps the most important aspect of reliability to consider for the USNS is inter-rater reliability. The 2020-21 USNS task revisions include detailed rubrics with task descriptions, including examples and other clarifications. These detailed rubrics and descriptions contribute significantly to the assessments’ inter-rater reliability and provide guidance for local experts to assist with training and scoring.

Meaningful and Valuable

Another aspect of the assessments beyond Screener validity and reliability is the meaningfulness and educational value of the assessments. The USNS target skills and concepts in mathematics that are critical for students. Task design connects to directly actionable, teachable ideas. The information is meaningful for collaborative groups and contributes to the learning of the adults as well. Watching students solve novel tasks helps practitioners gain a deeper understanding of the students they work with and the variety of conceptualizations of students’ mathematical understanding.

Bias

The tasks of the USNS are accessible to students across a wide spectrum of social and economic circumstances:

• Language is concise and precise.
• The interviews allow teachers to ensure that the student has understood the problems.
• The assessment can be translated into any language and Spanish translations are provided.
• Problem contexts and images also support access.
• Many of the tasks incorporate or allow for the use of physical materials.
• Lastly, students can use paper and pencil to solve problems as they choose.

Criterion Referenced with Norms

The USNS is a criterion referenced assessment system. The criteria are tied to the expectations defined by the Common Core State Standards for Mathematics. Furthermore, the detailed rubric section of the assessments includes descriptions of the target skills of each of the tasks. Task performance is indicative of each of the skills and concepts as defined by Common Core.

Forefront®, by Forefront Education, helps educators collect, aggregate, and visualize assessment results. Teachers and interventionists can understand the relative performance of students through the use of Forefront and participation in the Forefront Global Cohort for the USNS project. The software allows educators to isolate student performance and growth, as well as aggregate results across classes, schools, districts, and the Global Cohort. If you would like to explore how Forefront helps visualize assessment results, demo the software here.

Future Plans

The Universal Screeners for Number Sense are an ongoing project. I continue to refine and update the Screeners based on feedback from teachers and interventionists. Further research studies are planned. Alongside this research, planned development for the Screeners includes the assembly of sets of accompanying instructional tasks to provide guidance for responding to the assessment tasks.

To contribute to the USNS project, share your own feedback at screeners@forefront.education or join the Number Sense Group to discuss using the Screeners with a community of like-minded educators here.

Screener Task Design Webinar

David Woodward, the lead author on both the USNS project and the original BVSD Number Sense Screeners, led this conversation with educators about the design and development of the Screeners. Access a recording of the webinar from April 27, 2021, here.

Citation: Wilkins, J.L.M., Woodward, D. & Norton, A. Children’s number sequences as predictors of later mathematical development. Math Ed Res J (2020). https://doi.org/10.1007/s13394-020-00317-y

About the author: David Woodward, Founder and President of Forefront Education, has more than 25 years of experience in education as a classroom teacher and district leader in math education. Author of the Universal Screeners for Number Sense, David led the Boulder Valley School District (BVSD) Math Screener project during his tenure at BVSD. He founded Forefront Education to help educators better understand student learning with classroom assessments and support standards-based grading that is automatic, accurate, and reliable.