People for Education's submission to Ontario’s Independent review of assessment and reporting
A new approach to
|In Ontario, we need more and better information on what students are learning; we do not need a large-scale testing industry or an educational system that is driven – and limited – by the need to teach only what is easily measured, or to measure only what is easily taught.
For the Love of Learning, the Report of the Royal Commission on Learning, 1995
|We recognize that children’s cognitive, social, physical, and emotional well-being are interwoven, and the agency is committed to working with the rest of the education community to support the growth of the whole child.
Modernizing EQAO to Better Support Student Learning, EQAO, 2017
Key principles for effective assessment
Ontario has launched a review of assessment in the province – in classrooms, schools, school boards, provincially and internationally – to ensure that at every level it is possible to know whether we are developing young people who have the “knowledge, skills and work habits for economic and employment needs” and who are “confident, capable and caring citizens, who respect the diversity of Ontarians, who honour and value Indigenous knowledge, who appreciate and respect cultures and communities in a global context, and who will thrive in their personal and future professional lives and well-being.” (A Learning Province Discussion Paper 2017)
One part of the review is focused on large-scale assessment and is examining assessments conducted by the Education Quality and Accountability Office (EQAO).
People for Education has spent the last four years identifying, defining, and field testing essential competencies for long-term success. We have partnered with education experts, measurement experts, policy makers, and teachers in exploring assessment and measurement in five domains of learning – citizenship, creativity, health, social-emotional learning, and quality learning environments.
As a result of that work, we believe that any ‘modernization’ of assessment needs to adhere to the following five principles:
- Large-scale assessment must be part of an overall coherent framework for education.
- Public reporting on student performance should connect to the broad purposes of education.
- Don’t assume that narrow measures can give complete information about complex competencies. In other words, don’t confuse the proxy for the whole pie.
- The purpose for the assessment must match the tool being used.
- The organization that designs and administers assessments is responsible for the way results are used.
Click through the tabs to explore the principles in more detail.
Large-scale assessment is only one piece of a larger education system. Any changes to the existing assessment strategy need to be purposefully integrated with other aspects of the system. In September 2017, Ontario launched an education renewal process that includes:
- A curriculum refresh to put emphasis on “transferable life skills” such as communication, problem-solving, critical thinking, creativity, and global citizenship. (Ministry of Education media release, 2017)
- An update of report cards to “better track a young person’s development of these essential and transferable life skills.” (Ministry of Education media release, 2017)
- “A renewed approach to grade 9 in which all students are better able to explore pathways to work, college, apprenticeship, and university.” (Education Equity Action Plan, 2017)
Before moving ahead with any changes, it is important to understand the connection between these initiatives, including how they support each other and the order in which they happen. For example, it makes little sense to design assessments and change report cards to include transferable skills before integrating these skills into the curriculum.
Equally, it is critical to recognize that the most productive assessment – the assessment that leads to the deepest learning and development and has the greatest impact on both students and teachers – happens inside classrooms.
With this in mind, it is essential that the entire system is using a common lexicon in reference to things such as global competencies or transferable skills. Specifically defining these skills as sets of competencies may be the first step in a renewal process through which the development of initiatives, policy, and supports should flow.
There is wide-spread agreement that public education has multiple purposes beyond teaching literacy and numeracy skills. The public education system plays a central role in ensuring that the next generation is equipped to address the complex social, economic, and environmental challenges that the world is facing. Developing competencies in areas such as creativity, citizenship, health, and social-emotional learning, provide students with the skills they need to live happy, healthy, economically secure, civically engaged lives.
Currently, EQAO testing is limited to student performance in reading, writing, and mathematics. While the EQAO also surveys students, principals, and teachers in relation to school strategies focused on mathematics, reading, and writing, including things like outreach to parents, school improvement planning, and students’ attitudes, public and policy focus appears to be driven mainly by test scores. The EQAO assessment data is used both as a diagnostic and as a tool to plan and evaluate overall system performance. Using the EQAO assessments in this way results in multiple issues:
- A “narrowing” in the things that are taught and the ways they are taught (Ravitch, 2012).
- A reduction of educators’ licence to experiment and take risks for fear of poor performance (Kempf, 2014).
- A limited view of any potential problems in student learning outside of these areas, as they are the only ones readily available (Bereiter, 2005).
To ensure our public education system is supported to achieve its broader purpose, it is vital that we move beyond using narrow measures as proxies for monitoring overall “success” of the education system.
Large-scale assessments can create a false assumption that achievement on a single test provides a complete picture of a student’s ability in that area. It is critical to acknowledge that standardized assessments or surveys can only give information of limited quality about complex competencies (Sellar, Thompson, & Rutkowski, 2017).
Currently, the role of EQAO assessments is to “serve as a basis for meaningful improvement planning at the individual student, school, school board, and provincial levels” (EQAO, 2017, p. 7). However, this may be asking too much of a single measure.
The purpose of any assessment must be determined when designing the tool, as it has a marked influence on how the results are interpreted (Black & Wiliam, 2007). Classroom assessments will look different from provincial assessments because they serve different purposes.
Large-scale assessments provide information that, while limited, is comparable across a large population of people from diverse communities and cultures. Information gathered at a large scale is most productive in providing broad, landscape views that can inform provincial directions and strategy in relation to policy, resources, and infrastructure.
Classroom assessment provides rich and diverse information about learning at an individual or small group level. It is most productive in supporting teachers and schools in designing learning conditions, lessons, school and classroom direction, and communicating to parents and local communities.
It is possible to capture some information about student capacity in relation to broader sets of skills and competencies through large scale assessment (e.g. see PISA, 2017) and surveys, and it would be useful to do so at provincial levels for provincial purposes. However, educators have a far greater and more diverse range of information that can be used to assess their students. This is especially true in relation to hard-to-measure, complex areas.
For many years, the EQAO itself has said that the results of its assessments are often used inappropriately (e.g. in real estate listings, for ranking and comparing schools etc.). This misuse is not only an unintended consequence of census-based testing, it has an impact on the validity (the credibility and believability of a measure) of the testing itself.
Validity is best thought of, not as a property of a test, but a property of all possible interpretations of the results of that test. Inappropriate interpretations made based on EQAO results challenges the test’s validity. We recommend that, moving forward, EQAO attends to the appropriate definition of validity by ensuring that the design of the test and the results available cannot be misinterpreted or misused (American Psychological Association, 2014).
The value of independent assessment and evaluation
An independent provincial agency that collects information about system performance is vital to ensure the strength of Ontario’s public education system. Having a body that is arms-length from government allows some freedom from the political pressure that may come as a result of vested interests in specific outcomes of system performance. It may also help to guard against biases that may be present when ministries conduct internal evaluations of policy.
While it is clear that an independent agency plays an essential role in public education, the way forward in re-thinking the role of large-scale assessment in Ontario cannot be to simply add more content or areas of measurement to the existing ways that EQAO operates. However, the agency has the capacity to collect vital data –through both assessment and surveys – and this data can be essential for communicating to the public, understanding the effectiveness of policy, and developing strategies for overall system improvement.
The pros and cons of large scale assessment
There are both benefits and challenges in using large scale assessments to assess both students’ academic achievement and their development of transferable skills.
Large-scale assessment and evaluation of a broad range of learning areas, as well as of the resources and opportunities available to students, can:
However, large-scale assessments can also:
Attending to areas beyond achievement
Our work has shown that it is vital to understand and support other areas of schooling beyond student achievement. The quality of the learning conditions, the resources available, and the quality of policy and curriculum all play major roles in the support of transferable skills. These areas are amenable to different types of information collection beyond assessments.
School, student, and educator questionnaires can provide essential information; as can supporting provincial opportunities for educators to collaborate on assessing transferable skills or competencies. Providing opportunities for teachers to work together and share strategies, experience, and professional practice in assessing these areas of learning is critical to building coherence in system assessment and connecting any potential large scale approach to relevant information about classroom learning processes.
Sampling versus census-based testing
One way to realize the benefits of large-scale assessments, while mitigating the challenges, is through a sampled assessment environment that eliminates access to individual student and school level performance measures on provincial assessments. A sampled environment can be coupled with a coherent strategy for information collection in diverse areas using existing, well-designed information collection processes like the Early Years and Middle Years Developmental Instrument(s) (e.g. EDI, 2016; MDI, 2016). Such a provincial level strategy could provide a broad landscape view of system performance that would reap the benefits of large-scale data collection, while mitigating many of its challenges.
The challenge in moving away from census-based testing is that many school boards and school leaders have come to rely on existing EQAO assessment measures when working on school improvement.
Moving away from whole population testing may create some challenges for school boards:
- It would limit their ability to use easily comparable achievement indicators as a means to address conditions that might support more equitable outcomes for students
- It would limit boards’ and educators capacity to compare results between schools with similar demographics. Currently these comparisons of “statistical neighbours” are one of the methods the province, school boards and educators use to both evaluate programs and strategies and strategically direct resources.
- Smaller school boards, with little or no local research capacity or expertise, rely on this data.
- Most, if not all school boards, use this data to report about their performance to their local communities. In many cases, the local reporting is underscored by comparisons to other schools, or school boards, in the province.
However, these challenges are not insurmountable.
It is possible to develop a careful, coherent assessment strategy that addresses local challenges while also supporting a common competency framework through which curriculum and policy, large scale assessment, and school assessment could flow.
It is also possible to develop assessments that provide much richer information about the system’s performance and its capacity in a wide range of vital areas. For example, standardized assessments given to a sampled population of students, on a sampled selection of competencies, could provide information about system performance, while avoiding some of the negative consequences of large-scale measurement (e.g. FINEEC, 2016). Or, regionally-based teams of teachers could develop standards and then conduct team-based assessment at different levels/grades in the system. This would provide jurisdictional information about student performance, but it would require more resources, infrastructure, and capacity (e.g. Mooney & Mausbach, 2008).
Even more importantly, developing a coherent assessment strategy could open up the opportunity to include a variety of other data to complement results from sampled assessments at the provincial level, while also supporting capacity and resources to enhance classroom assessment practices in these critical areas of learning.
It is time to update our assessment system in order to address areas of student learning that are central to success in school and life. However, in so doing, it is also vital to creatively and collectively address the challenges and limitations that all large scale assessment present.