A new approach to
provincial assessment

People for Education's submission to Ontario’s Independent review of assessment and reporting

In Ontario, we need more and better information on what students are learning; we do not need a large-scale testing industry or an educational system that is driven – and limited – by the need to teach only what is easily measured, or to measure only what is easily taught.

For the Love of Learning, the Report of the Royal Commission on Learning, 1995

We recognize that children’s cognitive, social, physical, and emotional well-being are interwoven, and the agency is committed to working with the rest of the education community to support the growth of the whole child.

Modernizing EQAO to Better Support Student Learning, EQAO, 2017

Key principles for effective assessment

Ontario has launched a review of assessment in the province – in classrooms, schools, school boards, provincially and internationally – to ensure that at every level it is possible to know whether we are developing young people who have the “knowledge, skills and work habits for economic and employment needs” and who are “confident, capable and caring citizens, who respect the diversity of Ontarians, who honour and value Indigenous knowledge, who appreciate and respect cultures and communities in a global context, and who will thrive in their personal and future professional lives and well-being.” (A Learning Province Discussion Paper 2017)

One part of the review is focused on large-scale assessment and is examining assessments conducted by the Education Quality and Accountability Office (EQAO).

People for Education has spent the last four years identifying, defining, and field testing essential competencies for long-term success. We have partnered with education experts, measurement experts, policy makers, and teachers in exploring assessment and measurement in five domains of learning – citizenship, creativity, health, social-emotional learning, and quality learning environments.

As a result of that work, we believe that any ‘modernization’ of assessment needs to adhere to the following five principles:

Large-scale assessment must be part of an overall coherent framework for education.
Public reporting on student performance should connect to the broad purposes of education.
Don’t assume that narrow measures can give complete information about complex competencies. In other words, don’t confuse the proxy for the whole pie.
The purpose for the assessment must match the tool being used.
The organization that designs and administers assessments is responsible for the way results are used.

Click through the tabs to explore the principles in more detail.

1 2 3 4 5

Large-scale assessment is only one piece of a larger education system. Any changes to the existing assessment strategy need to be purposefully integrated with other aspects of the system. In September 2017, Ontario launched an education renewal process that includes:

A curriculum refresh to put emphasis on “transferable life skills” such as communication, problem-solving, critical thinking, creativity, and global citizenship. (Ministry of Education media release, 2017)
An update of report cards to “better track a young person’s development of these essential and transferable life skills.” (Ministry of Education media release, 2017)
“A renewed approach to grade 9 in which all students are better able to explore pathways to work, college, apprenticeship, and university.” (Education Equity Action Plan, 2017)

Before moving ahead with any changes, it is important to understand the connection between these initiatives, including how they support each other and the order in which they happen. For example, it makes little sense to design assessments and change report cards to include transferable skills before integrating these skills into the curriculum.

Equally, it is critical to recognize that the most productive assessment – the assessment that leads to the deepest learning and development and has the greatest impact on both students and teachers – happens inside classrooms.

With this in mind, it is essential that the entire system is using a common lexicon in reference to things such as global competencies or transferable skills. Specifically defining these skills as sets of competencies may be the first step in a renewal process through which the development of initiatives, policy, and supports should flow.

There is wide-spread agreement that public education has multiple purposes beyond teaching literacy and numeracy skills. The public education system plays a central role in ensuring that the next generation is equipped to address the complex social, economic, and environmental challenges that the world is facing. Developing competencies in areas such as creativity, citizenship, health, and social-emotional learning, provide students with the skills they need to live happy, healthy, economically secure, civically engaged lives.

Currently, EQAO testing is limited to student performance in reading, writing, and mathematics. While the EQAO also surveys students, principals, and teachers in relation to school strategies focused on mathematics, reading, and writing, including things like outreach to parents, school improvement planning, and students’ attitudes, public and policy focus appears to be driven mainly by test scores. The EQAO assessment data is used both as a diagnostic and as a tool to plan and evaluate overall system performance. Using the EQAO assessments in this way results in multiple issues:

A “narrowing” in the things that are taught and the ways they are taught (Ravitch, 2012).
A reduction of educators’ licence to experiment and take risks for fear of poor performance (Kempf, 2014).
A limited view of any potential problems in student learning outside of these areas, as they are the only ones readily available (Bereiter, 2005).

To ensure our public education system is supported to achieve its broader purpose, it is vital that we move beyond using narrow measures as proxies for monitoring overall “success” of the education system.

Large-scale assessments can create a false assumption that achievement on a single test provides a complete picture of a student’s ability in that area. It is critical to acknowledge that standardized assessments or surveys can only give information of limited quality about complex competencies (Sellar, Thompson, & Rutkowski, 2017).

Currently, the role of EQAO assessments is to “serve as a basis for meaningful improvement planning at the individual student, school, school board, and provincial levels” (EQAO, 2017, p. 7). However, this may be asking too much of a single measure.

The purpose of any assessment must be determined when designing the tool, as it has a marked influence on how the results are interpreted (Black & Wiliam, 2007). Classroom assessments will look different from provincial assessments because they serve different purposes.

Large-scale assessments provide information that, while limited, is comparable across a large population of people from diverse communities and cultures. Information gathered at a large scale is most productive in providing broad, landscape views that can inform provincial directions and strategy in relation to policy, resources, and infrastructure.

Classroom assessment provides rich and diverse information about learning at an individual or small group level. It is most productive in supporting teachers and schools in designing learning conditions, lessons, school and classroom direction, and communicating to parents and local communities.

It is possible to capture some information about student capacity in relation to broader sets of skills and competencies through large scale assessment (e.g. see PISA, 2017) and surveys, and it would be useful to do so at provincial levels for provincial purposes. However, educators have a far greater and more diverse range of information that can be used to assess their students. This is especially true in relation to hard-to-measure, complex areas.

For many years, the EQAO itself has said that the results of its assessments are often used inappropriately (e.g. in real estate listings, for ranking and comparing schools etc.). This misuse is not only an unintended consequence of census-based testing, it has an impact on the validity (the credibility and believability of a measure) of the testing itself.

Validity is best thought of, not as a property of a test, but a property of all possible interpretations of the results of that test. Inappropriate interpretations made based on EQAO results challenges the test’s validity. We recommend that, moving forward, EQAO attends to the appropriate definition of validity by ensuring that the design of the test and the results available cannot be misinterpreted or misused (American Psychological Association, 2014).

The value of independent assessment and evaluation

An independent provincial agency that collects information about system performance is vital to ensure the strength of Ontario’s public education system. Having a body that is arms-length from government allows some freedom from the political pressure that may come as a result of vested interests in specific outcomes of system performance. It may also help to guard against biases that may be present when ministries conduct internal evaluations of policy.

While it is clear that an independent agency plays an essential role in public education, the way forward in re-thinking the role of large-scale assessment in Ontario cannot be to simply add more content or areas of measurement to the existing ways that EQAO operates. However, the agency has the capacity to collect vital data –through both assessment and surveys – and this data can be essential for communicating to the public, understanding the effectiveness of policy, and developing strategies for overall system improvement.

The pros and cons of large scale assessment

There are both benefits and challenges in using large scale assessments to assess both students’ academic achievement and their development of transferable skills.

The benefits

Large-scale assessment and evaluation of a broad range of learning areas, as well as of the resources and opportunities available to students, can:

identify where additional system support is needed, and provide accountability to the broader public in relation to system performance in these areas.
provide information about the impact of policies and programs in relation to areas such as creativity, citizenship, social emotional learning, and health.
track the system’s progress in providing the conditions of learning that support equitable outcomes in these vital areas of schooling.
promote the visibility and importance of transferable skills in a climate of educational accountability that relies heavily on narrow indicators of achievement in reading, writing, and mathematics.

The challenges

However, large-scale assessments can also:

provide information that can be misinterpreted. For example, it may be possible to measure one or two competencies in creativity at a jurisdictional level, but it cannot be assumed that this information represents system performance in creativity overall.
create a false assumption that a particular skill level demonstrated in a specific competency on a test provides a complete picture of a student’s ability in relation to the competency being measured.
drive policy, funding and accountability in the system in ways that can narrow pedagogic approaches, and constrain experimentation and risk-taking. (Kempf, 2014).
increase time and resources spent in classrooms to prepare for and deliver the assessment, and resources spent at a provincial level to administer assessments.

Attending to areas beyond achievement

Our work has shown that it is vital to understand and support other areas of schooling beyond student achievement. The quality of the learning conditions, the resources available, and the quality of policy and curriculum all play major roles in the support of transferable skills. These areas are amenable to different types of information collection beyond assessments.

School, student, and educator questionnaires can provide essential information; as can supporting provincial opportunities for educators to collaborate on assessing transferable skills or competencies. Providing opportunities for teachers to work together and share strategies, experience, and professional practice in assessing these areas of learning is critical to building coherence in system assessment and connecting any potential large scale approach to relevant information about classroom learning processes.

Sampling versus census-based testing

One way to realize the benefits of large-scale assessments, while mitigating the challenges, is through a sampled assessment environment that eliminates access to individual student and school level performance measures on provincial assessments. A sampled environment can be coupled with a coherent strategy for information collection in diverse areas using existing, well-designed information collection processes like the Early Years and Middle Years Developmental Instrument(s) (e.g. EDI, 2016; MDI, 2016). Such a provincial level strategy could provide a broad landscape view of system performance that would reap the benefits of large-scale data collection, while mitigating many of its challenges.

The challenge in moving away from census-based testing is that many school boards and school leaders have come to rely on existing EQAO assessment measures when working on school improvement.

Moving away from whole population testing may create some challenges for school boards:

It would limit their ability to use easily comparable achievement indicators as a means to address conditions that might support more equitable outcomes for students
It would limit boards’ and educators capacity to compare results between schools with similar demographics. Currently these comparisons of “statistical neighbours” are one of the methods the province, school boards and educators use to both evaluate programs and strategies and strategically direct resources.
Smaller school boards, with little or no local research capacity or expertise, rely on this data.
Most, if not all school boards, use this data to report about their performance to their local communities. In many cases, the local reporting is underscored by comparisons to other schools, or school boards, in the province.

However, these challenges are not insurmountable.

It is possible to develop a careful, coherent assessment strategy that addresses local challenges while also supporting a common competency framework through which curriculum and policy, large scale assessment, and school assessment could flow.

It is also possible to develop assessments that provide much richer information about the system’s performance and its capacity in a wide range of vital areas. For example, standardized assessments given to a sampled population of students, on a sampled selection of competencies, could provide information about system performance, while avoiding some of the negative consequences of large-scale measurement (e.g. FINEEC, 2016). Or, regionally-based teams of teachers could develop standards and then conduct team-based assessment at different levels/grades in the system. This would provide jurisdictional information about student performance, but it would require more resources, infrastructure, and capacity (e.g. Mooney & Mausbach, 2008).

Even more importantly, developing a coherent assessment strategy could open up the opportunity to include a variety of other data to complement results from sampled assessments at the provincial level, while also supporting capacity and resources to enhance classroom assessment practices in these critical areas of learning.

It is time to update our assessment system in order to address areas of student learning that are central to success in school and life. However, in so doing, it is also vital to creatively and collectively address the challenges and limitations that all large scale assessment present.

References

American Psychological Association. (2014). The Standards for Educational and Psychological Testing. Jointly published: APA, AERA, and NCME. Washington D.C.

Bereiter, C. (2002). Education and Mind in the Knowledge Age. Mahwah, NJ: Lawrence Erlbaum.

Black, P. & Wiliam, D. (2007). Large-scale assessment systems design principles drawn from international comparison. Measurement, 5(1), 1-53.

EQAO. (2017). Modernizing EQAO To Better Support Student Learning. Information Paper. EQAO. Accessed:http://www.eqao.com/en/about_eqao/modernization/Communication%20Documents/information-paper-modernizing-eqao.pdf.

Finnish Education and Evaluation Centre (FINEEC). (2016). National plan for education and evaluation 2016-2019 Helsinki, Finland: FINEEC.

Janus, Magdalena & Offord, David R. (2007) Development and psychometric properties of the Early Development Instrument (EDI): A measure of children’s school readiness. Canadian Journal of Behavioural Science / Revue canadienne des sciences du comportement, Vol 39(1), Jan 2007, 1-22.

Kempf, A. (2014). “Revising the pedagogic form: Test-oriented teaching and learning” In The Pedagogy of Standardized Testing. New York, NY: Palgrave MacMillan, 71-97.

Kimberly A. Schonert-Reichl, K. A., Guhn, M., Gadermann, A. M., Hymel, S., Sweiss, L., Hertzman, C. (2013). Development and Validation of the Middle Years Development Instrument (MDI): Assessing Children’s Well-Being and Assets across Multiple Contexts. Social Indicators Research. 114:345–369. Springer.

Mooney, N. & Mausbach, A. (2008) Align the Design: A Blueprint for School Improvement. Alexandria, USA. ASCD.

OECD (2017), Collaborative problem solving, PISA in Focus, No. 78, OECD Publishing, Paris. Accessed: http://dx.doi.org/10.1787/cdae6d2e-en.

Ontario, Ministry of Education. (2017) A Learning Province, Discussion Guide – Public Engagement on Education Assessment in Ontario link to our insights. Queens Park, Toronto, Ontario. Accessed: https://www.ontarioalearningprovince.ca/assets/downloads/A_Learning_Province_Discussion_%09Paper_EN.pdf?v=20171121″

Ontario, Ministry of Education. (2017). Updated Curriculum, New Report Cards Coming to
Ontario Schools: Province Modernizing School System to Unleash Full Potential of Every Student. News Release. Queens Park, Toronto, On. September 6, 2017. Accessed: https://news.ontario.ca/opo/en/2017/09/updated-curriculum-new-report-cards-coming-to-ontario-schools.html.

Ontario, Ministry of Education. (2017). Ontario Boosting Opportunities for Students’ Long- Term Success New Focus on Equity Will Better Support Students in Exploring Educational and Career Pathways. News Release. Queens Park, Toronto, On. September 7, 2017. Accessed: https://news.ontario.ca/edu/en/2017/09/ontario-boosting-opportunities-for-students-long-term-success.html

Ontario. Royal Commission on Learning . (1995). For the Love of Learning: Report of the Royal Commission on Learning. Volume 4. Toronto, Ontario. Accessed: http://www.archive.org/details/forloveoflearnin02onta.

Ravitch, D. (2012). No Student Left Untested. New York Review of Books. New York. February 21, 2012.
Accessed: http://www.nybooks.com/daily/2012/02/21/no-student-left-untested/.

Sellar, S., Thompson, G., & Rutkowski, D. (2017). The Global Education Race: Taking the Measure of PISA and International Testing. Canada: Brush Education.

A new approach to provincial assessment