For the last two decades, success in our education system has largely been determined by narrow indicators of achievement in literacy and numeracy. While there have been significant changes in educational thinking over the same period, the goals and measures of success articulated in accountability frameworks and by government have stagnated.

It is time to challenge measurement to extend beyond the “3 R’s” and include broader areas of success for students and the education system^1,2.

In a call to action, People for Education launched the Measuring What Matters (MWM) project in 2013 to develop a new set of goals and measures of success for schools that reflect the long-term needs of our graduates and our society. People for Education has gained support and increased traction in this initiative through partnerships with a number of education stakeholders, including:

• Ontario’s Ministries of Education, Health, and Children and Youth Services
• Ontario’s Education Quality and Accountability Office (EQAO)
• Ontario’s Principals’ councils
• A number of Canadian universities
• The Higher Education Quality Council of Ontario (HEQCO) 
• UNESCO and The Brookings Institution
• The Alberta Teachers’ Association
• The McConnell Foundation

To achieve the goal of creating a set of reliable, valid measures that are publicly understandable, educationally useful and reflect the broad skills students will need to live happy, healthy, economically secure, civically engaged lives, the MWM project has been designed as a multi-year initiative with 4 phases over 5 years:

Phase 1: research and public consultations

Phase 2: review of curriculum and policy in Ontario’s English- and French-language systems, development of sets of competencies and conditions, and outreach to Ontario schools

Phase 3: field trials in a range of Ontario elementary and secondary schools and development of measurement instruments.

Phase 4: a series of provincial conferences leading to a national symposium.

Phase 1 of the MWM project was described in our “Measuring What Matters: Beyond the 3 ‘R’s’” 2014 report that discussed the research conducted by lead scholars in each of the MWM domains—creativity, citizenship, health, social-emotional learning, and quality learning environments.

Two main conclusions were derived from Phase 1:

• First, consultations with the public, educators and academics (i.e., surveys, focus groups, workshops) revealed that the existing measures in literacy and numeracy are critical, but limited.

Second, as evidenced from the research papers written by the domain scholars, the skills, knowledge, and competencies across creativity, citizenship, health and social-emotional learning, are as or more important than literacy and numeracy in supporting and defining long term student success³.

Building on the findings and insights from Phase 1, this report will provide an overview of Phase 2 and describe the key findings, challenges, and implications of the work done over a 9-month process. This will include a discussion of the review of curriculum and policy in Ontario’s English- and French-language systems, development of sets of competencies and conditions, and outreach to Ontario schools.

Creativity⁴

According to the Measuring What Matters lead scholar on creativity, Rena Upitis (Queen’s University), creativity is more of a process than an outcome, and can be identified in the learning experience in two interrelated ways: the generation of ideas and possibilities and the process of critique, evaluation and improvement.

Creative competencies and skills help to foster self-understanding and an appreciation of culture, and to build students’ capacities to imagine, persist and synthesize. These skills and competencies are vital for problem solving and for developing ways of adapting knowledge to new contexts; they apply across the curriculum—from arts to science to math.

Incorporating creativity into the learning environment helps foster students’ critical thinking skills and helps students develop resilience, resourcefulness, and confidence. It is positively linked to engagement, achievement, and innovation.

Citizenship⁵

Alan Sears (University of New Brunswick) defines citizenship education “as the acquisition of a core set of civic knowledge, skills, and values”. To be effective, citizenship education must include not only knowledge components, but behaviour components as well, that describe ways of being and acting for students both in the school setting and the community.

The competencies of citizenship fall into four categories: civic engagement, civic knowledge, civic dispositions/attitudes, and civic skills. These categories provide an organizational framework to support students in becoming active, engaged citizens by promoting knowledge of historical/political concepts, knowledge of institutions/mechanisms of civic engagement, understanding of social issues, sense of personal identity and voice, and engagement in school and community.

Social-emotional Learning⁶

The MWM lead scholar for social-emotional learning, Dr. Stuart Shanker (York University), identifies five vital competency-based areas in the learning experience: self-awareness, self-management, social awareness, interpersonal relations, and decision-making. In this domain, there are two fundamental principles underlying social-emotional learning as a broad area of success for students: 1) the skills and competencies in social-emotional learning are as critical as formal academic disciplines; and 2) students can learn social-emotional competencies just as they learn formal academic skills – through regular interactions with peers, teachers, and school staff inside and outside of the classroom.

Health⁷

Bruce Ferguson (SickKids Hospital and the University of Toronto) and Keith Power (Memorial University) authored the MWM project domain paper on health and described how the incorporation of school-based health programs benefit society in individual, social, and economic ways.

The Health domain is separated into four main areas: student engagement in physical activity for fitness and pleasure; students’ capacity for making healthy choices; student well-being in relation to sexuality; and students’ understanding and management of mental illness.

By promoting physical and mental health in the learning experience, students are given the tools that facilitate improved academic outcomes, sense of self, confidence, interpersonal relationships, and resilience. It is important to note that the health competencies and skills are closely related to the competencies and skills in social-emotional learning.

Quality Learning Environments⁸

Common conditions in schools are required to provide a foundation for the competencies and skills across the domains. Nina Bascia (OISE, University of Toronto) described these conditions in the MWM domain paper on quality learning environments by looking at three contexts/settings: within the classroom, within the school, and beyond the school.

• The student learning experience in classrooms involves a dynamic interrelationship between students, teachers and content. These interrelationships require attention to the classroom environment and the presence of a range of learning and assessment opportunities. 

• The operation of the school mirrors ideals of citizenship in democratic societies. Social relationships, characterized by trust, interdependence and empathy amongst all members, are centrally important to the school’s overall focus. Imagination, experimentation and risk-taking are all part of a creative pedagogic for teachers and students. 

• The school and community build partnerships to enhance learning opportunities and well-being for students. Promoting a cross-cultural perspective contributes to the development of social awareness and informs decision-making and the development of meaningful school and community relationships. There are two overarching concepts that are core components within this area of learning and school operation:
• the relationships that exist between a school and the community beyond the school,
• the learning opportunities for students that are provided by community organizations or people who are not part of the immediate school membership

Ontario’s French-language Education System

We asked education consultant, Michelle Boucher, to examine the connections between initiatives undertaken in the French-language system, and the five domains in People for Education’s Measuring What Matters initiative⁹.

In Ontario, schools in the French-language system have assumed a mandate far beyond teaching academic subjects and disciplines. These schools – in great part because they serve minority-language communities – feel the weight of an additional responsibility: supporting the vitality of the language and the culture of the communities they serve. In this way, French-language schools provide examples of some of the possibilities for public education when the goals for education are expanded beyond the “3 R’s”.

Citizenship, creativity, social-emotional learning, health and the school environment itself, all play key roles in supporting culture and identity. By adding this responsibility to that for academic achievement, French-language schools have articulated a broader purpose and broader goals for education. For French-language schools, the implementation of these competencies and conditions could play an important role in supporting culture and identity as well as community-building.

There are a number of core challenges inherent in translating each domain research paper into sets of competencies and skills, and outlines of the school conditions required to support them:

The competencies and school conditions must be observable and ‘concrete’ in order to be useful in relation to: planning teaching and assessment in schools; and, policy and measurement for school districts and Ministries.

Each domain in Measuring What Matters (MWM) offers its own discrete meaning or perspective of student learning and draws from different bodies of research— different ontologies and epistemologies in relation to student learning and school conditions. However, in working together, the domain scholars recognized that the domains are also closely related to each other. This led to an approach to building domain specific competencies and conditions that focused on both the discrete meaning within the domain and the interconnected relationships across the domains. A key challenge in Phase 2 and moving forward is to create a framework that articulates both its interconnectedness, while also affirming the specific meanings that each domain offers to broad areas of student learning.

Building a set of student competencies that provide meaning within each domain was an iterative process in which competencies and conditions within the domains were refined, revised and combined to produce the current sets. The competencies and conditions are relevant for students and schools from kindergarten to grade 12, but they will have a different kind of relevance for different subjects and different grades, so the competencies and conditions must also allow for variation and interpretation depending on the context.

The competencies within each domain are also related or connected across the domains. As detailed in earlier reports, competencies in areas like ‘risk taking and experimentation’ in creativity are also related to competencies in other domains like ‘student agency’ in citizenship and ‘self-awareness’ in social-emotional learning¹⁰.

All five domain scholars articulate the kind of learning conditions that best enable the development of student competencies,¹¹ and they emphasize that the competencies cannot be separated from the conditions. In fact, each domain paper could be interpreted as one that articulates ‘Quality Learning Environments’ in each domain (e.g., creative learning environments, social-emotional learning environments, etc.).

Throughout the process of moving from discrete papers to a useful framework, we did not want to create a disconnected and redundant set of school learning conditions that describe five different ways to support and enhance student learning and schooling processes. The MWM framework needed to be consolidated into an interconnected set of learning conditions that honoured each domain’s discrete meanings, but did not repeat very similar school processes or conditions within which students had opportunities to develop these areas.

Bascia’s recommendation that MWM follows Talbert and McLaughlin’s model on the school as a cluster of ‘nested’ contexts from classrooms out to school/community interactions became a useful way to situate and organize the conditions across the domains.12 A nested context model allows educators opportunities to make connections or links from one context to another—to trace or map potential competency development for students as they move through the various learning contexts, from the classroom to the playground to the library. And, in so doing, analyze and adapt the context(s) of learning in relation to the variety of ways students might express specific competencies under study or focus.¹³

At the outset of the project, People for Education recognized that the domains in Measuring What Matters are not new to public education in Ontario. In fact, they are ubiquitous across many aspects of Ontario’s policy, curriculum and practice.14 It could also be argued that they form critical components in Ontario’s new educational goal of student well-being.¹⁵

The problem lies in the explicit articulation of overall goals for education. Despite being recognized throughout much of Ontario’s curricula and policy, the competencies and conditions in the domains are not identified as overall goals, they are not publically measured, and in fact they are often subservient to the narrow proxy indicators discussed at the outset of this paper.

The purpose of Measuring What Matters is to establish these domains as a central construct or value in schooling in a way that is both publically understood and educationally useful. Further, the project proposes that these domains can be evaluated and measured through a careful, participatory and collective effort by the public sector. Existing policy and curricula clearly supports this purpose.

What’s present and what’s absent in Ontario’s current curriculum and policy

People for Education had the domain scholars review a representative sample¹⁶ of policy and curricula in the English- and French-language system in relation to each domain. Here, we were interested in understanding how these domains were recognized and supported within the current policy and curricula environment.

Overall, scholars found the domains to be widely represented in existing policy and curricula. For example, reviewing creativity and health, Upitis; and Atkins, Ferguson, Leschied, Rodger, Tucker and Hibbert state:

The possibilities for learning opportunities that engage students in creative processes are rich and diverse. The curricula that are most supportive of creative engagement are those associated with the fine and performing arts and technological education, the latter being, perhaps, the most integrated and exciting curriculum, with the most potential to involve creativity in multiple contexts and across fields of study.¹⁷

…successful ‘academic’ learning is comprised of numerous factors that include emotional wellbeing. This acknowledgment we believe opens the door for increasing emphasis on supporting our students as learners within the broadest definition possible. It is, again from our reading of the documents, evident that we have finally put to rest what has been for too long, a too narrow definition of what it takes to be academically successful that has been limited to an emphasis on cognitive development as opposed to physical and emotional development.¹⁸

This sentiment was expressed across all domain reviews to varying degrees. The larger implication is that the domains are well represented in policy and curricula from kindergarten to grade 12 and, should, logically, exist within Ontario’s core practices of learning in schools. That they are not currently subject to evaluation and measurement to the degree that traditional academic subjects like reading, writing and mathematics may have implications in the quality of attention that schools and school boards may give to broad areas in relation to resources and focus.

Along with noting the domain’s presence in policy, the review brought up gaps, inconsistencies and weaknesses in the ways that the domains are articulated and supported within curriculum and policy in the following areas:

Citizenship

The review of citizenship noted some important strengths across Ontario’s policy and curricula documents, including a pedagogic approach to student learning that recognizes students as active, meaning-makers who need to use previous knowledge and ideas in building new knowledge or perspective.¹⁹ Sears also praises Ontario for explicitly articulating citizenship’s cross-curricular importance within the front matter of all curricula documents.

However, the review also flagged several concerns in relation to citizenship education. In particular, Sears found that differences in the grades 9 and 10 applied and academic social studies curriculum may have serious implications for citizenship education. Sears states:

First, I am struck by different treatment of Aboriginal issues and questions. They are prominent in the academic course but absent from the applied course (this difference is maintained in other areas of the course as well where significant questions about Aboriginal peoples are raised on eight occasions in the academic course and only two in the applied course). I am also struck by the much more critical and potentially political nature of the questions raised for academic students [than] applied students. The implication is that academic students are much more ready for critical citizenship than applied students. There is no research evidence I know of to support this.²⁰

While citizenship was emphasized as a cross-curricular area of focus in the front matter of the curriculum documents, the review highlighted the lack of depth of integration of citizenship education across policy and curricula. Opportunities to cultivate citizenship education by making explicit connections with Ontario’s service learning requirement (the 40 hours of required community service) are not explicitly developed, nor are explicit connections made in curriculum or policy between citizenship education and areas such as critical literacy or the arts. This represents a loss of opportunities to cultivate citizenship education from a number of different perspectives and learning opportunities.²¹

The reviewers also noted the amount of what they referred to as “sloganeering” present across the documents reviewed. Terms like ‘civic engagement’ or developing students as ‘partners in their own learning’ are frequently used without substantive meaning describing what these are. This, in turn, can limit how these ideas are used to develop good evaluative and educative practices in relation to citizenship development²².

In examining citizenship education in the French-language system, Michelle Boucher, the MWM lead on French-language education, emphasizes the additional responsibility that the French-language education system has to support identity-building for students and a sense of belonging to the community²³. She noted that Ontario policy for French-language schools reflects this added responsibility, but provides few practical materials to support evaluation and monitoring.

Creativity

While elements of creativity were found in almost every document analyzed, the critical, evaluative, disciplined-thinking aspects of the creativity domain were consistently emphasized with few references to the other half of the creativity domain, including competencies associated with imagination, exploration, pursuit of interest and play. The notion of play and the importance of cultivating ‘natural curiosity and exuberance,’²⁴ is regularly referenced in relation to early years education, but this strand of creativity is not carried through policy or curricula in relation to elementary and secondary school learning.²⁵

Equally, by consistently naming creativity in conjunction with critical, disciplined thinking processes, Ontario policy and curricula are potentially misrepresenting the creativity domain’s value in learning; disregarding the long-term value and impact creativity offers in cultivating students’ interests, curiosity, exploration and imagination.²⁶

Health and Social-emotional Learning

Ferguson et al raised concerns in the areas of assessment and measurement of mental and physical health. While the intention to support physical and mental health is clearly articulated across much policy and curricula, little is provided that substantiates these areas in ways that can serve evaluation and measurement. They state:

In contrast to the markers set out in EQAO or other structured assessment protocols for literacy and numeracy, goal statements that are explicit in desirable and measurable outcomes are absent in both Physical and Mental Health [PMH].²⁷

The absence of these protocols for this area of student learning relates to Sears’ discussion on sloganeering in citizenship in that it sets up an aspirational quality with little substance provided in how and what these critical areas look like in practice and evaluation.

The review also noted the relative lack of physical health references when compared to mental health or well-being. The reviewers highlighted the need to emphasize this area of student learning more thoroughly with a special emphasis on the importance of an appreciation and enjoyment for physical activity to be drawn through the curricula in elementary and secondary schooling. While the importance of physical health and activity is articulated in many of the early years’ documents, much of the thread is lost after that.²⁸

Shanker and Bayrami say that clearer emphasis is needed to more effectively support integration of social-emotional learning competencies within academic areas of learning.²⁹ When addressing social-emotional learning, the curricula and policy, at times, do not address the critical connections between and across all the five core areas —self-awareness, social awareness, interpersonal relationships, decision making and self-management. Equally, connections between social-emotional learning competencies and academically oriented skills like problem solving could be more clearly articulated in some curricula (e.g. mathematics and science).³⁰

In the French-language system, Boucher notes similar gaps. Ontario’s Francophone communities exist within a dominant Anglophone culture, and many children come to school with limited grasp of the French-language, and little or no immersion in a Francophone culture. To ensure students can express themselves and live their lives in French they must possess a range of social-emotional skills and competencies that build self-confidence, self-esteem and a sense of self-worth. Boucher says that French-language educators do not have the research and development tools needed to document and evaluate their practices, and that concepts such as language learning, culture acquisition, identity-building, and community engagement require further fleshing out, particularly where their implementation is concerned.³¹

In the reviews of both health and social-emotional learning, the scholars found that references to the importance of establishing those relationships are common in and across policy and curricula. However, the core competencies that support the establishment of positive relationships, such as active listening, peer mediation and conflict resolution, were not found to be consistently integrated within policy and curricula where relationships are discussed.³²

Measurement matters. Although reason and imagination also advance knowledge (Kuhn, 1961), only measurement makes it possible to observe patterns and to experiment—to put one’s guesses about what is and is not true to the test (Kelvin, 1883). From a practical standpoint, intentionally changing something is dramatically easier when one can quantify with precision how much or how little of it there is. (Drucker, 1974).³³

While we were working to translate the MWM domains into a set of student competencies and supporting conditions and reviewing these in relation to Ontario’s curricula and policy, we engaged research, measurement and policy experts in a series of working meetings and discussions regarding the possibilities and the problems in measuring broad areas of student success at both system and local levels.³⁴

These conversations covered a wide range of topics and issues from the technical challenge of measuring ‘non-cognitive’ competencies (competencies that do not measure subject or discipline knowledge through academic performance) to Ontario’s current measurement system—its problems and benefits. The discussions challenged the viability of achieving system coherence towards measuring broad areas of student learning.

People for Education used the previous year’s insights and questions generated from a large public consultation to further refine thinking and generate understanding of implications in creating a public set of measures for creativity, social-emotional learning, citizenship, health, and quality learning environments.³⁵ Ideas from these meetings were then consolidated and used to generate further discussion at various international research conferences. Presentations at the conferences were designed to provide People for Education with educational expert feedback and perspective on effective ways to establish a publicly reported and educationally useful set of measures for broad areas of student learning and school conditions. Ideas from the sessions further refined challenges that cluster around 5 large questions:

• How can we introduce broader goals and measures without simply adding to schools’ workloads?
• What steps can be taken to ensure that new goals and/or measures won’t be misused in the same way current test scores are often used to rank schools?
• Are there communications strategies we should undertake to ensure that this initiative doesn’t reinforce the misconception that schools bear sole responsibility for students’ success in all of the domains?
• Is it possible to develop sets of measures that can be relevant to both local needs and context, as well as applicable to a central understanding of the domains?
• How can we resolve the tension between the complexity of education and the public desire for concise reports or simple proxy measures of success?

Some of these questions move beyond the scope of this section and all remain at least, partially, unresolved. However, the questions help frame our thinking around a potential approach to measurement that seeks to address these issues. Throughout the year several themes consistently came up in our work on measurement. These themes provide insights into a potential way forward that recognizes the inherent challenges articulated in the project. It is clear to us now that any good system of public measurement needs to attend to each of these issues or ideas.

Purpose of measurement guides type of measurement

The purpose of measurement is a valuable anchor to any work that advances broader measures in student success. All educational measurement has implicit or explicit purpose(s) including:

• individual student assessments both formative and summative to advance learning
• insights into classroom and/or school practices generated through student learning experiences
• evaluation of the value of a program and policy
• information to support revisions in organizational strategy
• accountability to the public
• a way of drawing attention to what is important in schools
• a way to support policy change

The purpose of measurement should guide the form of measurement used. In many instances, it may very well be the case that there are multiple purposes involved in a single type of measurement. For example, Ontario’s Education Quality and Accountability Office (EQAO) measures in reading, writing and mathematics at grades 3, 6, 9 and 10 are used at times for all of the above stated examples. However, this does not have to be the case as different forms of measurement serve certain purposes better than others.

Classroom level measurement or assessment

Classroom level measurement or assessment happens every day. While there are formal periodic moments where educators assess and evaluate students (e.g. grades), ‘in the moment’ forms of assessment are a critical aspect of daily lessons.³⁶ While not often thought of as a form of measurement, student and classroom assessment is the most frequently applied form of evaluative thinking within K-12 schooling and, potentially, underestimated in its value toward re-thinking approaches to system measurement. That said, caution should be used in measuring students. It is critical that school level measures inform new learning experiences in classroom and schools. Any measures or assessment of students should be balanced, judiciously used and involve educators.³⁷

School level measurement—Learning conditions are central to measuring broad areas of learning

Focusing on the conditions or opportunities that students have to exhibit the competencies and skills is critical. The competencies articulated across Measuring What Matters are ‘dependent on situational factors for their expression’.³⁸ Evaluating conditions through a variety of sources gathered from inside classrooms and across the schools can serve as productive ways to ground understanding of the qualities of school level conditions, their purposes and potential impact.³⁹

The use of student sampling

The use of sampling for the competencies within Measuring What Matters or any provincial assessments needs careful consideration. Good samples can provide rich information for the public, the province and regions while also drawing productive attention to critical issues at school levels without providing opportunities for misuse of data.

On the other hand, using individual student outcome measures in areas like creativity or citizenship in tests of whole student populations could constrain or inhibit the very areas that assessments are designed to enhance. High and medium stakes testing and reporting environments can have adverse effects on competencies and supporting conditions in areas such as risk taking, experimentation and student, teacher and school confidence.⁴⁰ Forms of testing and public reporting have been shown in a variety of studies to inhibit professional risk taking and experimentation while narrowing pedagogic approaches used by educators for the fear of potential failure.⁴¹ This is problematic for teacher professional growth in that the reporting on school performance can inhibit key qualities needed for educators to build reflective and reflexive stances to their practice.⁴²

While some areas of each domain can be expressed in a ‘one off’ testing environment, others cannot. Competencies within ‘social awareness’ in social-emotional learning or ‘taking feedback from peers to improve work’ in creativity, for example, are better assessed within and through critical evaluation of the learning environment by educators. Whole population testing and reporting lends itself to misconceptions on what is being measured and what can be reliably said about the quality of learning within broad student areas of learning. In short, whole population testing to achieve system perspectives in these areas of learning may not be the most productive approach to the problem.

Working in an alliance of education stakeholders/organizations

This work is clearly not the work of one organization. It requires an alliance of diverse education stakeholders. The collaborative use of a matrix of opportunities/ weaknesses of different approaches to potential measurement, for example⁴³, could produce new, innovative approaches in Ontario to measurement and reporting.⁴⁴ Such collective work would help unpack measurement methods, the purpose of measurement and trade-offs regarding types of measurement used (e.g. sampling versus whole population assessment) as well as the potential local-provincial trade-offs in either approach.

Overall, People for Education’s work on measurement helped refine and articulate opportunities and problems within the work. Different approaches to measurement and accountability cannot occur without careful consideration of what currently is measured and reported on—its value, its purpose and its problems. We are now working to continue to promote this discussion, to enable a deepened responsive and balanced approach to measurement that may require a different way to think about what it means for systems, schools and teachers to be publicly accountable.

In 2015/16, we are partnering with a wide range of stakeholders including school districts and schools to use the Measuring What Matters sets of competencies as a means to plan, act, evaluate and reflect on their existing work while also providing us with information about the competencies and conditions as drafted. We will use the practical knowledge generated from partnering to re-draft the competencies and conditions for public use and understanding in practice.

Along with building a refined set of competencies and conditions from the ongoing work detailed below, we will produce a report on measurement and assessment that includes a critical discussion of Ontario’s existing assessment and accountability system, and suggest a way forward.

Overall, the work in Phase 2 of the Measuring What Matters initiative has brought definition and insights to support new perspectives about school measurement and accountability. We now know that despite the fact that the opportunities for students to develop broad learning competencies are supported within current curricula and policy, there are also significant gaps. We also recognize that any new approaches to reporting, evaluation and measurement of these competencies will require a different way of thinking about measurement and accountability than currently exists in Ontario. Moving forward with broad areas of student success will require system coherence and a degree of inter-organizational collaboration across the education sector that has not yet been achieved. This is an incredibly exciting opportunity for Ontario to move from a largely rhetorical stance that points at broad definitions of student success, to purposeful and innovative actions, which better serve the needs of society today and in the future.

The list below provides a brief description of our work with a variety of partners.

Field Trials in Schools and School Districts

We are partnering with a set of schools across the province in order to field test MWM competencies and conditions within school and classroom assessment or measurement.⁴⁵ For schools, the use of relevant areas of competencies and conditions in Measuring What Matters (MWM) offers opportunities for them to frame their work and provides a common lexicon for specific areas of creativity, citizenship, social-emotional learning and health. It also allows them to reflect on their current conditions, comparing them against the conditions outlined in MWM that support student learning in the domains. The field trials are designed in order to provide an approach to measurement that is collaborative, situated within existing assessment practices and flows through their existing work.

For Measuring What Matters, systematic collection of school and school district data, as they use the competencies, provides us with information⁴⁶ about the domains’ potential working interrelationships, the language currently used for the competencies and conditions and the ways that the MWM competencies/ conditions inform or inhibit planning, local measurement and evaluation. This information will then be used to interrogate the existing MWM framework towards a refined draft, as well as provide a deepened understanding of approaches to local, specific measurement/assessment practices in schools.

Ontario’s Education Quality and Accountability Office

With EQAO, we are exploring relevant competencies and conditions within EQAO’s student, teacher and principal questionnaires. This joint exploratory study will provide insights into the domains and their usefulness within large-scale assessment and measurement environments. Specifically, we are interested in the language of some of the domain competencies and conditions and the degrees to which they might exist in school practice as referenced by educators and students. Ultimately, this work is meant to explore different ways to consider assessment for broad learning areas, its potential and its problems in provincial assessment environments.

Ontario Ministries of Education, Children and Youth Services and Health

We are working with a variety of policy specialists within these Ministries to explore the use of a common lexicon through which we can gain coherence in relation to policy and evaluation. The work provides opportunities to build coherence across diverse policy areas that target children and youth, such as Ontario’s fourth education goal in well-being and MCYS’s “Stepping Stones” youth development resource.⁴⁷ As is the case across our work with the education sector, we are continually gathering information about the language of the MWM framework as well as its usefulness in informing policy design and evaluation.

Ontario Institute for Studies in Education and York University

With faculties of education in Ontario, we are working with graduate students and scholars to help puzzle through the tricky, complex problems in building innovative and broad assessment systems. People for Education is bringing specific information and challenges to a variety of scholars and graduate seminars in policy and measurement for discussion and further insights. We are also tapping into current international thinking in education research and scholarship to further deepen our thinking in relation to the project.

Other partnership work across Canada and internationally

McConnell Foundation

With the McConnell Foundation’s WellAhead project, People for Education is partnering to elicit new understanding of how systems can integrate broad areas of student well-being and how we might expand Canadian understanding of the role schools and communities play in supporting broad areas of success.

Brookings/UNESCO

As part of the Brookings/UNESCO Learning Metrics Task Force (LMTF), People for Education is working internationally with a variety of countries worldwide to gain insight into the connections between infrastructure, practice and policy in designing and developing broad assessment areas for students.

Higher Education Quality Council of Ontario

People for Education is working jointly with HEQCO on a collection of Learning Outcomes Assessment projects. The central purpose of the partnership is to make connections and explore areas of alignment between the work in Measuring What Matters in kindergarten to grade 12 assessment and the post-secondary school environment.

MWM Advisory Committee

With key Canadian education leaders⁴⁹, we are gathering information through consultation and ongoing collaborative work to continue to infuse the project with perspective about qualities of educational practice, policy and measurement.

1. Shanker, S. (2014). Broader Measures for Success: Social/Emotional Learning. In Measuring What Matters, People for Education. Toronto: November 8, 2014.
2. People for Education (2015). Measuring What Matters: A New Way of Thinking about Skills. In Measuring What Matters, People for Education. Toronto: 2015.
3. People for Education (2014). Beyond the “3R’s”. In Measuring What Matters, People for Education. Toronto: November 8, 2014.
4. Upitis, R (2014). Creativity: The State of the Domain. In Measuring What Matters, People for Education. Toronto: November 8, 2014.
5. Sears, A. (2014). Measuring What Matters: Citizenship Domain. In Measuring What Matters, People for Education. Toronto: November 8, 2014.
6. Shanker, S. (2014). Broader Measures for Success: Social/Emotional Learning. In Measuring What Matters, People for Education. Toronto: November 8, 2014.
7. Ferguson, B. and Power, K. (2014). Broader Measures of Success: Physical and Mental Health in Schools. In Measuring What Matters, People for Education. Toronto: November 8, 2014.
8. Bascia, N. (2014). The School Context Model: How School Environments Shape Students’ Opportunities to Learn. In Measuring What Matters, People for Education. Toronto: November 8, 2014.
9. Boucher (2015). What Matters in the French-language Schools: Implementing a Broader Vision for Student Achievement in Minority Setting. In Measuring What Matters, People for Education. Toronto: March 2015.
10. People for Education (2015). Draft Competencies and Skills. In Measuring What Matters, People for Education. Toronto: June 2015
11. Upitis, R (2014). Creativity: The State of the Domain. In Measuring What Matters, People for Education. Toronto: November 8, 2014.
12. Bascia, N. (2014). The School Context Model: How School Environments Shape Students’ Opportunities to Learn. In Measuring What Matters, People for Education. Toronto: November 8, 2014.
13. Students may not express the same level of success in regards to any one competency from one learning context in the school to the next—from their math class, for example, to their English class. In fact, student learning behaviours vary widely within their experience in a school. For example, Schmidt, W. H., Burroughs, N.A., Zoido, P., and Houang, R.T. (2015) The Role of Schooling in Perpetuating Educational Inequality: An International Perspective. Educational Researcher, Vol.44, No.7, pp.371-386.
14. Ministry of Education (2013). The K–12 School Effectiveness Framework (2013): A support for school improvement and student success. Link: https://www.edu.gov.on.ca/eng/literacynumeracy/SEF2013.pdf. Accessed October 15, 2015.
15. Ministry of Education (2014). Achieving Excellence: A Renewed Vision for Education in Ontario. Link: http://www.edu.gov.on.ca/eng/about/renewedVision.pdf. Accessed October 15, 2015.
16. The policy and curricula sampling process consisted of several different processes. First, all curricula across kindergarten to grade 12 were reviewed. Second, policy documents that were considered ‘major’ (discussed large areas of policy and practice, such as The K-12 School Effectiveness Framework: A Support for School Improvement and Student Success; Growing Success, Assessment, Evaluation and Reporting in Ontario Schools; or Stepping up: A Strategic Framework on Youth Opportunities) were reviewed by each domain scholar. Third, related and relevant areas of complimentary policy to each domain were selected for review. Here, the selection was guided by similarities of content, purpose or target audience. Fourth, outside partners with working experience in specific policy areas like Health were used to consult on key policy selections. Each domain scholar reviewed between 69 to 76 documents.
17. Upitis, R. (2015). Review of Ontario Policies and Curricula: Creativity and Innovation. Toronto: November 3, 2015.
18. Atkins, M. A., Ferguson, B., Leschied, A., Rodger, S., Tucker, B. & Hibbert, K. (2015) Review of Ontario Policies and Curricula: Physical and Mental Health. Toronto: November 3, 2015.
19. Sears, A. (2015) Review of Ontario Policies and Curricula: Citizenship. Toronto: November 3, 2015.
20. Ibid.
21. Ibid.
22. Ibid.
23. Boucher (2015). What Matters in the French-language Schools: Implementing a Broader Vision for Student Achievement in Minority Setting. In Measuring What Matters, People for Education. Toronto: March 2015.
24. Upitis, R. (2015). Review of Ontario Policies and Curricula: Creativity and Innovation. Toronto: November 3, 2015.
25. Ministry of Children and Youth Services (2007). Early Learning for Every Child Today: A Framework for Ontario Early Childhood Settings. Link: http://www.children.gov.ca/htdocs/English/topics/earlychildhood/early_learning_for_every_child_today. aspx#principles. Retrieved August 2015.
26. Upitis, R. (2015). Review of Ontario Policies and Curricula: Creativity and Innovation. Toronto: November 3, 2015.
27. Atkins, M. A., Ferguson, B., Leschied, A., Rodger, S., Tucker, B. & Hibbert, K. (2015) Review of Ontario Policies and Curricula: Physical and Mental Health. Toronto: November 3, 2015.
28. Ibid.
29. Shanker, S. & Bayrami, L. (2015). Review of Ontario Policies and Curricula: Social-emotional Learning. Toronto: November 3, 2015.
30. Ibid.
31. Boucher (2015). What Matters in the French-language Schools: Implementing a Broader Vision for Student Achievement in Minority Setting. In Measuring What Matters, People for Education. Toronto: March 2015.
32. Shanker, S. & Bayrami, L. (2015). Review of Ontario Policies and Curricula: Social-emotional Learning. Toronto: November 3, 2015.
33. Duckworth, A. & Yeager,D. S. (2015) Measurement Matters: Assessing Personal Qualities Other Than Cognitive Ability for Educational Purposes, Educational Researcher, (44) 4, 237-251. Cited from p.237.
34. Two meetings took place on January 12, 2015 and May 31, 2015 with measurement experts in a variety of fields to discuss the core tensions in measuring broad areas of student success. These meetings followed an iterative process where the partially transcribed discussion notes were sent to the measurement experts and returned with feedback. From these meetings, ideas were generated for moving forward with the problems and possibilities for Measuring What Matters. Personal communication. People for Education. Meeting dated January 12, 2015 and May 31, 2015.
35. Gallagher, K. (2014). Measuring What Matters: Report on Public Engagement in a Broader Measure of Success. In Measuring What Matters, People for Education. Toronto: November 8, 2014.
36. For an example see, Black, P., & Wiliam, D. (1998). Inside the Black Box. Phi Delta Kappan, 80(2), 139–148.
37. Personal communication. Canadian Society for the Study of Education (CSSE) Conference. Meeting dated May 31, 2015.
38. Duckworth, A. & Yeager,D. S. (2015) Measurement Matters: Assessing Personal Qualities Other Than Cognitive Ability for Educational Purposes, Educational Researcher 44 (4), 237-251. Cited from p.239.
39. Bascia, N. (2014). The School Context Model: How School Environments Shape Students’ Opportunities to Learn. In Measuring What Matters, People for Education. Toronto: November 8, 2014.
40. While Ontario’s provincial reporting environment is not considered ‘high stakes’ or an environment where people lose their job and school based on student assessment, it could be considered a ‘medium stakes’ environment. A reporting environment where resource allocation and organizational practices are seriously influenced by student performance. What is important within this part of the discussion is that reporting environments which create simplistic links in which student performance serves as a key indicator for teacher and school performance. At times this creates a dynamic where personal, professional value is directly assigned or linked to student achievement on standardized tests. This need not be the case. Supovitz, J. (2009). Can High Stakes Testing Leverage Educational Improvement? Prospects from the Last Decade of Testing and Accountability Reform. Journal of Educational Change, 10(2-3) 211-227;
41. For some examples see, Gillborn, D. & Youdell, D. (2000). Rationing Education, Policy, Practice, Reform and Equity. Buckingham: Open University Press; or Thrupp, M. (1999). Schools Making a Difference Lets Be Realistic! Philadelphia, PA: Open University Press.
42. Chris Argyris, & Donald A. Schon. (1974). Theory in Practice, Increasing Professional Effectiveness. San Francisco, CA: Josey-Bass.
43. See Appendix 1. Childs, R. (2015). Some possibilities for administering the measures. Accessed November 3, 2015.
44. Ibid.
45. See Appendix 2. Measuring What Matters: School Field Trial Summary and Description.
46. Ibid.
47. MCYS (2012) Stepping Stones: A Resource on Youth Development. Queens Park, Ontario, Canada. Link: http://www.children.gov.on.ca/htdocs/English/documents/topics/youthopportunities/steppingstones/SteppingStones.pdf. Accessed November 3, 2015; and
48. Ministry of Education (2014). Achieving Excellence: A Renewed Vision for Education in Ontario. Queens Park, Ontario, Canada. Link: http://www.edu.gov.on.ca/eng/about/renewedVision.pdf. Accessed October 15, 2015.
49. People for Education. (2015) Advisory Committee.

Possibilities for Administering the Measures
By Ruth Childs, February 6, 2015 draft

Click for table

Measuring What Matters – School field trials: Summary and description

Overview

Over the past 12 months, People for Education has partnered with lead education scholars and completed a wide public consultation to produce a draft set of student competencies and learning conditions for the Measuring What Matters (MWM) project. MWM is proposing broader goals and measures for education across five domains: creativity, citizenship, social-emotional learning, physical and mental health, and quality learning environments. Moving forward, the next phase for the MWM project is the school field trial study. In this phase, People for Education will establish partnerships with schools and school districts to assess the usefulness of the identified student competencies and learning conditions.

The next phase of MWM project: School field trials

The school field trials are framed by 5 key questions:

1. What are the implications of using the five domains—creativity, citizenship, social-emotional learning, physical and mental health, and quality learning environments—and related competencies and learning conditions as a framing and evaluation tool?
2. What are some interrelationships between and across the five domains, as expressed in school and school district practices?
3. How and in what ways do the definitions of the competencies and conditions articulated in the MWM framework translate into learning experiences in classrooms?
4. What are some potential implications of the local findings from the school field trials on the development of a central set of broader student competencies and a central set of measures?
5. What is the relationship between ongoing classroom measurement or assessment (both formative and summative) and large-scale assessment through student testing? How do they influence (constrain and enhance) each other and their distinct purposes in articulating student achievement in broader areas of student learning?

School field trials: Getting involved

The central objective of the school field trials in the Measuring What Matters (MWM) project is to integrate at least some of the competencies and conditions defined in the MWM framework into current research, practices, structures, and collaborations in participating schools and school districts. For example, the domains may already be incorporated into schools’ and school districts’ planning, use, and evaluation of areas such as student well-being, 21st century learning, Dweck’s growth mindset, or Fullan’s New Pedagogies for Deep Learning. These areas of work are all centrally related to the domains articulated above.

Schools and school districts participating in the MWM project would use the competencies and conditions in the five domains to understand the ways in which they support existing work. Participation in this project will include the use of the MWM framework, allowing educators to create a common language that defines what the student competencies and learning conditions are and how they might be supported, understood, and evaluated in daily practice. While participating schools and school districts would not necessarily integrate all of the competencies or learning conditions described by the MWM project, the following provides a description of the protocol for each case site (i.e., schools and school districts) involved in the project.

Protocol: Roles and activities

Roles: school and school district levels

• Local facilitator: the key contact for People for Education within a participating school or school district (e.g., principal, teacher, teacher-librarian, school coach, district researcher or external researcher); the onsite collector who co-selects key pieces of information gathered, synthesized, and processed into a single narrative.
• Participating educator(s): observes and documents moments of relevance in relation to the competencies and conditions selected across the five domains for the project.
• School team(s): comprised of participating educator(s) who may plan a unit, activity, ongoing lesson or integrate the competencies into existing plans (e.g. summary classroom discussions, small group work) in a way that will focus the classroom work to an area of observation and action for the teachers relating to the MWM project.

Activities: data collection and analysis

We propose that the individual case sites will work through the following protocol:

• Selection of competencies and conditions:
  • Local facilitator and participating educator(s) meet to review competencies and conditions across the five domains.
  • The educator(s) selects a relevant set of specific competencies and conditions in relation to their current work. Note: the quantity of competencies and conditions selected and the domains in which these items exist will vary.
    • E.g., educators may select a competency from creativity that describes the ability to use peer feedback to improve a product with a competency from social-emotional learning that describes students’ ability to work adaptably and flexibly.
• Collection of information and data:
  • A variety of documentation techniques can be used to collect information about the student experiences (e.g., sticky notes, audio or video recordings, student work, student interviews or notebook recordings) designed to jog their own memory.
  • Information needs to be regularly collected (e.g., weekly), while not overwhelming the participating educator(s).
  • Information collected should enhance assessment and planning demands of educator(s), as well as, inform immediate next steps educator(s) take.
• School team meetings:
  • Periodic team meetings will serve as sources of information/data sharing and points of analysis in the project’s process.
  • The development of new understandings and potential actions across educators will occur here.
  • These meetings will also serve to anchor group and individual understanding across participating educators by revealing what the relevant competencies under study mean; how they differentiate in understanding; and, how they develop a contextual ‘sense’ of the competency in relation to each educator’s classroom.
• Development of the single narrative:
  • The local facilitator will co-select information from participating educator(s) and school team(s) to build a single narrative.
  • The single narrative summarizes and defines core parts of the process describing the inquiry and its relationship to the five domains, as well as, the related competencies and conditions being studied.

School field trials: Results and applications

The information generated from the school field trials will produce three key pieces of information in response to the 5 questions (above):

• Refinement of student competencies and learning conditions:
  • Information from the schools about the value of the competencies and conditions as a framing and evaluation tool for addressing the broader areas of success will be used to re-fine, adapt and re-draft the competencies and conditions where relevant.
• Understanding the interrelationships across the five domains:
  • The school field trials will provide descriptive cases of classroom and educator experiences for the MWM competencies and conditions.
  • The research narratives will allow for a local, specific understanding of what these competencies and conditions ‘look and feel’ like in a variety of classroom and school settings. They will also provide deepened understanding, for educators, policy makers and the public, of the competencies and conditions ‘in action’ and/or the school experiences that support the five domains of student success.
• Relating the local field trial findings to the central policy level:
  • The field trials will provide insight into the ways the competencies across the domains in the project interrelate in given student learning experiences. These insights will serve to further deepen understanding at both policy and school/classroom level of the different aspects that make up general student well being.
  • The field trials combined with a variety of other People for Education-led research and consultation activities in 2015-16 will support and deepen perspective and understanding of assessment, evaluation and measurement policy for broader areas of student success at both central and local, classroom levels.

People for Education – working with experts from across Canada – is leading a multi-year project to broaden the Canadian de nition of school success by expanding the indicators we use to measure schools’ progress in a number of vital areas.

Authors
David Hagen Cameron
Elyse Watkins
Annie Kidder

The Measuring What Matters reports and papers were developed in partnership with lead authors of each domain paper. Permission to photocopy or otherwise reproduce copyrighted material published in this paper should be submitted to People for Education at [email protected].

Document Citation

This report should be cited in the following manner:

Cameron, D., Watkins, E., & Kidder, A (2015). Measuring What Matters 2014-15: Moving From Theory to Practice. In Measuring What Matters, People for Education. Toronto: November 3, 2015.

We are immensely grateful for the support of all our partners and supporters, who make this work possible.

Annie Kidder, Executive Director, People for Education

David Cameron, Research Director, People for Education

Charles Ungerleider, Professor Emeritus, Educational Studies, The University of British Columbia and Director of Research, Directions Evidence and Policy Research Group

Lindy Amato, Director, Professional A airs, Ontario Teachers’ Federation

Nina Bascia, Professor and Director, Collaborative Educational Policy Program, Ontario Institute for Studies in Education, University of Toronto

Ruth Baumann, Partner, Directions Evidence and Policy Research Group

Kathy Bickmore, Professor, Curriculum, Teaching and Learning, Ontario Institute for Studies in Education/University of Toronto

Michelle Boucher, University of Ottawa, Advisors in French-language education and Ron Canuel, President & CEO, Canadian Education Association

Ruth Childs, Associate Professor, Leadership, Higher and Adult Education, Ontario Institute for Studies in Education/University of Toronto

Jean Clinton, Associate Clinical Professor, McMaster University, Dept of Psychiatry and Behavioural Neurosciences

Gerry Connelly, Director, Policy and Knowledge Mobilization, The Learning Partnership

J.C. Couture, Associate Coordinator, Research, Alberta Teachers’ Association

Fiona Deller, Executive Director, Policy and Partnerships, Higher Education Quality Council of Ontario

Kadriye Ercikan, Professor, Measurement, Evaluation and Research Methodology, University of British Columbia

Bruce Ferguson, Professor of Psychiatry, Psychology, Dalla Lana School of Public Health, University of Toronto; Community Health Systems Research Group, SickKids

Joseph Flessa, Associate Professor, Leadership, Higher and Adult Education, Ontario Institute for Studies in Education, University of Toronto

Joan M. Green, O.Ont., Founding Chief Executive O cer of EQAO, International Education Consultant

Andy Hargreaves, Thomas More Brennan Chair, Lynch School of Education, Boston College Eunice

Eunhee Jang, Associate Professor, Department of Applied Psychology & Human Development, Ontario Institute for Studies in Education, University of Toronto

Christopher Kotz, Senior Policy Advisor, Ontario Ministry of Education

Ann Lieberman, Stanford Centre for Opportunity Policy in Education, Professor Emeritus, Teachers College, Columbia University

John Malloy, Director of Education, Hamilton-Wentworth District School Board

Roger Martin, Premier’s Chair on Competitiveness and Productivity, Director of the Martin Prosperity Institute, Rotman School of Management, University of Toronto

Ayasha Mayr Handel, Ontario Ministry of Children and Youth Services

Catherine McCullough, former Director of Education, Sudbury Catholic District School Board

Robert Ock, Healthy Active Living Unit, Health Promotion Implementation Branch, Health Promotion Division, Ontario Ministry of Health

Charles Pascal, Professor, Ontario Institute for Studies in Education, University of Toronto

Jennifer Riel, Associate Director, Desautels Centre for Integrative Thinking, Rotman School of Management, University of Toronto

Joanne Robinson, Director of Professional Learning, Education Leadership Canada, Ontario Principals’ Council

Bruce Rodrigues, Chief Executive Officer, Ontario Education Quality and Accountability Office

Pasi Sahlberg, Director General, Centre for International Mobility and Cooperation, Finland

Alan Sears, Professor of social studies and citizenship education, University of New Brunswick

Stuart Shanker, Research Professor, Philosophy and Psychology, York University; Director, Milton and Ethel Harris Research Initiative, York University; Canadian Self-Regulation Initiative

Michel St. Germain, University of Ottawa, Advisors in French-language education

Kate Tilleczek, Professor and Canada Research Chair, Director, Young Lives Research, University of Prince Edward Island

Rena Upitis, Professor of Education, Queen’s University

Sue Winton, Assistant Professor, Faculty of Education, York University and former Early Learning Advisor to the Premier Deputy Minister of Education