Evaluation and Learning Partnership

Diverse Voices

The Client

Diverse Voices is a leading provider of education and performing arts education for young people. Its main aim is to train and inspire young people in performing arts to realise their creative potential, improve life skills and explore sensitive and important issues.

The Challenge

Building on a 2019/2020 pilot project supported by the Paul Hamlyn Foundation, Diverse Voices is now running a 3-year programme. This programme comprises e-Learning, Theatre in Education (TIE), and Performing Arts Training and Workshops. The aim of the programme is to explore online safety and relationship education for children aged 9 – 11 across ten primary schools in the London Boroughs of Waltham Forest, Hackney and Islington.

The Project

inFocus was commissioned to be the Evaluation and Learning Partner for this programme, leading and guiding on the development and deployment of data collection tools. InFocus also developed and facilitated training workshops on the data collection tools to prepare a cohort of young people working for Diverse Voices to carry out the data collection process. The workshops dedicated time to refinement and appropriateness, as well as valuing and incorporating the views and feedback of the young data collectors back into the tools.

The Deliverables

The tools were deployed at three points during an academic year: before programme implementation to establish a baseline, after programme implementation to measure any changes, and towards the end of the academic year to determine any long-term results. As the tools were implemented throughout the year, the inFocus team incorporated the personal, social, health and economic (PSHE) expertise of both the Diverse Voices team and the data collectors to draw lessons from each round of data collection in an iterative process. This led to changes to the session setup, rephrasing of questions, and re-focusing on particular messages. In the case of the provision for SEND children, the team also utilised feedback received from facilitators to make the tools as accessible as possible.

The data collection kit (developed by senior consultant Tom Keyte and consultant Iulia Costache) comprised of a guide for the data collectors, a worksheet for children and young people to fill out, an accompanying slideshow to act as a visual guide for children and young people and print-outs for some of the exercises.

Child-Safe Evaluation Considerations

Throughout the evaluation, the inFocus team kept a focussed eye on maintaining a high level of safety and anonymity for Diverse Voices’ participants. In one such example aimed to measure the prevalence of online abuse and harassment within the classrooms, the two questions developed for this part of the evaluation were the following:

Have you or anyone you know ever received online abuse?
Have you or anyone you know ever received an image you felt was inappropriate?

These two questions were deemed as ‘sensitive’ by nature. Thus, caution needed to be taken to protect the young respondents from any potential harm incurred by the data collection process as a result of providing answers to these questions in front of their peers. A version of the ‘Heads Down, 7 up!’ game was adopted to maintain respondent anonymity. They were asked to place their heads on their desks, close their eyes and raise their hand when they heard the option that most represented how they felt. Facilitators then noted how many hands were raised in the classroom.

The Iterative Process

When developing evaluative tools for this programme, the team at inFocus set out to build upon previous M&E tools & processes used in the 2019/2020 pilot. They also wanted the tools to reflect Diverse Voice’s creative approaches to education. One of their goals was that the children and young people taking part in the M&E process would find their involvement fun and engaging.

The available research suggests that most children 9 years and older are able to complete a written survey. However, they may perceive it as an onerous task akin to schoolwork. This was especially significant to the settings of this programme, since the data collection process would take place in the classroom. Shifting away from traditional data collection methods could help facilitate the process and provide more enjoyment to children¹ (read more on the theory behind engaging children in evaluation here).

The evaluators from inFocus proceeded with this knowledge in mind when developing the data collection methods, opting for a mixture of activity-style worksheets, visual cues and game-style and movement-based exercises.

The first exercise of the first academic year baseline data collection workshop tested this approach. Four images relating to different activities were placed in different corners of the classroom (Gaming, Watching videos/TV, Social Media and Listening to Music) and facilitators used a physical game-like approach to engage students in identifying what activities they participated in most (pictured above). The activity garnered a self-assessment of which online activities students chose to spend most time on average week to week. Follow-up questions were used to determine details about what type of content they engage with and from whom, allowing students to reflect on their online activities. The activity also effectively broke the ice and helped the respondents become more comfortable around the data collectors. Utilising this physical movement approach was adopted multiple times at different points during the workshops to diversify the evaluation activities and keep students engaged.

The Learnings

In practice, however, it was found that the inclusion of too many physical elements into the data collection process slowed things down considerably and the data collectors found it difficult to settle the class back into the written parts of the evaluation. The inFocus team applied this feedback into subsequent iterations of the data collection tools, still retaining some physical elements, but reducing their number and carefully considering when during the workshop they should be implemented. They also incorporated other methods, such as using vignettes in the form of short films that referred to previously taught material and asked hypothetical questions to gauge comprehension and retention of the curriculum (image below).

This attention to detail and iterative cycle was maintained throughout the entire evaluation project, allowing the inFocus team to assist Diverse Voices in adapting the evaluative process in order to produce the best possible results by the end of the project.

¹ McDonald & Rosier, (2011). Collecting data from parents and children for the purpose of evaluation: Issues for child and family services in disadvantaged communities. Australian Institute of Family Studies.