Outcome monitoring in Strong Voices 

21 April 2015

(Image credit: Simon Way for A New Direction)

The Strong Voices programme was a national programme funded by the Department for Education to engage vulnerable and disadvantaged 13-19 year olds in arts and culture. A national group of lead organisations (5 independent arts and creative learning organisations) explored the role the arts and cultural infrastructure can play in improving the lives of the 6% most vulnerable young people in England through early interventions.

As part of this programme A New Direction looked at how London’s arts and cultural sector can come together with organisations that work with disadvantaged young people to ensure equity of access. We did this by working with London Youth and four core youth clubs; The Avenues Youth Project, Dragon Hall, Haringey Shed and KORI. Art forms explored throughout the programme included theatre, visual art and textiles printing, musical theatre, photography, forum theatre, physical theatre, music and music production, dance and choreography, music video production, videography, spoken word and film. You can read more about the programme here.

The framework

As part of Strong Voices, we sought to develop a monitoring system that would enable us to track young people’s progress throughout the programme.

We decided to use social and emotional capabilities outlined in The Young Foundation’s Framework of Outcomes for Young People to create an ‘Outcomes Web’. The framework is designed to highlight the fundamental importance of social and emotional capabilities to the achievement of all other outcomes for all young people. The Young Foundation developed this framework on behalf of the Catalyst consortium and addresses the key challenges in measuring impact on the lives of young people. We felt that the social and emotional capabilities it highlights are reflected in both arts and youth work interventions, but that a common language was needed to reflect this.

The Young Foundation’s report gathers evidence that shows a clear connection between social and emotional capabilities and positive life outcomes, such as educational attainment, employment, health and behaviour. As The Young Foundation point out, the Government’s Positive for Youth strategy states that the process of personal and social development includes developing social, communication, and team working skills; the ability to learn from experience, control behaviours, and make good choices; and the self-esteem, resilience, and motivation to persist towards goals and overcome setbacks.

Through research into and consultation on the above outcomes, The Young Foundation proposed a model of seven interlinked clusters of social and emotional capabilities that are of value to all young people. These are:

• Communication
  
  

• Confidence and agency

• Planning and problem solving

• Relationships and leadership

• Creativity

• Resilience and determination

• Managing Feelings
  

The Tool

Strong Voices focused on positive and sustained personal and social development, and whilst this was evidenced through our Action Research, we needed ‘harder’ quantitative evidence of the difference our interventions were making. As with all personal development, the observation of progress requires a good measurement of the baseline. We therefore chose to implement a consistent approach using a quantitative tool to capture young people’s progress across the programme.

Using the framework of capabilities enabled us to more clearly articulate the value our programme produced for the young people involved. It worked for our cross-sector approach because it charts outcomes that both the youth and arts sectors seek to influence when addressing young people’s development. Outcome webs are already used by some organisations to measure change when working with young people, but we felt that interweaving this with the framework aligned the sectors and our aims more closely.

In order to measure any changes taking place, young people filled in an outcomes web at the beginning and end of each quarterly programme. Each point of the web represents one social or emotional capability, and we added arts engagement as the eighth point to encourage the young people to reflect on their engagement within the programme and their creativity (in terms of innovation and learning) as two separate points. They would rate themselves between one and five to indicate where they place themselves on the scale.

A key explaining each capability and web point was available for the groups, but it was essential that the youth workers catered the process to the group and individual’s abilities (Outcome web key downloadable at bottom of page?). For example, when working at The Squad, an inclusive youth group for young learning disabled people, a Dragon Hall youth worker used a visual method to represent the stars. He broke down the language to the young people’s literacy levels, demonstrated this visually and one to one support was offered. Each young person was assisted in reflected on their own abilities through targeted questions before indicating where they would place themselves.

It was important that young people were able to track their own progression in a standardised way that could be collected to show the general percentages of increase and decrease across the programme. The percentages take all progression into account, so even if a young person was moving from a one (lowest) to a two, this was still considered progress. This was in order to allow for each young person to undertake their own journey of progression.

Specific Findings

All findings come with the caveat that numbers were relatively small, with approximately 50 young people worked with per quarter

• ‘Planning and Problem Solving’, ‘Creativity’ and ‘Arts Engagement’ were consistently highest.
• By the last Strong Voices quarter, 67% of young people felt their ‘Arts Engagement’ had increased.
• By the last quarter approximately half of all young people involved stated increased ‘Communication’, ‘Confidence and Agency’ and ‘Planning and Problem Solving’ (47%, 53%, 48% respectively).

• Different kinds of interventions naturally had different outcomes. ‘Confidence and Agency’, ‘Planning and Problem Solving’ and ‘Resilience and Determination’ saw larger uplifts in our residential retreat quarters, for example. This is particularly interesting because they are very transferable skills. This could be because the residential retreats provided an opportunity to work intensively with young people over a smaller period of time. It could also be because the mix between outdoor education at residential centres coupled with arts delivery will have influenced the outcomes.

• The longer the project, the more varied the responses. Rolling two quarters into one, for example, saw more generalised outcomes, rather than certain areas standing out. This could potentially be due to the young people not remembering how they felt at the beginning of the project (6 months apart) and therefore not identifying large increases in their capabilities.

• ‘Relationships and Leadership’ and ‘Managing Feelings’ consistently saw less progression. Whilst only speculation can be had as to why young people felt these areas increased less, it is interesting to note that.

Conscious Competence

The "conscious competence" learning model explores four stages of competence and the process of progressing from incompetence to competence in a skill. We realised that this model was taking place when some young people were declining in certain capabilities rather than increasing. Through our interventions, they were undergoing the process of identifying that they were perhaps not as good at team work, for example, as they thought they were.

Our aim was to enable young people to move from Unconscious Incompetence (where they don’t know what they don’t know) to Conscious Incompetence (know what they don’t know) – this is indicated through decreased Outcome Web percentages. Ideally, the young people then move from Conscious Incompetence to Conscious Competence; aware that they are improving in social and emotional capabilities. The final stage is Unconscious Competence, where a young person is able to use all 7 seven core social and emotional capabilities without having to think about it.

The highest percentage decreases in our programme indicate that these are competency areas young people originally had the least awareness of and then identified as point for improvement.

Early on in the programme, young people decreased in ‘Planning and Problem solving’, ‘Managing Feelings’, and ‘Relationships and leadership’, indicating that these are areas they learned they need to focus on.

Our Learning

Our two year Strong Voices programme was long enough to pilot the outcome webs as a method for gathering data on outcomes for young people. This time allowed for significant learning that could be used to implement the webs in a longer term programme.

• ‘Buy in’ from the professionals involved was necessary for the tool to be integrated in their facilitation approach. We therefore organised training sessions for the youth workers involved, so that they understood the tool, its origins and the importance behind the social and emotional capabilities. The settings that embraced the tool and structured a creative session around it were the most successful in engaging the young people to consider their own progression.

• Capacity is limited in the voluntary sector and so it was necessary to budget for time for the youth workers to fill the outcome webs in with the young people.

• Capturing results over a six month period with a consistent group of young people and midpoint outcome webs provides the most reliable results and the clearest trajectory with conscious incompetence moving to unconscious competence.

• In future, tracking outcomes related to specific art forms and groups of participants would provide interesting information in order to compare interventions.

Conclusions

Testing the outcome webs over two years has left us with a few thoughts. Firstly, that a common language across the arts and youth sectors is useful for articulating outcomes. We know that funders and commissioners like to see ‘harder’ outcomes as well as the case studies and soft outcomes that our sectors are so good at sharing. Using tried and tested frameworks enables both sectors to work more cohesively under aligned aims and visions, thus allowing us to know when a programme is successful and why.

Secondly, it enabled us to question what it is about arts interventions that can affect young people’s life outcomes and to test how specific methods (such as arts based residential retreats) influence young people’s social and emotional capabilities.

All these ideas and themes will continue to permeate AND’s work

Our new research report from Public Perspectives, ‘Disadvantage and Cultural Engagement – a study into the lives of young Londoners’, explores the complex and multi-faceted notion of disadvantage by looking at four groups of young people facing specific forms of disadvantage - young people who are looked after, young carers, disabled young people and those at risk of gang activity. The research highlights the importance of social capital (i.e. social networks such as families, friends, schools) in building cultural interventions that genuinely engage disadvantaged young people and can be sustained over time.

You can also read detailed findings of all our research and conversations related to our Cultural Capital Inquiry (into inequality of access to culture for children and young people in London). We are inviting comments which help contribute ideas for further research, for campaigns and action that can help build more equal access to the arts. If you would like to contribute to the discussion, email us or tweet via the hashtag #ANDCulturalCapital