Case study: creating VR experiences with PMLD students

23 June 2021

In the lead up to I Am Festival 2021 – our yearly celebration of creativity with D/deaf, disabled, and neurodiverse young people – filmmakers and practitioners Justin Allder and Richard Elson worked in collaboration with staff and students at the Vale School in Tottenham to create an immersive, multi-sensory experience using 360° video and virtual reality. The project was aimed at young people with Profound and Multiple Learning Disabilities (PMLD) and, following the I Am Festival theme of 'love', explored the love of nature.

Aims and objectives

• To place the young people at the heart of the action and bring the natural world and its associated benefits into the classroom
• To explore the impact of a virtual natural world on young people's mood and stress levels
• To explore what interactive possibilities could be developed within the experience

Outline

This was an intensive three-day project with a group of seven young people with PMLD. Each student had three individual sessions over three days building to a bespoke VR experience on the final session.

looking at pictures of green scenery in nature has been linked to stress relief, better impulse controln and improved focus

Gamble KR (2014)

Session 1

An immersive installation of the woods with still images

The aim of the first session was to prepare the young people for their VR experience by creating an immersive installation using still images and projectors on the theme of woodlands. The theme of woodlands was chosen as the setting for the initial experience because the colour green provided a low-intensity environment that was both familiar to and safe for the young people.

The young people were shown a series of images of woodlands on a single screen in front of them and then additional screens were added on either side until the student was enclosed and immersed in the experience. The projected slide show of still images was accompanied by a relaxing ambient soundtrack and then repeated a second time with a more natural soundscape of the woodlands.

All of the young people were naturally wary of us and the screens and took time to settle into their new environment. However, as the images progressed they adapted to the new experience and became less agitated and more focused on the images and sounds.

Initially, their focus was on the main screen in front of them but as they relaxed they slowly began to explore the screens and environment all around them. This was particularly apparent when we added the more natural soundscape of birdcalls in the trees above. Many of the students responded physically and vocally to the changes in scene and the different sound effects.

One consistent reaction was a definite change in the young people's mood as a result of their immersion in the digital environment. It was observed that nearly all of the students became calm and focused during the experience. This would seem to support psychological studies that show that spending time in natural green environments or even looking at pictures of green scenery in nature has been linked to stress relief, better impulse control, and improved focus.

Creating the virtual experience

To assess reactions to the experience the project was split into two scenarios: a non-interactive journey into woodland and an interactive scene with large, bright objects to select.

The positive reactions of the students to movement and change led to a decision to add in animals and associate them with sounds. In case there was time in the sessions, an interactive element was added to the woodland scene as well, this time with a smiley face reward.

The interaction was based on 'choose your own adventure' books where the reader has the chance to choose from a series of decisions at various points to steer the direction of the story.

NON-INTERACTIVE JOURNEY INTO WOODLAND

The woodland journey used a combination of graphical colours, stock footage stills, bespoke filmed woodland landscapes and stock footage of animals (a seagull, rabbit and deer), and the experience gradually built in intensity.

A day's shoot with 360° cameras in Wimbledon Common was used to obtain footage of woodland and lake environments. Once all the footage was aquired it was edited together with graphical elements and animals overlaid on top of the video. This was then exported and loaded onto VR headsets.

The video is 360° so can be viewed all round. To do this on a phone or tablet, simply move the device around. To do this on a laptop or desktop, scroll around by clicking and moving within the video window.

INTERACTIVE EXPERIENCE BY LAKE

For the interactive experience, a simple scene was filmed in woodland by a lake. Overlaid onto this were three distinct, large objects (a red petal, an orange bucket and a blue-purple pigeon) To provide a reward for looking at them a graphical and aural reward was added, a circle with a thumbs up and the sound of a tinkle of bells.

This is also in 360° but it is not possible to access the interactive elements.

To create the interactive elements, Dutch company Warp VR kindly let the project use their software to load in the separate videos and link them together. The headset user could then change the video by looking at certain points on the image.

This worked very well for the test, but limited the selection area to a choice of three sized circles and did not have the option of an on screen counter that added up the objects spotted. To explore these possibilities a software engineer from One Dead Pixel was commissioned to test out something bespoke, which you can see here.

This demonstrated the possibilities of having a counter add up on screen and allowed for any shape or size image, but had the drawback of only being viable on still images and accessed via a weblink within the headset. The developer tried various methods of working with the video but each required it to be streamed and could not be achieved at a fast enough rate. For bespoke video experiences, a new solution would be needed.

Session 2

Immersive installation of the woods with moving images and soundscapes

In the second session, the still images were replaced with film clips of the woods. These clips included complicated moving and tracking shots and also included close-up and slow-motion footage of the various animals as an additional stimulus.

All of the young people were able to transition quickly and smoothly from the classroom to the installation and again showed no signs of distress or agitation. Once again, there was also a shift in their moods as they quickly became calm and relaxed, and some of the group even showed signs of anticipation as they waited for the experience to begin.

At this point several of the young people were selected to trial the headsets prior to the final session. This gave the young people time to acclimatize to the VR headsets and highlighted potential problems when wearing the headsets. We were also able to come up with creative ways to overcome resistance to wearing the headsets, like wearing swimming goggles, watching the VR films through cardboard headsets, looking through binoculars, etc – all of these activities could be done slowly in class prior to wearing the headsets. It also provided an opportunity to trial the interactive experience and make any final adjustments prior to showing it to the rest of the group.

Session 3

VR headsets with woodlands setting

In the third session, the young people were shown the film from the previous session. After they were relaxed and engaged they were introduced to the headsets. Having the films projected in the background helped them to make the transition from looking at the images on the screen to watching similar landscapes and images in the headset.

Each of the young people were shown the 360 woodland experience that which allowed them to use sounds to locate and find different animals in the woods. The experience was screen cast to a mobile device so staff could watch what the students were looking at in real time and observe if they were able to find and then sustain their focus and attention on the different animals and birds that appeared in the woods.

By the end of this final session, five of the seven children were happy to have the headsets on. The remaining two young people still engaged well with the images and sounds around them.

Lessons learnt

• The importance of carefully scaffolded activities and the use of repetition to prepare young people with disabilities for a VR experience
• Tailoring the experiences to individual needs
• The importance of the use and impact of color when designing VR experiences for young people with PMLD
• The heightened impact of visual grammar when creating VR films for young people. For example, the young people in the project showed a notable physical reaction to cuts as opposed to dissolves between different images
• The potential possibility to create interactive experiences that enable the young people with PMLD to have agency and choice

Feedback

Child C: This child responded extremely well to the whole experience... I (as his teacher) was anxious about trying the headset on him ... I was so very pleased to see how easily it was put on and how easily he accepted it. He was very relaxed about it. He didn’t try to take it off at all. He touched the head set a few times but this was more to do with exploring what was on his face as opposed to feeling anxious about it. He kept it on until it had finished. When the adults took it off he reached for it to have another go. He had three turns because he enjoyed it so much. We could see what he was looking at and it was clear that he was spotting the animals.

Child B, Exploration: The headset was offered to her this time without the build up from the space in the sensory room so it was taken a little bit out of context for her. She was still willing to put it on. It was offered to her very gently which helped her feel safe enough to put it on. She kept it on for about one minute and turned her head slightly to look at the images. She put it on again a second time kept it on for less time. Persistence: The second time she took it off more quickly and made it quite clear through her facial expression and body language that she didn’t want it on.

Child D: On his first turn in the space surrounded by images and some sound he was relaxed... The first time the headset was offered to him he rejected it very clearly. The next day when he was offered it again he did allow it to be put on. He was relaxed about it and didn’t try to take it off. You could see that he was looking at the images around him. They had created some images of fish and a diver underwater as his favourite song is 'Slippery Fish'. He responded really well to this. Lots of smiles and happy vocalisations.

Future work

Lucy Brunnen, the PMLD lead at Vale School, and other staff have discussed the VR project and identified the following possible areas for development:

• Images of the children themselves for them to turn and look at
• Images of their friends: The adult could ask “Can you see (e.g. Fred), can you look at Fred?” The faces could be in bubbles that appear so the children have to turn their head to find them. The amount of friends could build up so the children with the headset would have to choose who to look at. The adult would know from the screen the headset is casting to what they were looking at and know if it was the correct image when the friend was named
• Counting: Asking the children to look at objects, such as five fish. The experience would count how many images the children looked at and reward them in some way when they have looked at all of them. Perhaps each time they look at the image it could change into a number.
• Interactive story: For example, adapting 'Going on a Bear Hunt' as a story for VR. The children could be in the "swishy swashy grass" or through the storm. They could possibly spot the family members on the walk as though they were on the walk with them.
• Choice-making activities: If software was available that was easy enough for staff to learn they could set up different choice-making activities on a daily basis. This could be a specific activity to learn how to make choices or to make a genuine choice e.g. during music to say which instrument they want. For instance: the children see a tambourine and bells and are asked to look at their choice with a sound of the image triggered when they look at it.

Where next?

I Am Festival 2021 – find out more

Reset – our pandemic support programme

our work with schools

Image credits: Richard Elson