Bonsai: an augmented reality weather vane

Bonsai is an augmented reality weather vane, conveying weather data in a visual format. Created by the team at Voyant AR, it was deliberately designed to explore a potential future state of an augmented reality (AR) user experience.

Today’s AR experiences on mobile devices are confined in many ways from being a truly immersive experience.

    • Field of view is restricted to the rectangular frame of a smartphone or tablet.
    • User inputs are via touch screen.
    • At least one hand is required to hold the device while in use which results in user fatigue and discomfort over long periods of time.
    • Although voice and gaze user inputs are possible with today’s mobile AR technology, to date these have not been utilised widely in applications that are currently available via app stores.

But the future state of AR will look (and feel) very different.

Major established brands (Apple, Samsung, Google) as well as startups (Meta) have either patented or rumoured to be working on lightweight glasses that will deliver AR content to mainstream consumers at (hopefully) an affordable price point. Other companies such as ODG and Epson have had lightweight AR glasses for some time but applications have been predominantly for the corporate sector. In December of last year Magic Leap announced a creator/developer edition of it’s lightweight headset.

MagicLeap01
Magic Leap Creator Edition.

Lightweight AR glasses is an important step change for the delivery of AR content. It will enhance user experience by:

    • Increasing the user’s field of view for a more natural and immersive experience.
    • Allowing the user to experience AR for longer periods of time (potentially all day).
  • Providing additional interaction methods such as hand gesture recognition as well as voice and gaze.

The team at Voyant AR considered these factors and how the evolution of AR hardware may influence the design of AR content and user experience (UX). As a potential use case, we wondered how weather data might be served, in a future where everyone is wearing lightweight AR glasses all the time?

Bonsai: a concept design for the future state of AR UX

Imagine one morning, waking up at home. After a few moments, you reach out and pick up your AR glasses. You put them on and get out of bed. Walking through your living room, you notice a bonsai tree on your coffee table, resting on a stack of books.

Figure 1: Mockup of an AR bonsai.

Bright green foliage and emerging pink blossoms remind you it’s spring. But grey clouds hide the Bonsai’s highest branches and there’s a slight mist of rain, foreshadowing the need for you to pack your umbrella and coat should you venture outside today.

Using your hand to gesture over the tree, a display is triggered, hovering in the air with today’s date and temperature. Alternatively you could ask what the weather is, just as you would with today’s voice activated assistants such as Siri, Alexa or Google Home, but instead of hearing the weather forecast spoken aloud, you could “see” it via 3D weather animations.

This future state design presents a frictionless user interface. There is no app to “open” per se. The bonsai is “always on”. It is always located on the coffee table in your lounge room. Of course, you could choose to relocate it elsewhere; the kitchen bench, bathroom counter or hallway table. Or make it bigger and place it in your courtyard (for a beautiful tree, you never have to water) as per Figure 2.

Figure 2: Mockup of a large-scale AR bonsai.

This concept design proposes several key features for the future state of AR.

The app is “always on”

You don’t need to turn it on or off each time you want to access content, the way you would with today’s apps. It simply “lives” in your environment and displays or changes depending on contextual-awareness (more on this later). Of course, you could also hide it temporarily when you wanted to view other content.

Aesthetics meet functionality

If an AR app is “always on”, it makes sense that its content should be aesthetically appealing to the user. (Otherwise irritation may trigger deletion.) One might want an animated AR bonsai tree because it’s visually appealing in your home and easier than looking after a real one. But if we extend this notion, AR will also allow designers to ascribe practical function to beautiful objects – both AR and real. A sculpture at your front door reminds you of a colleague’s birthday, a candle holder signals a room’s ambient temperature, or a lamp displays the time.

Frictionless UI

The future state of AR will include content that is so seamlessly integrated with the real world, that it will be difficult to distinguish between what is real and unreal. Thus, the way we interact with AR content should be natural and require little thought or training. Lightweight glasses, wireless headphones/microphones (potentially integrated with glasses), voice and gesture recognition will blend user inputs with AR outputs.

Object permanence

The AR asset is anchored to a specific location in your home. If you left the lounge room, it would still be sitting on your coffee table. Unless of course you chose to take it with you, for example while traveling.

Multiple users and unique instances

The AR asset may live in your home but you choose whether it’s shared and visible to visitors. If the AR asset is a unique instance, one user could alter the asset and other users could see that change. For example, if I “blew” the flowers on the bonsai my visitors would see them sway.

Design and development of “Bonsai” iOS app

To explore the “bonsai weather” concept further, the team at WCMR developed an iOS app prototype with ARKit, Apple’s AR platform.

Our design considerations included the following:

    • Build a native AR app. Our objective was to adopt an “AR-first” design approach. We didn’t want to port an existing mobile weather app design into AR. For example, placing a 2D screen hovering in the air or mounted against a vertical plane such as a wall or fridge. We felt that this would be a missed opportunity of what is possible with AR/ mixed reality. Why place a flat object when we could place a 3D object in the real world instead?
    • Simple user interface (UI). We wanted to design a simple AR interface that would allow the user to interact with AR content in an unobtrusive way.
    • Real time data. The app would pull live location-based weather data from OpenWeatherMap and manifest this information as 3D AR animations.
  • Aesthetics. The bonsai was selected because it is a deciduous tree, allowing time of year to be represented through leaf changes. We also felt that bonsai’s promote a feeling of peace and tranquility, making it suitable for “always on” display.

Prototyping, testing and lessons learned

Initial design

We started with a very minimal design. There was no explicit menu UI. In fact there were no buttons or text at all initially! If we had to introduce any explicit UI, it would only be to facilitate the most essential interactions/ data. Our hypothesis was that users would naturally try to interact with the bonsai.

Figure 3: Bonsai weather AR app design.

Initial development work started with testing ARKit, Apple’s AR platform for iOS devices with an A9 chip or higher, running iOS11 or later. This was our first foray with ARKit and we were pleasantly surprised by how robust it was. Sure, it didn’t work well on plain/untextured surfaces but we knew those were limitations from other developer’s reported experience. (For a more detailed account of ARKit’s technical features, Matt Miesniek’s article is a great reference.) In these situations, we found that the bonsai would hover, float away, jump from spot to spot or scale unreliably (too big or too small). But once a horizontal plane was detected, it was pretty good at tracking from there. (At the time of writing Apple had just released ARKit 1.5, a beta version that includes vertical plane detection.)

ARKit was also fairly good at tracking “session permanence” via visual inertial odometry (i.e. using camera feed and motion sensors to estimate change in position over time). So it didn’t lose too much tracking when you pointed away from the target and then back again.

We tested different lighting conditions and the bonsai model worked well and looked great outdoors in full daylight.

Prototype development

The next step was integrating live location-based data from OpenWeatherMap which had a good API. Once the data was feeding through we created two animation states:

  • Season (summer, autumn, winter, spring) demonstrated through leaf colour, leaf number and blossoms.
  • Weather (sun, clouds, rain, snow). The weather effects took time because they needed to appear “volumetric” (that is, more 3D). Although the sun is a sphere, the clouds are made from a particle system, basically a scattered group of 2 dimensional planes, which each display an animated sprite sheet. Specific “shaders” (code detailing how an object should be rendered/displayed) contributed to its transparent and volumetric effect. The final effect looks quite convincing and realistic when viewed up close.

When both of these systems were working (pulling live data and animations) we programmed animation states to trigger based on relevant data:

  • Season: Real time date + real time location tracking via device coordinates +  seasonal model (correct hemisphere).
  • Weather: Real time date +real time location tracking via device coordinates + BOM data

User testing

We started user testing our minimal UI approach. This produced interesting results. Users loved the AR bonsai  and animated weather effects. However, they didn’t realise that the bonsai was animating live weather data for their current location. So we introduced a location plaque in front of the pot so the user could recognise their current location (see Figure 4).

Figure 4. Bonsai prototype with location plaque added.

Our design included a forecast panel which displayed the current date and maximum temperature. To trigger the display, we had a single interactive element: the bonsai itself. The user had to tap the bonsai (on the device screen) to trigger the display of the forecast panel. Unfortunately, users did not realise the bonsai was interactive and didn’t tap the bonsai (unless by accident).

The issue was designing an appropriate “cue” to signal to the user that there was an interactive element that would trigger an action. (For example, websites have buttons with specific calls to action like “learn more”.) We tried adding a small white sphere that hovered in the front of the bonsai to act as a 3D button (see Figure 5). But users didn’t notice the sphere or didn’t realise that they could interact with it. In this prototype iteration we also introduced shadows for the bonsai, pot and plaque. These looked great especially when viewed in full daylight.

Figure 5. Bonsai prototype with the all-too-subtle white button. But the shadows were looking good.

In the end, we fell back to a more familiar UI from mobile design and placed a “Forecast” button on the location plaque (see Figure 6). It was a concession to our minimal UI approach but given that users were using a touchscreen for inputs, it was perhaps too ambitious given that there weren’t any other cues for interaction (i.e. traditional sticky UI placed at the bottom or side of the screen). We did resist adding a “back button” to the forecast panel and were rewarded by the fact that users naturally tapped the panel to hide it.

Figure 6. Bonsai prototype with button to trigger forecast panel display. (Top: default state. Bottom: triggered state.)

Rather than have the bonsai appear immediately once the app started, we gave the user control over when the experience started and where the bonsai would be placed via a bonsai icon in the bottom left corner of the screen.

For our own convenience, we also introduced an admin panel (accessed via a cog wheel icon in bottom right hand corner) to control the weather settings manually. It never snows in Fremantle so that was the only way we were ever going to see the animation!!

The final experience can be seen in the video below.

Summary

Apple released ARKit in June 2017 and is a major milestone for the development of mobile AR applications. Our experience developing this prototype was very positive. It’s exciting that Apple has made the leap into AR and as some articles suggest have big plans ahead. 

Horizontal plane detection is definitely a step in the right direction. Other AR platforms require image tracking or a QR code to spawn an AR object. But plane detection removes this level of friction (the need to print or have a physical marker) and introduces the freedom for an AR object to appear wherever you like.

Our prototype experience also taught us the value of aesthetics in AR. The bonsai model we used was realistic and lifelike rather than abstract or cartoon in style. Users often moved very close to the object to marvel at the detail of the leaves, flowers and gravel in the pot. When viewed outdoors in full sunlight, the AR shadows were very adept at enhancing that sense of realism particularly when there were other real objects in the same view casting similar shadows. The forecast panel label was deliberately reversed from behind, forcing the user to walk around the bonsai, if they wanted to read it. We realised that users want to be fooled into believing that this AR object was real. (Interestingly, when many people view the above video halfway, they don’t realise immediately that the bonsai isn’t real.)

That’s not to say that there isn’t a place for abstract, cartoon or even 2D objects in AR – indeed there is! – but that we have to consider the context for this content. Our vision of a future state of AR, where the virtual blends in seamlessly with the real world, is not based on AR content looking as real as possible but on the user’s experience with that content. Does it behave in a way that I would expect? For example, sitting on a flat horizontal plane or disappearing behind another object which is in the foreground (object occlusion). Does it react appropriately to environment cues in the real world (shadows cast in the right direction)?

There is much to learn about user interface design for AR. Importantly, how to design for AR without simply replicating traditional mobile or web interfaces. No one wants to see a world of 2D buttons hovering in space. But what’s the alternative? 3D buttons? Or a new design lexicon based on the idea that any (or all) AR objects have inherent interaction properties? Perhaps minimal UI in AR will only come of age and provide the most affordance when users are able to interact with AR assets directly using their hands (e.g. brush their hand over or through an object), voice and gaze rather than through the intermediary of a screen. But that doesn’t mean we can’t start imagining and designing now, what the future state of AR might be like.

Mixed reality for the perfect pilates work out

One of the most obvious use cases for mixed reality is the ability to locate two dimensional displays in space. These days digital screens seem to be everywhere: airport lounges, bus shelters, highway billboards, fixed to the ceiling in dentist rooms. But there are a number of drawbacks including:

  • Cost (initial purchase and electricity).
  • Installation.
  • Fragility.
  • Maintenance.
  • Obsolescence and need for replacement.

So the ability to place a virtual screen wherever you like and at little or no cost, presents an almost infinite range of potential use cases.

But there’s one particular situation I’ve been considering for a while: how mixed reality displays could improve exercise.

Proprioception is the sense of one’s own body. Where our body is located in space, how it moves and the relative position of parts of the body. For example, close your eyes, stretch out an arm, then try to touch your nose. In order to complete this task you need to have an idea of the length of your arm, the angle at which your arm is bent and the location of your nose. But this ability is not just for touching your nose in the dark.

“The ability to sense force, which is known as mechanosensation, provides humans and other animals with important information about the environment; it is crucial for social interactions, such as comforting or caressing, and is required for motor coordination… Similarly, proprioception is considered to be essential for posture and controlled movement, but little is known about the underlying mechanisms and the precise role of this sense.” (From The New England Journal of Medicine.)

Undertaking any new physical activity provides a real work out for not just the body but also the brain. If you’ve ever taken up a new exercise that requires a lot of coordination, you know what I’m talking about. There’s a clumsy dance between your brain and body, as they both try to talk to each other about how to coordinate these new movements.

How can we help this body/brain connection while learning new movements?

We can assist this process by showing the body what it’s doing in real time. Once we’ve been shown how to do something (by an instructor or video) we understand what the correct movement looks like: we just don’t understand what it should feel like. So it makes sense that if we could see what we’re doing, we can compare that image with a mental map of what we should be doing and then make the necessary adjustments.

How do we see what we’re doing? Easy. We can use mirrors. We could also record ourselves but we would’t receive the real time feedback comparing what we see with what we feel.

As humans we’ve long understood the use of this aid. It’s why dance studios have mirrors along their walls. As well as gyms and yoga studios.

I am relatively new to pilates reformer classes but have practiced pilates and yoga on and off for a while. For those who aren’t familiar this is what pilates reformer equipment looks like and some of the associated postures.

Figure 1. Various postures on a pilates reformer machine. (Original images from http://www.pilates-exercises.info)

I know. Looks like medieval torture. But I’ve gained a lot of health benefits like strength and flexibility. I also like the way the exercises force me to use my brain. I have to really concentrate. Part of this focus comes from watching my body in mirrors around the studio, adjusting my posture and movements with what I’ve been shown by the instructor. But this has some difficulties. Let’s take a look at the layout of my pilates studio including the placement of mirrors.

Figure 2. Layout of a pilates reformer studio.

Mirrors are located as follows:

  • One mirror in front of each reformer machine on the North wall.
  • A large mirror on the East wall.
  • A large wall mirror on the South wall located behind a trapeze table.
  • No mirrors on the West wall.

There are a lot of mirrors in the studio but it’s not always possible to see my reflection. This depends on my body position, posture, head angle, location of nearby mirrors and other people/objects in the studio.

In Figure 3, I’ve highlighted a rough estimate of a user’s central gaze in each posture (“near peripheral vision” approximately 30° either side of straight ahead, 60° total field of view).

  • A blue mirror indicates the user can see her reflection.
  • A red mirror indicates they cannot.

 

Figure 3. Central eye gaze of pilates reformer user and visibility of her own reflection in a nearby mirror.

As illustrated, the user can only see her reflection in three postures: A, E and F. Even then, it’s not possible to see her whole body at once.

Another complication is that these postures are not static as the user will move in different directions as per Figure 4.

 

Figure 4. Directional movement of pilates reformer user.

Ok, so today’s pilates reformer setup isn’t ideal.

But how could mixed reality help a student see how her body moves while she works out?

A mixed reality design facilitating proprioception during exercise

(Note, the following designs are based on the user wearing a mixed reality headset/glasses. Ideally the device should not impede movement or cause discomfort during the activity. Realistically, no current mixed reality hardware would be useful in this context as they are either tethered and/or too heavy for prolonged use during exercise. When mixed reality content can be displayed on to light weight glasses or contact lenses, this use case will be more likely.)

Mixed reality displays

Mixed reality offers the use of dynamic displays. Basically, these look like a two-dimensional screen hanging in space. This presents many advantages over a mirror or any type of digital screen in the real world.

A mixed reality display can be:

  • Displayed anywhere. It could be fixed to any plane/flat surface in the real world e.g. wall, table, floor or ceiling. If there are no planes, the display could “hover” in the air at a fixed and comfortable distance from the user within their central eye gaze.
  • Displayed at any size or ratio. The display could fill an entire wall of the studio or be the size of a smartphone screen.
  • Adjusted to move with the user. During exercise the user may move their head and/or body in different directions. The mixed reality display can move with the user to ensure continuity.

For our pilates reformer student, we can position a display at a comfortable distance and within their central eye gaze – no matter what position they are in or which direction they look as illustrated in Figure 5.

Figure 5. Mixed reality displays located within central eye gaze of pilates reformer user.

Capturing the ideal perspective

A mirror can only show a user’s reflection. Mixed reality can show much more. During a pilates class a user would ideally like to see their body from multiple angles without having to turn their head (which in itself could disrupt their balance, posture or movement).  Thus, in an optimal situation we should capture a user’s movement from different angles then select the ideal perspective that highlights overall movement.

We need cameras to capture the user’s movements. The number and placement of these cameras requires consideration.

  • Ideal recording angle and distance. For each possible posture, we determine the best angle(s) from which to view the entire body. Figure 6 illustrates that for a seated posture an ideal camera location is a 90 degree angle at approximately 1200mm height. The x distance depends on where other objects may be located in the vicinity of the user.

 

Figure 6. Camera location for ideal perspective.

As illustrated in Figure 7, the ideal recording angle could be obtained from a camera located on either side (B or C) or above the user (A) depending on the user’s posture.

Figure 7. Multiple camera locations providing alternative recording angles.
  • Camera locations. Reviewing the overall studio layout, we can locate cameras where they facilitate ideal recording angles for each pilates reformer machine. One thing to note is cameras need to be fixed to something. In this case study, I’ve attached them to the wall, ceiling and standing objects (e.g. weight machine, trapeze table).
Figure 8. Layout of pilates reformer studio with potential camera locations.

Camera feeds and content display feeds

Now that multiple cameras are capturing feeds from different angles, we can display these feeds to the user. But which one?

One solution is to allow the user to scroll through feeds and select a preferred view as demonstrated in Figure 9.  (Although this basic mockup uses images of different users it illustrates how the user can see themselves from different angles and make the selection through a rotating carousel format.)

Figure 9. Mockup of basic UI for a mixed reality display. Right and left arrows facilitate navigation through a rotating carousel of live camera feeds.

Interaction design

It’s important to note that in most pilates’ postures, both hands are occupied during exercise. So what types of interactions are possible while wearing a mixed reality headset?

  • Gesture. A gesture-based user interface could be used to access the menu before and after exercise. The user could open the system, navigate through menus and commence “exercise mode”. Once this mode has been engaged, the user must rely on other system inputs.
  • Gaze tracking. This may not be entirely useful during exercise mode as the user’s head will move during exercise. However, like gesture inputs, gaze tracking could be used to open and navigate menus, before and after exercise mode.
  • Voice commands. As long as the interface design is quite simple, voice commands can facilitate navigation through camera feeds and open/close “exercise mode”. In the mockup illustrated in Figure 9, the user could say “next feed” to view the next camera feed (i.e click the right arrow) or “previous feed” to view the previous camera feed (i.e click the left arrow). One drawback is that some users may feel uncomfortable about issuing voice commands in a public space. My experience in pilates reformer classes and gyms is that there is often music playing or people chatting so it might not be too awkward.

Potential issues

  • Unobstructed feeds. Clear camera angles may be difficult to obtain in a busy studio. People and equipment are constantly moving. When reformers are side by side, it isn’t always possible for the user to have an unobstructed view of themselves in a mirror or a camera feed. This may require rearranging the location of machines within a studio.
  • Which camera feeds? The system must be set up in such a way or “smart” enough to know which camera feeds to display to each user. Thus, the user is only presented with angles of themselves. One solution is that the system recognises which reformer machine is in use and therefore which cameras will provide the ideal recording angles.
  • Privacy. Capturing live video feed presents an opportunity to record these feeds for later viewing. This may be useful to individual users to improve their practice. However, all studio clients should consent to recording.
  • Safety. Overlaying a mixed reality display over the real world is a potential safety hazard. Display feeds should not obstruct vision. One solution is for the system to recognise when objects enter the gaze area or within prescribed distance around the user. For example, someone in the real world  “walks through” the display and appears to stand in front of it.

Additional features

The rapidly expanding field of Artificial Intelligence (AI) could take these live camera feeds and provide the user with additional information.

  • Live personal instructor. Monitor a user’s movements and alert them if they move beyond an “ideal” range. For example, when the user’s posture is out of alignment or limbs are in an incorrect position which could lead to discomfort or injury. The alert could be displayed through a visual alert and/or audio message.
  • Guide attention. Highlight within the live camera feed areas on the user’s body which muscles should be engaged for a particular posture. For example, using hamstring muscles rather than back or abdominal muscles.

Future applications

Pilates may seem like a quirky use case to explore but there are serious applications that could benefit from mixed reality assisted proprioception.

Stroke is one of Australia’s biggest killers and a leading cause of disability. One particular problem stroke victims can experience is difficulty planning or coordinating movement known as apraxia. They can also feel slow or clumsy when coordinating movements which is known as ataxia.

In conjunction with a physiotherapist a mixed reality system could become part of a Proprioceptive Neuromuscular Facilitation (PNF) program:

  • Real time feedback. Showing a stroke victim how they walk from different angles as they walk. The visual input helps them make a mental association between what they are seeing with what they are feeling.
  • Mixed reality displays versus mirrors.  Stroke victims may find it difficult to turn their head and look into a mirror while trying to coordinate their own walking movements at the same time. Additionally, it is not always possible to locate large mirrors in many positions around a rehabilitation centre.
  • Voice commands. Stroke victims rely on their hands and arms to steady themselves while learning to walk. Thus their hands are not available to operate a smartphone or other device. While someone else could operate a system on their behalf, voice commands provide stroke victims with autonomy and independence to use the system themselves.
  • Record and analysis. Later viewing can assist a stroke victim to see and understand their own movements. Working with a physiotherapist they can plan future therapy sessions collaboratively.

A ghost game prototype for mixed reality

The 2015 film It Follows is one of my favourite horror films. Besides the disorienting feel, an unsettling soundtrack and measured pacing, it has a very simple premise. Everyday people are turned into objects of abject terror.

In a nutshell and without spoilers, a spirit (IT) takes the shape of regular people. IT follows its victim until IT catches and kills them. IT doesn’t run and scream towards you. IT just walks. Silently. At a very steady constant pace. At first, one can take comfort in that walking pace. There’s more than enough time to run away, right? But it’s the single minded focus with which IT comes towards you which becomes truly terrifying.

it-follows-grandma-1
One of the creepier versions of IT…

Because you soon realise that you can never rest. You must be alert and plan ahead. How much time do you have before IT catches up? How long can you spend at a coffee shop? What if your bus is late? Can you even afford to sleep?

It’s a wonderful riff on the traditional cat-and-mouse game that serves up a tasty dish of impending dread. So of course I thought, how do you turn that feeling into an experience? And specifically, a mixed reality game?

Mixed reality versus virtual reality

The mixed reality universe includes virtual objects that interact with the real world environment (including real objects, people, locations and/or systems). This new technological approach offers a host of opportunities to reinvent a good scary story, above and beyond what could be experienced in virtual reality alone.

  • Enhanced narrative. The object of fear is often a supernatural figure such as a ghost or spirit who can only be seen by the protagonist. This works well in mixed reality as content can be filtered or restricted depending on a player’s level of access. That is, only the player/protagonist can see the ghost. Moreover, seeing virtual objects in the real world catapults that ghost story straight into the player’s own reality.
  • Strange things in familiar settings. A mixed reality experience plays out in the real world, in familiar and natural surroundings to the player. Additionally, the juxtaposition of disparate worlds (natural vs supernatural, safe vs dangerous, familiar vs strange) can disorientate and intensify feelings of anxiety. A few hours ago you saw a ghost in the kitchen. Now you’re making lunch when something flickers in your periphery… but you’re not wearing your headset anymore. Unsettled much? Playing out a ghost story in everyday life can also lull you into a false sense of security. As you muddle through the normal aspects of your daily routine you could be most vulnerable to surprise.
  • Enhanced characterisation of non-playing characters. Supernatural figure(s) are often bestowed with magical qualities that defy what is normally possible. For example, walking through walls, levitating through ceilings, and materialising from everyday household objects. This might be cool in the virtual reality version of a haunted house but experiencing exactly the same event in your own house can only heighten that experience.
  • The internet of things as your supporting cast. It only takes a little fuel to spark the fire of imagination. Combining mixed reality with smart objects can transform the most mundane events into moments of heart stopping terror. The microwave turns on for no reason, hall lights flicker, speakers spontaneously blare creepy music. Horror movies are full of these delicious moments.

So how do we flesh out a mixed reality horror game that can be played in the real world?

Mixed reality game design

Using the basic premise of the film It Follows we can quickly outline some key features of the game design.

  • The opponent. IT is a spirit that haunts a single victim at a time. The beauty of a mixed reality game for “It Follows” is twofold. Not only is the real world environment part of the game space but everyone in it becomes a character by default because the film features regular people as IT. You now have a cast of thousands from which the player must try to distinguish IT.
  • Player Mode. The film denotes IT can only follow one victim at a time. We’ll call this single-player mode.
  • Objective. Evade IT. In the film, a victim can pass IT on to someone else by having sex with them. But for this exercise, we’ll stick to single player format so that we can fully explore how this game can play out in mixed reality.
  • Win/Lose definition: In the film, if IT catches you, you’re dead. Pretty simple loss scenario. But the film doesn’t have a specific win scenario. This is something we can also explore.

But there are other factors to consider with our mixed reality designer hat on:

  • How does the player and IT navigate the built environment?
  • Are there safe zones? What are the rules associated with their use?
  • How practical is this game when the player has to go to work, school etc?

The best way to understand and test different rules and game mechanics is to create prototypes that test various hypothesis. So let’s get started.

Prototype 01: spreadsheet

I modelled basic game mechanics in a spreadsheet to understand the parameters that a single player would face while they were being followed by IT in the real world. I included a lot of assumptions as a starting point.

IT walks at a steady pace. The average walking speed is around 5 meters per second. For this basic prototype, I assumed the player also walks at the same speed. I took a map of my local area, pinpointed several locations and calculated the time to walk between these points to create a player’s typical journey over several hours. I then modelled different walking routes and stopping times to see if the player could evade IT successfully.

it_prototype_01
Spreadsheet model for It Follows mixed reality game.

What I learned:

  • This experience works well as a game. IT travels at a leisurely walking pace but it’s surprising how quickly IT can catch up with you. Yes, if you walk faster you’ll gain some distance but stopping, even briefly, will allow IT to catch up quickly. Even while playing with such a simple model I felt the urgency to move faster and not stop for too long.
  • A starting point. In the film, IT is an entity that is always following someone. Even when IT has been passed on to another player IT just starts following the new player. That is, IT doesn’t respawn in new locations. But in a mixed reality game for single players, I will need to decide where IT is located when the game commences. In my spreadsheet I randomised this element.
  • IT follows you, not your route. My spreadsheet was based on IT following a single player. But after testing a few routes I realised the spreadsheet model had a serious flaw: IT followed the player’s route instead of the player’s current location. The model didn’t take into account that when the player changes direction, IT would also adjust its course so that IT is always heading towards the player, not just following the player’s previous route. My model was too linear. So the spreadsheet needs to be a lot more complicated and include geo coordinates for the simultaneous locations of both IT and the player.
  • Player access to information. While playing with the model I quickly realised I was privy to very important information that impacted the way I played the game. At any given time, I knew exactly how far IT was from me, both in distance and time. I could calculate exactly how long I could stay in one location before needing to move on. But in the movie, victims had no idea where IT was until they had visual confirmation. Even then, they had to distinguish IT from a crowd. This gives the victim only a few minutes warning (if they’re lucky). This is critical for player experience. If a player has too much information they will feel relaxed and in control i.e Not very scared at all. But too little information will make the player feel hopeless and anxious all the time. They might just give up. So I must decide how much information to reveal to players or risk an unbalanced play experience i.e The game will be too easy or too hard.
  • A lot more questions. If the player catches public transport could IT board the same vehicle and keep chase? Does IT ignore the built environment of the real world and just walk through cars and buildings towards the player?

My spreadsheet model, although basic, surfaced a lot of interesting points and questions. I could continue to tweak the model and attempt to answer all these questions but it would take considerable time and not necessarily capture all the nuances of playing this game in real life. So leaning on my experience as an experience designer, I decided it would be quicker and easier to test different scenarios and hypothesis using very basic rapid prototyping techniques.

Prototype 02: Paper

Using hexagonal graph paper, a few coloured pens and dice, I quickly playtested different versions of the game. A few elements remained unchanged between versions.

  • One player was IT (red)
  • The other player was the victim (blue).
  • Each player rolls the dice to determine how far they can travel in a single move. They can move in any  direction.
  • The game ends when IT lands on the same or an adjacent tile to the player.

In each iteration I changed only one parameter to test that feature alone. If I tested more than one feature at a time it would be difficult to discern the impact between them. In the image below, I’ve mapped out all the various iterations and how each play test explored a different game mechanic.

20170210_090531
Prototype/ playtest iteration framework.

During the playtests I played IT while a friend played as the victim. We recorded our moves on paper.

Version A. Both players can only move in a straight line. Victim is caught within 7 moves.

20170103_085201_a-copy
Version A prototype.

Findings:

  • This initial prototype demonstrated within a few moves exactly what the spreadsheet model could not: IT will always turn in the direction of the victim.
  • Despite the initial distance IT catches up to the victim surprisingly quickly.
  • My friend who played the victim (and had also seen the movie It Follows) described how anxious he felt while rolling low dice values and IT continued to get closer.
  • We need to test parameters that mimic playing this game in the real world.

Version B. Test element: safe zones are introduced (green areas) where a victim can enter but IT cannot.

20170103_085343_b-copy
Version B prototype.

Findings:

  • The victim headed straight for a safe zone and tried to stay there for as long as possible. To prevent them from just staying there forever the game needs a game mechanic to draw them out. (In real life this would happen naturally as players have to go to work, school, run errands etc.) We explore this idea further in version D.
  • This prototype allowed IT and the victim to cross paths. So we added a new rule: the victim and IT can’t cross paths in the same move, otherwise the victim will be caught and the game ends.
  • Safe zones in real life have ingress/egress limitations, for example you can only enter via existing points e.g. doors and potentially windows. The ingress/egress points in this prototype are not sophisticated enough to represent the real world. A design iteration could be once the victim has entered a safe zone (e.g. via door A) IT loses track of them. IT would stop and wait at that point. Once the victim leaves the building (via door B), IT becomes aware of the victim’s current location and follows in that direction. This idea is illustrated in the image below. Ideally the victim should use an egress point as far as possible from the original ingress point, forcing IT to travel around the entire building and buying the victim some extra time.
The ingress and egress points of a safety zone. Blue path indicates “victim”. Red path indicates “IT”. Note, while the victim is within the safety zone, IT has no knowledge as to their location (except within the building) and is forced to wait at the ingress point (A) until the victim leaves the building (point B).

Version D. Both players can only move in a straight line. There are safe zones. Test element: the victim must complete an objective (land in all three locations, not just pass through). The game ends when the victim has completed the objective (win condition) or is caught by IT (loss condition).

As you can see from the prototype test below, this was by far the longest game before IT caught the victim.

20170103_085311_d-copy
Version D prototype.

Findings:

  • This had the best game balance. There was a nice ebb and flow between high tension moments when IT almost caught the victim and when the victim was forced to complete objectives that sometimes put them in danger of being caught.

I won’t post all the subsequent versions here but you get the idea. I was learning a lot of from each version of the game. Talking aloud with the other player allowed us to express and document how we felt while playing the game which was really helpful to isolate and understand moments of frustration, anxiety, or confidence.  The next interesting iteration was Version G.

Version G. Test element: IT could always see where the victim was located but not visa versa until they were within 7 tiles of each other (i.e. a minimum distance of 2 moves = rolling the dice and getting 6 + 1). We playtested this feature by placing a card between the two player sheets (like playing Battleship). By this stage of prototyping we had also introduced ingress/egress points for safe zones (noted in yellow).

img_0297
Version G prototype.

The interesting finding from this playtest was that the victim found themselves in a constant state of anxiety. Each move was stressful because they did not know whether they were moving towards or away from IT. When the location of IT was revealed, it was almost a relief. (The situation was likened to knowing there’s a spider in your bathroom: it’s worse when it suddenly disappears and you have no idea where it went.)

Summary of findings

So what did we learn from all these prototypes that we didn’t know before?

  • Being safe does not equal fun. Safety zones are an important game mechanic. But you can’t just hide in safe zones forever. There’s no fun in that. The sweetest tension is when you’re almost caught and the flood of relief when you escape.
  • Push and pull. Players rush into safety zones but need something to draw them out. We introduced specific objectives that players had to meet in our playtests. This mimicked how a mixed reality game could play out in real life as players already have obligations that will force them outside (going to work, leaving for school etc).
  • Where is IT?  The most difficult element to game balance is the amount of information to reveal to players about IT’s current location. In Version G we limited this to 2 dice rolls. In a mixed reality game you could reveal IT’s location in several ways. For example, a player could have one chance each day to know IT’s current location. The player would have to use this opportunity wisely. Knowing IT’s location first thing in the morning would allow the player to calculate how far IT was (distance and time) and plan their movements from then on.
  • Respawning. Once IT catches the player the game is over. Although our playtests had the player and IT start the next game in the same original positions (this was for version comparison) in mixed reality we would program IT to respawn randomly. This prevents a player from predicting IT’s path and then creating an optimal getaway route. Random respawn locations also introduces variety since the player would most likely have the same start point each day (home).
  • Win condition. Perhaps the game does not have a specific win condition. Rather, it could be an endless running game like Temple Run, where the player’s aim is to simply last longer than she did last time and earn her position on a ranked leaderboard.

User Interfaces (UI)

I’ll be exploring mixed reality user interfaces in a future post but for now, I’d like to focus on what the player would see if they were playing this game.

Keeping the UI to a bare minimum has a number of benefits:

  • The game would be played over a long period, so minimal UI allows the player to get on with their daily tasks without distraction or annoyance.
  • It shouldn’t conflict or detract from other mixed reality UI that may be engaged for work or entertainment purposes. (In the future, people will rely on mixed reality heavily to communicate, work and learn.)
  • It lulls the player into a sense of normalcy.

The image below outlines a simple UI for various game states.

The UI displays five game states:

  • Game in play denoted by countdown timer (days, hours, minutes, seconds).
  • IT contact less than 10 minutes: timer turns red.
  • IT contact less than 1 minute: player’s peripheral vision turns red and IT is identified via a red sphere. Audio includes a heartbeat.
  • IT contact less than 15 seconds: player’s vision turns red. Audio heartbeat gets faster and faster.
  • Game over: game over announcement appears.

The key design element is a countdown timer. The prototyping playtests (specifically version G) revealed that the player required a minimal level of information as to IT’s current location for optimum game balance. (Too little and the player didn’t have time to make decisions, too much and IT became too easy to evade.)

Note, these images are only 2D representations of the player’s field of view. The countdown timer could be displayed anywhere within the player’s field of view. In fact, it could be “attached” to any object in the real world. For example, the player could simply tap the back of their hand to display the timer display on their skin.

Mixed reality considerations

Built Environment

This game would be great to play outdoors. In a forest, park, or city centre. Navigating this environment would require IT to walk around objects (trees,  sign posts, cars). Street navigation can also be aided through tools such as Open Street Maps but we also need real time object recognition and spatial mapping of the environment. Even more so if we allow IT to follow the player into indoor public spaces (libraries, shopping malls, museums). We need to recognise the layout of enclosed spaces and objects located within them. Mapping internal spaces is tricky but possible. This has been demonstrated through mixed reality hardware such as Microsoft HololensMeta and Google Tango.

Did anyone lock the front door? (Still from the film “It Follows” via Film-Grab.com).

Following the film narrative, IT can enter rooms via conventional ingress/egress points such as doors or windows, but only when these are open. Within enclosed spaces IT cannot pass through walls, floors or ceilings, navigating only as a regular person would from room to room via internal doors.

Safety zones (areas where IT cannot enter)

The prototypes outlined earlier revealed the importance of safety zones. Essential for not only gameplay but also for integrating the game into the player’s daily life. If you go to school or work, you’ll be based in the same location for at least several hours. This would give IT ample time to catch up with you (even if you gained time earlier via public transport or a car). Translated into a game mechanic, the player could designate several areas as safe zones. As the player levels up, the number of safe areas could decrease and increasing difficulty (i.e. as you become more experienced ,there are less places to hide).

Players’ physical safety

The break out augmented reality hit of 2016 was, without a doubt, Pokémon Go. But the game’s mainstream popularity also created safety hazards for players. The game’s official website outlined Safety FAQs which included a note on physical safety.

“PokéStops and Gyms are created from historical sites, public artwork, and user-designated locations. They exist in many places, including trails, parks, and urban areas. The safety of any given area depends on the user, the time of day, and many other factors. We encourage users to use their own judgment about which parts of the city or countryside they feel safe going to at various times of day or night.”

Pokemon Go. Image by PaintImpact via Flickr.

There have been a number of stories (some perhaps overstated by the media in the hype following game launch and surge in popularity) of players entering dangerous areas, trespassing, or just not watching where they were going because they were too focused on their screens (including walking out into traffic). This was a danger to all players not just younger ones. Designs should incorporate as much context or location-based data as possible to warn players of potential danger. In addition the colour, shape and placement of UI elements within the player’s field of vision should not obscure vision in a dangerous way. This is another benefit of mixed reality over augmented reality.

Pokémon are displayed over your phone’s view of the world. In mixed reality, Pokémon would be an object in the world. For example, the image below shows a Nidoran in front of a dog. A mixed reality version of Pokemon Go would allow the Nidoran to run behind the dog.

Pokemon Go. Image by Albert Hsieh via Flickr.

Unfortunately not all situations can be foreseen by designers. The system was also “gamed” by criminal opportunists who created Pokéstops to lure unsuspecting players into secluded areas. It is important to learn from these early examples of augmented reality games as well as behaviour from massively multiplayer online games (MMOGs). They can help us understand how players might interact with the game and each other, to design systems and game mechanics that minimise risk while ensuring everyone still has fun.

Players’ psychological safety

We have an obligation to our players to consider their comfort and well being, even when they have opted in and provided consent to playing a game like this. Video games have age restrictions recommended by the Entertainment Software Rating Board (ESRB).  Theme park rides provide warnings to people based on height and medical conditions. I believe mixed reality games will also require ratings or adequate warnings to users.

While researching this topic, I really immersed myself and imagined how it would feel to play IT Follows in real life in various situations and locations. Perhaps I had primed myself too well. One night, I went to bed early. I must have heard something because I woke up. I was still half asleep when I rolled over in bed. In the darkness, I saw a figure standing over me. (It turned out my partner had come into the room and been talking to me for a few seconds but didn’t realise I had already fallen asleep.) Well my friends. That scared the crap out of me. This personal experience leads me to believe that your home should always remain a safe zone. Additionally you should only be allowed to play this game of your own volition. A person should not be forced to play or be tricked into playing this game.

You may think I’m overstating the power a simple game like this has to scare people. But if you want an insight into how scary the horror genre could become while experienced within virtual or mixed reality, I strongly recommend watching the “Playtest” episode from the outstanding TV series, Black Mirror.

I am a techno-idealist at heart and truly believe in the potential for mixed reality to create, facilitate, assist and guide people in the future. But it also has the power to confuse, disorient and harm people. As computing power improves it will become increasingly difficult to distinguish between what is real and what is not.  What if you couldn’t trust what you saw or heard?  How would you differentiate between a mixed reality experience and the symptoms of a psychological disorder? Are they projected images or hallucinations? Are those voices part of a game or in your head? As the classic film The Game so aptly presents, you can quickly become entwined within your own paranoia. Unfortunately there is a significant risk of gaslighting in mixed reality.

Mixed reality designers of the future must recognise these potential issues from the outset and design experiences accordingly. Of course, this should be just part and parcel of what a designer is trying to achieve, which is an ideal user experience.

Augmented reality content streams: communication at concerts

I went to an awesome concert last night. Vincent McMorrow played at the Chevron Festival Gardens in Perth. Throughout the evening I enjoyed the music, venue and great atmosphere. But before and during the concert, I couldn’t help notice the other concert goers and imagine how mixed reality might improve everyone’s future experiences.

Crowd safety and security

I don’t like the term crowd control. It makes it sound like you’re herding cattle or something. I think what we’re all aiming for, as concert attendees and staff, is crowd safety. I’ve been to quite a few concerts and dance parties in Australia and overseas. It’s great being up the front, dancing near the DJ or band, in the thick of it among all the other enthusiastic fans. But it can get hectic at times and I’m short. On more than one occassion the crowd has surged forward and I’ve felt a sense of panic and claustrophobia. Thankfully, those moments have only lasted a few seconds. But others are not so lucky. Sometimes it’s not the crowd en masse that presents a potential danger but a one off situation. Two guys start a fight. Someone faints. Someone else climbs the lighting rig at Alison Wonderland’s secret warehouse party (seriously dude, not cool). In those instances, venue staff have to get to those people and quickly. But it’s not that easy.

Firstly, someone has to alert staff to what’s going on. A staff member has to make their way to that spot. Then they have to assess the situation. They might have to administer first aid. If they need additional help, they have to call for other staff and explain where they are located. If people have to be moved, they need to clear a path through the crowd.

Essentially, what we’re trying to improve is communication between security, first aid responders and the audience. How do we transmit important information between people quickly and efficiently?

A key design consideration is the environment. A concert space is sensory overload. An audience wants to be wowed with amazing visual and audio effects from the artist they’ve come to see. Crowds make a lot of noise and can obscure vision. This all makes it very difficult for venue staff to see and hear (even with headphones) what’s going on around them and to communicate with their colleagues.

How could mixed reality aid communication in situations like these?

I’ve explored some key considerations from the perspective of someone who works at the concert. I’ve mocked up some very simple designs and will take you through my thought process.

Location of an incident

The central element in my design is a totem or navigational pole that pinpoints a potential incident as per the image below. I’ve deliberately taken something from the real world in order to capitalise on an existing system of meaning. That is, it’s easy for people to understand what an object may mean or do in a mixed reality universe because it behaves in a similar way to its real world counterpart. I’m not creating or trying to teach them a whole new system of meaning (not in this case anyhow but there are exceptions).

crowd01
Design No.1 (Original image by Kmeron via Flickr).

So these poles would function just like navigational signs in the real world. They are fixed to one location so that poles in the distance appear smaller than those in the foreground. This might sound very obvious but it’s actually a key difference between augmented reality and mixed reality. Augmented reality would layer these objects statically over your vision. But in mixed reality these poles behave like real objects: they are fixed to a single specific location, remain a fixed size in space, disappear from view when something is placed in front of it and you can approach and walk around them. (Of course you could also adjust any of these properties as the situation required.)

Type of incident

Each totem pole is topped with an emoji which describes the current issue. For example:

  • The “angry emoji” represents an aggressive person.
  • Two angry emojis represent a fight between two people.
  • The “emoji with mask” is a person in need of medical attention.

I’ve used emojis rather than text because I think images and symbols communicate faster than words. I also believe in the KISS principle and feel that text would be unnecessary clutter. If the priority is to get assistance to a location as quickly as possible then we should only communicate pertinent information. Emojis have another advantage over text: they are not bound by language. So two different people can quickly understand the system, and each other, even if they don’t speak the same language. A great idea for large festivals with attendees from different countries.

I’ve steered clear of menus or other complicated UI, and with good reason: it’s difficult to introduce elements to someone’s field of view when they’re mobile (or stationary but their head moves frequently). At Occulus Connect, Patrick Harris (Lead Game Designer at Minority Media) described the difficulties of designing UI for the virtual space. “There are no corners of the screen when you put on an Oculus Rift! That’s not a location that exists… You need to realize that anything in the ‘corner’ is actually [the player’s] peripheral vision.”

So rather than impose a static menu over a person’s vision, mixed reality objects can be more natural because they appear as real objects blending in with the existing environment. But therein lies another problem. Mixed reality elements need to contrast enough so that they are noticeable within the real world environment.

Use of colour

I’ve used a bright orange colour for the totem poles. I could use red and green in the colour palette later but I’d have to be careful not to assign opposing values (on/off, yes/no) that aren’t also distinguishable by symbols or text since that could disadvantage people who are colour blind. However, my mockup quickly revealed that orange is difficult to see against bright stage lights. It might be better to use a brighter colour with stronger contrast against a range of different lighting and potential stage effects (pyrotechnics, cryogenics, flames etc).

In fact, it may be necessary to use mixed reality to occlude vision from those effects before overlaying the totem poles. That is, overlay a dark background and then add the totem poles. (This will be easy to do in the future, as I predict that concerts will have a high degree of mixed reality content in themselves. If so, venue staff could simply opt out of the “concert content stream” and focus on the “venue/security/safety stream”. I’ll explore the notion of mixed reality content streams in a future post.)

Urgency

Based on experimenting and some findings with the first design, I iterated and created an updated version. I’ve changed the pole colour to bright blue which stands out much better.

crowd03
Design No.2

So next I wanted to consider: what if multiple incidents are happening simultaneously? How do I know which one is the most urgent? There needs to be a design element to communicate a sense of urgency, allowing staff to prioritise their actions. In the image above, I used solid blue lines for the poles to indicate more serious situations versus dashed lines for less serious situations. But thanks to the mockup I can already see a design flaw: from a distance dashed lines look solid anyway, so you can’t really distinguish between them.

Another way to communicate urgency could be to make the totem poles and emojis flash or animate. But this may still be hard to distinguish against crowd movement, stage lighting and special effects. A single static icon may be the simplest way to express this which I have incorporated as a red exclamation point over the emoji (see image Design No.3 ).

Assigning tasks

With multiple staff on shift, how would I know whether someone was already on their way to deal with a specific incident? My design mock up includes a simple white circle placed at the bottom of the pole to represent a staff member. At first, I included profile images within the circles. But for the sake of simplicity I removed them. I think the important thing is to know that someone is on their way, not who. (This information could always be added later). Multiple people attending an incident is represented by multiple white circles.

I quickly realised that the totem pole design also provides a neat solution to the problem of knowing how far a staff member was from an incident. As the staff member approaches the totem pole, the white circle that represents that individual also moves up the pole. Once the white circle reaches the emoji you know that someone has reached the incident location. And because the totem poles are showing data in real time, you can see how fast someone is moving towards that location.

Design No.3

Emergency exits

If a patron needed emergency medical attention they might need to be moved out of the crowd or building as quickly as possible. Although venue staff may know exactly where exits are located, it may not be clear how to reach them. Especially if it’s dark, noisy and crowded.

Mixed reality could map the room and obstacles, then indicate the quickest and safest route to the nearest exit. In part this could be done via object recognition but integrating the venue’s floor plan would be ideal. If patrons had access to the venue’s mixed reality content stream, a message could also be transmitted, requesting them to move aside to allow staff to move an injured person. In this way, a mixed reality stream could also serve as a public address system.

Design No.4

UI and feedback

User interface (UI) design for virtual, augmented and mixed reality is a very nascent field. I would like to cover it in more detail in another article but for this present design I will keep things simple. I wish to explore how a user could navigate this information in the context of a concert but omit how the user may input their selection (e.g. eye tracking, hand gestures, voice commands).

Design No.5 presents several key frames in this design:

  • The user can see all the totem poles at once.
  • The user activates a “selector tool” that allows them to focus on a particular mixed reality element in their field of view (illustrated via a simple blue crosshair to denote centre of screen) and select that element for further inspection.
  • Once an element has been selected a dialog box opens with additional detail (nature of the incident, specific requests and profile image of first responder) while all other elements in the field of view are “muted” (they become transparent).
Design No.5

Streams of mixed reality content

In this article I’ve predominantly covered the perspective of venue staff and first aid responders. The various mixed reality elements in this design are focused on providing those individuals with the information they need, at the right time (Just In Time Content). The “stream of content” is for their view only and can be switched on or off. In theory, elements of this stream could be shared with audience members.

As mentioned earlier I believe that mixed reality will become an integral part of future concert experiences. The audience will be privy to a private “stream” through which they can view all the mixed reality content presented by an artist. While venue staff could tune out/opt out of that stream and tune into their own security stream.

However, in emergency situations there could be elements that are made public and shared to all streams like a public broadcast, so that the audience and staff can access the same content. For example, if a fire broke out in one corner of the building a mixed reality interface could:

  • Trigger an announcement that the concert has stopped and requesting everyone to evacuate the building calmly.
  • Map safe evacuation routes.
  • Trigger a “warning” attached to all doors and routes leading to the dangerous area.
  • Highlight individuals who work at the venue so that they are easily identified.
  • Allow people who need assistance to “flag” themselves (elderly, parents with young children, physically disabled).
  • Allow dangerous obstacles or situations to be “flagged”.

Emergency responders could also access this mixed reality content stream to assist their response.

  • The exact location of the fire could be mapped out on a floor plan but also through mixed reality elements indicating multiple access routes.
  • Access to information about “flagged” areas indicating potential danger.
  • Easily highlight venue staff as points of contact.
  • Highlight important building features such as the location of water and gas pipes and electrical wiring.
  • The number and location of people still remaining in the building (assuming that patrons would allow themselves to be tracked at an event as a condition of entry).

So the next time you go to a concert, consider the kind of information you may want to access if you or a friend were in need of assistance. Or how the information you share could assist other patrons. In the meanwhile, enjoy the show!

Image credit: Kmeron via Flickr

Centre for Social Impact

Spotlight-banner-images_005 copy

Since February 2016 I’ve been liaising with the Centre for Social Impact (CSI), part of the Business School at the University of Western Australia, as their Digital Media Advisor.

I’m working with the amazing team at CSI on digital media strategies for the following projects:

  • Social Impact Festival 2016. The festival includes a series of workshops, free public lectures, and forums, designed to educate and challenge thinking around achieving social impact.
  • Learning for Purpose Program.  Learning for Purpose: Researching the Social Return on Education and Training in the Australian Not-for-Profit Sector is a ground breaking, national research program that investigates capacity building through professional development for the Australian Not-for-Profit sector. The aim is to systematically understand, evaluate, and improve the means through which individuals and organisations gain and sustain the key competencies for realising social change.

Creating (and tweaking) a video content strategy

While working as a Content and Social Media Manager I was lucky enough to lead the development of an online video series about the rollout of Australia’s National Broadband Network (NBN). I say lucky because at that time many companies believed you had to contract agencies to create video content. Once upon a time that was true. Recording equipment and editing software was expensive and required training to use. But since the advent of YouTube, the drop in price and complexity of software and the fact that every phone is also a video camera – literally anyone can create video content. So today the main task is understanding why you’re creating that content and ensuring that what is produced meets that need.

A strategy for video content

At my previous company, market research had revealed that many people were confused about the NBN including the technology itself, how it was being deployed and what they had to do to get it installed at their property. This information vacuum was an opportunity to educate the public through entertaining and factual content. We also knew that most people found technical information very dry to digest and would rather watch a video rather than read through reams of paper. So video became the medium of choice.

The company had a few existing NBN videos, created a few years earlier but these were very outdated. They had no interactive elements and were more suited to the traditional TV commercial or training video format with a passive viewer . Moreover, the call to action was simply a static screen (without annotations) that featured the company phone number and website URL. The videos didn’t take into account the fact that most viewers today would be using an internet enabled device, and a mobile one at that.

Our focus was to create short video clips specifically for online consumption that would educate viewers in an entertaining way. We would host these clips on YouTube, promote them using geo-targeted social media posts on Facebook and use embedded annotations to direct viewers to more detailed information on the company website or blog.

Design

I created a user journey map that outlined how video content could help answer the public’s questions about the NBN. Specifically, not just what people were asking but when they were asking those questions, as the timing of the NBN rollout differed depending on a person’s location in Australia. We used SEO data and findings from market research to determine which topics to cover. We then ordered the topics into a specific order, noted the relationship between videos (for subsequent annotations) and mapped how the viewer would navigate through our content across various platforms. For example, one journey looked like this: Facebook post -> YouTube video 1 -> YouTube video 2 -> blog article -> sign up to newsletter for future information.

Production

Everything was done in house and on an extremely small budget. I researched early topics with industry experts, created storyboards and wrote the scripts. Once our format and style had been established, I supervised the process while other copywriters in my team worked on subsequent storyboards and scripts. Talent was selected from staff within the call centre and filmed at the head office in Perth, Western Australia. This fit in nicely with the company’s brand which was friendly, a little bit quirky and made tech accessible for everyone. I worked closely with the Video Producer in my team on direction, editing and interactive elements. Once the live action footage had been shot the videos were created using Cinema 4D (animations) and Adobe Premiere Pro (editing). We also added Google Analytics to relevant online interactions so we could monitor and measure performance.

Learnings

  • Batching. We created the videos in batches. This allowed us to iterate on the format and make improvements with subsequent videos. If you watch the whole series you’ll see how the videos changed slightly as we learned what worked and what didn’t. For example, viewers love to click on annotations! So we included more of them but only where it was appropriate, like learning about different plans or referring to another video in the series. We didn’t want the annotations to be distracting or feel spammy.
  • Plan interactions ahead of time. During the storyboard and script writing phase we made note of where interaction elements (links, buttons, transitions) would be included. This allowed us to have some fun (letting the talent point to a virtual button) and made the interactions feel more natural rather than just “stuck on” later.
  • Learn from the professionals. There are lots of free resources online depending on how deep you want to get into video production. But there were lots of simple things we did. For example, we undertook location scouting trips around head office. We wandered around and took photos of different rooms and areas at different times of the day. This helped us understand the way sunlight moved through the building, how rooms were used at certain times, chat to people who worked in those areas and learn about other constraints we couldn’t have anticipated (like customers potentially wandering through filming).
  • Thumbnails are important. Jack Donaghy from 30 Rock once said, “Your hair is your head suit.” And for an online universe, thumbnails are your video suit. Whenever your video is featured around the web the thumbnail is what people see, whether it’s on a Google search result page (SERP) or embedded within a blog article. To encourage people to click on your video it’s best practice (IMHO) to use a branded thumbnail that’s either an engaging screenshot from the video or a customised image that features the title of the video. Because we were creating educational content, we chose the later. Also, if you have a series of videos (like we did) it’s nice to go the extra mile and create a consistent look using similar colours, fonts, layout etc.
  • Music sets the scene. A video of someone talking straight to the camera feels a little flat. Adding some background music lifts the mood and provides continuity when you change the scene. For example, switching from the narrator to an animation and then back to the narrator.
  • Just one more change… During the research phase, we consulted with technical experts and product managers, wrote a draft script and then sought their initial feedback. Down the track we occasionally had to tweak the scripts further (adding more information, changing the focus etc). We would then check with the same technical experts to make sure the new content was still technically correct. However we found that people would sometimes take that as an opportunity to request additional changes: things that had been previously overlooked, phrases they didn’t like or additional information they wanted to include. Most of these changes had nothing to do with technical accuracy. It was just their preference. Our team was very collaborative and initially we incorporated these changes. But over time, these concessions slowed us down. A lot. It also impacted on our budget because sometimes changes were requested when we had already started filming or editing. This meant we had to reshoot some scenes. We quickly learned that we had to be firm with requests for changes. We valued our colleagues’ technical expertise but we helped them understand our constraints and challenges by taking them through our production process. We all agreed that changes could be made up to the second version of the script, after that only critical elements that impacted accuracy could be amended. Following that discussion and agreement, things worked much smoother.

Measuring success

The success of this strategy was measured by:

  • Traffic from the videos to the company’s main website, either directly or via the blog (Google Analytics)
  • Engagement through video view counts and duration of views as well as comments, likes and shares (YouTube analytics), annotation click rates to other content (Google Analytics) and whether viewers joined our newsletter for more information (number of subscribers).

Although I can’t divulge specific numbers, I can say that the videos were very successful and exceeded our targets for traffic and engagement. Even when paid promotion ceased, the videos continued to increase in popularity as viewers found the content informative, interacting and sharing that content. It also provided online publications like Mashable with additional free content to share when they were covering a story about the NBN or the company (which also increased video view counts and interactions). All these benefits were a great return on investment, given our small budget.

I’m grateful for the opportunity to develop this video content in house with my team. I would not have learned so much if an external agency had been contracted to create them. From the overall strategy and researching topics to rolling our sleeves up and getting behind the camera, we learned that video is a powerful communication tool online. But that may only last as long as it takes for virtual, augmented and mixed realities to catch up and become adopted in the mainstream.