One of the most obvious use cases for mixed reality is the ability to locate two dimensional displays in space. These days digital screens seem to be everywhere: airport lounges, bus shelters, highway billboards, fixed to the ceiling in dentist rooms. But there are a number of drawbacks including:
- Cost (initial purchase and electricity).
- Obsolescence and need for replacement.
So the ability to place a virtual screen wherever you like and at little or no cost, presents an almost infinite range of potential use cases.
But there’s one particular situation I’ve been considering for a while: how mixed reality displays could improve exercise.
Proprioception is the sense of one’s own body. Where our body is located in space, how it moves and the relative position of parts of the body. For example, close your eyes, stretch out an arm, then try to touch your nose. In order to complete this task you need to have an idea of the length of your arm, the angle at which your arm is bent and the location of your nose. But this ability is not just for touching your nose in the dark.
“The ability to sense force, which is known as mechanosensation, provides humans and other animals with important information about the environment; it is crucial for social interactions, such as comforting or caressing, and is required for motor coordination… Similarly, proprioception is considered to be essential for posture and controlled movement, but little is known about the underlying mechanisms and the precise role of this sense.” (From The New England Journal of Medicine.)
Undertaking any new physical activity provides a real work out for not just the body but also the brain. If you’ve ever taken up a new exercise that requires a lot of coordination, you know what I’m talking about. There’s a clumsy dance between your brain and body, as they both try to talk to each other about how to coordinate these new movements.
How can we help this body/brain connection while learning new movements?
We can assist this process by showing the body what it’s doing in real time. Once we’ve been shown how to do something (by an instructor or video) we understand what the correct movement looks like: we just don’t understand what it should feel like. So it makes sense that if we could see what we’re doing, we can compare that image with a mental map of what we should be doing and then make the necessary adjustments.
How do we see what we’re doing? Easy. We can use mirrors. We could also record ourselves but we would’t receive the real time feedback comparing what we see with what we feel.
As humans we’ve long understood the use of this aid. It’s why dance studios have mirrors along their walls. As well as gyms and yoga studios.
I am relatively new to pilates reformer classes but have practiced pilates and yoga on and off for a while. For those who aren’t familiar this is what pilates reformer equipment looks like and some of the associated postures.
I know. Looks like medieval torture. But I’ve gained a lot of health benefits like strength and flexibility. I also like the way the exercises forces me to use my brain. I have to really concentrate. Part of this focus comes from watching my body in mirrors around the studio, adjusting my posture and movements with what I’ve been shown by the instructor. But this has some difficulties. Let’s take a look at the layout of my pilates studio including the placement of mirrors.
Mirrors are located as follows:
- One mirror in front of each reformer machine on the North wall.
- A large mirror on the East wall.
- A large wall mirror on the South wall located behind a trapeze table.
- No mirrors on the West wall.
There are a lot of mirrors in the studio but it’s not always possible to see my reflection. This depends on my body position, posture, head angle, location of nearby mirrors and other people/objects in the studio.
In Figure 3, I’ve highlighted a rough estimate of a user’s central gaze in each posture (“near peripheral vision” approximately 30° either side of straight ahead, 60° total field of view).
- A blue mirror indicates the user can see her reflection.
- A red mirror indicates they cannot.
As illustrated, the user can only see her reflection in three postures: A, E and F. Even then, it’s not possible to see her whole body at once.
Another complication is that these postures are not static as the user will move in different directions as per Figure 4.
Ok, so today’s pilates reformer setup isn’t ideal.
But how could mixed reality help a student see how her body moves while she works out?
A mixed reality design facilitating proprioception during exercise
(Note, the following designs are based on the user wearing a mixed reality headset/glasses. Ideally the device should not impede movement or cause discomfort during the activity. Realistically, no current mixed reality hardware would be useful in this context as they are either tethered and/or too heavy for prolonged use during exercise. When mixed reality content can be displayed on to light weight glasses or contact lenses, this use case will be more likely.)
Mixed reality displays
Mixed reality offers the use of dynamic displays. Basically, these look like a two-dimensional screen hanging in space. This presents many advantages over a mirror or any type of digital screen in the real world.
A mixed reality display can be:
- Displayed anywhere. It could be fixed to any plane/flat surface in the real world e.g. wall, table, floor or ceiling. If there are no planes, the display could “hover” in the air at a fixed and comfortable distance from the user within their central eye gaze.
- Displayed at any size or ratio. The display could fill an entire wall of the studio or be the size of a smartphone screen.
- Adjusted to move with the user. During exercise the user may move their head and/or body in different directions. The mixed reality display can move with the user to ensure continuity.
For our pilates reformer student, we can position a display at a comfortable distance and within their central eye gaze – no matter what position they are in or which direction they look as illustrated in Figure 5.
Capturing the ideal perspective
A mirror can only show a user’s reflection. Mixed reality can show much more. During a pilates class a user would ideally like to see their body from multiple angles without having to turn their head (which in itself could disrupt their balance, posture or movement). Thus, in an optimal situation we should capture a user’s movement from different angles then select the ideal perspective that highlights overall movement.
We need cameras to capture the user’s movements. The number and placement of these cameras requires consideration.
- Ideal recording angle and distance. For each possible posture, we determine the best angle(s) from which to view the entire body. Figure 6 illustrates that for a seated posture an ideal camera location is a 90 degree angle at approximately 1200mm height. The x distance depends on where other objects may be located in the vicinity of the user.
As illustrated in Figure 7, the ideal recording angle could be obtained from a camera located on either side (B or C) or above the user (A) depending on the user’s posture.
- Camera locations. Reviewing the overall studio layout, we can locate cameras where they facilitate ideal recording angles for each pilates reformer machine. One thing to note is cameras need to be fixed to something. In this case study, I’ve attached them to the wall, ceiling and standing objects (e.g. weight machine, trapeze table).
Camera feeds and content display feeds
Now that multiple cameras are capturing feeds from different angles, we can display these feeds to the user. But which one?
One solution is to allow the user to scroll through feeds and select a preferred view as demonstrated in Figure 9. (Although this basic mockup uses images of different users it illustrates how the user can see themselves from different angles and make the selection through a rotating carousel format.)
It’s important to note that in most pilates’ postures, both hands are occupied during exercise. So what types of interactions are possible while wearing a mixed reality headset?
- Gesture. A gesture-based user interface could be used to access the menu before and after exercise. The user could open the system, navigate through menus and commence “exercise mode”. Once this mode has been engaged, the user must rely on other system inputs.
- Gaze tracking. This may not be entirely useful during exercise mode as the user’s head will move during exercise. However, like gesture inputs, gaze tracking could be used to open and navigate menus, before and after exercise mode.
- Voice commands. As long as the interface design is quite simple, voice commands can facilitate navigation through camera feeds and open/close “exercise mode”. In the mockup illustrated in Figure 9, the user could say “next feed” to view the next camera feed (i.e click the right arrow) or “previous feed” to view the previous camera feed (i.e click the left arrow). One drawback is that some users may feel uncomfortable about issuing voice commands in a public space. My experience in pilates reformer classes and gyms is that there is often music playing or people chatting so it might not be too awkward.
- Unobstructed feeds. Clear camera angles may be difficult to obtain in a busy studio. People and equipment are constantly moving. When reformers are side by side, it isn’t always possible for the user to have an unobstructed view of themselves in a mirror or a camera feed. This may require rearranging the location of machines within a studio.
- Which camera feeds? The system must be set up in such a way or “smart” enough to know which camera feeds to display to each user. Thus, the user is only presented with angles of themselves. One solution is that the system recognises which reformer machine is in use and therefore which cameras will provide the ideal recording angles.
- Privacy. Capturing live video feed presents an opportunity to record these feeds for later viewing. This may be useful to individual users to improve their practice. However, all studio clients should consent to recording.
- Safety. Overlaying a mixed reality display over the real world is a potential safety hazard. Display feeds should not obstruct vision. One solution is for the system to recognise when objects enter the gaze area or within prescribed distance around the user. For example, someone in the real world “walks through” the display and appears to stand in front of it.
The rapidly expanding field of Artificial Intelligence (AI) could take these live camera feeds and provide the user with additional information.
- Live personal instructor. Monitor a user’s movements and alert them if they move beyond an “ideal” range. For example, when the user’s posture is out of alignment or limbs are in an incorrect position which could lead to discomfort or injury. The alert could be displayed through a visual alert and/or audio message.
- Guide attention. Highlight within the live camera feed areas on the user’s body which muscles should be engaged for a particular posture. For example, using hamstring muscles rather than back or abdominal muscles.
Pilates may seem like a quirky use case to explore but there are serious applications that could benefit from mixed reality assisted proprioception.
Stroke is one of Australia’s biggest killers and a leading cause of disability. One particular problem stroke victims can experience is difficulty planning or coordinating movement known as apraxia. They can also feel slow or clumsy when coordinating movements which is known as ataxia.
In conjunction with a physiotherapist a mixed reality system could become part of a Proprioceptive Neuromuscular Facilitation (PNF) program:
- Real time feedback. Showing a stroke victim how they walk from different angles as they walk. The visual input helps them make a mental association between what they are seeing with what they are feeling.
- Mixed reality displays versus mirrors. Stroke victims may find it difficult to turn their head and look into a mirror while trying to coordinate their own walking movements at the same time. Additionally, it is not always possible to locate large mirrors in many positions around a rehabilitation centre.
- Voice commands. Stroke victims rely on their hands and arms to steady themselves while learning to walk. Thus their hands are not available to operate a smartphone or other device. While someone else could operate a system on their behalf, voice commands provide stroke victims with autonomy and independence to use the system themselves.
- Record and analysis. Later viewing can assist a stroke victim to see and understand their own movements. Working with a physiotherapist they can plan future therapy sessions collaboratively.