Mixed reality for the perfect pilates work out

One of the most obvious use cases for mixed reality is the ability to locate two dimensional displays in space. These days digital screens seem to be everywhere: airport lounges, bus shelters, highway billboards, fixed to the ceiling in dentist rooms. But there are a number of drawbacks including:

  • Cost (initial purchase and electricity).
  • Installation.
  • Fragility.
  • Maintenance.
  • Obsolescence and need for replacement.

So the ability to place a virtual screen wherever you like and at little or no cost, presents an almost infinite range of potential use cases.

But there’s one particular situation I’ve been considering for a while: how mixed reality displays could improve exercise.

Proprioception is the sense of one’s own body. Where our body is located in space, how it moves and the relative position of parts of the body. For example, close your eyes, stretch out an arm, then try to touch your nose. In order to complete this task you need to have an idea of the length of your arm, the angle at which your arm is bent and the location of your nose. But this ability is not just for touching your nose in the dark.

“The ability to sense force, which is known as mechanosensation, provides humans and other animals with important information about the environment; it is crucial for social interactions, such as comforting or caressing, and is required for motor coordination… Similarly, proprioception is considered to be essential for posture and controlled movement, but little is known about the underlying mechanisms and the precise role of this sense.” (From The New England Journal of Medicine.)

Undertaking any new physical activity provides a real work out for not just the body but also the brain. If you’ve ever taken up a new exercise that requires a lot of coordination, you know what I’m talking about. There’s a clumsy dance between your brain and body, as they both try to talk to each other about how to coordinate these new movements.

How can we help this body/brain connection while learning new movements?

We can assist this process by showing the body what it’s doing in real time. Once we’ve been shown how to do something (by an instructor or video) we understand what the correct movement looks like: we just don’t understand what it should feel like. So it makes sense that if we could see what we’re doing, we can compare that image with a mental map of what we should be doing and then make the necessary adjustments.

How do we see what we’re doing? Easy. We can use mirrors. We could also record ourselves but we would’t receive the real time feedback comparing what we see with what we feel.

As humans we’ve long understood the use of this aid. It’s why dance studios have mirrors along their walls. As well as gyms and yoga studios.

I am relatively new to pilates reformer classes but have practiced pilates and yoga on and off for a while. For those who aren’t familiar this is what pilates reformer equipment looks like and some of the associated postures.

Figure 1. Various postures on a pilates reformer machine. (Original images from http://www.pilates-exercises.info)

I know. Looks like medieval torture. But I’ve gained a lot of health benefits like strength and flexibility. I also like the way the exercises force me to use my brain. I have to really concentrate. Part of this focus comes from watching my body in mirrors around the studio, adjusting my posture and movements with what I’ve been shown by the instructor. But this has some difficulties. Let’s take a look at the layout of my pilates studio including the placement of mirrors.

Figure 2. Layout of a pilates reformer studio.

Mirrors are located as follows:

  • One mirror in front of each reformer machine on the North wall.
  • A large mirror on the East wall.
  • A large wall mirror on the South wall located behind a trapeze table.
  • No mirrors on the West wall.

There are a lot of mirrors in the studio but it’s not always possible to see my reflection. This depends on my body position, posture, head angle, location of nearby mirrors and other people/objects in the studio.

In Figure 3, I’ve highlighted a rough estimate of a user’s central gaze in each posture (“near peripheral vision” approximately 30° either side of straight ahead, 60° total field of view).

  • A blue mirror indicates the user can see her reflection.
  • A red mirror indicates they cannot.

 

Figure 3. Central eye gaze of pilates reformer user and visibility of her own reflection in a nearby mirror.

As illustrated, the user can only see her reflection in three postures: A, E and F. Even then, it’s not possible to see her whole body at once.

Another complication is that these postures are not static as the user will move in different directions as per Figure 4.

 

Figure 4. Directional movement of pilates reformer user.

Ok, so today’s pilates reformer setup isn’t ideal.

But how could mixed reality help a student see how her body moves while she works out?

A mixed reality design facilitating proprioception during exercise

(Note, the following designs are based on the user wearing a mixed reality headset/glasses. Ideally the device should not impede movement or cause discomfort during the activity. Realistically, no current mixed reality hardware would be useful in this context as they are either tethered and/or too heavy for prolonged use during exercise. When mixed reality content can be displayed on to light weight glasses or contact lenses, this use case will be more likely.)

Mixed reality displays

Mixed reality offers the use of dynamic displays. Basically, these look like a two-dimensional screen hanging in space. This presents many advantages over a mirror or any type of digital screen in the real world.

A mixed reality display can be:

  • Displayed anywhere. It could be fixed to any plane/flat surface in the real world e.g. wall, table, floor or ceiling. If there are no planes, the display could “hover” in the air at a fixed and comfortable distance from the user within their central eye gaze.
  • Displayed at any size or ratio. The display could fill an entire wall of the studio or be the size of a smartphone screen.
  • Adjusted to move with the user. During exercise the user may move their head and/or body in different directions. The mixed reality display can move with the user to ensure continuity.

For our pilates reformer student, we can position a display at a comfortable distance and within their central eye gaze – no matter what position they are in or which direction they look as illustrated in Figure 5.

Figure 5. Mixed reality displays located within central eye gaze of pilates reformer user.

Capturing the ideal perspective

A mirror can only show a user’s reflection. Mixed reality can show much more. During a pilates class a user would ideally like to see their body from multiple angles without having to turn their head (which in itself could disrupt their balance, posture or movement).  Thus, in an optimal situation we should capture a user’s movement from different angles then select the ideal perspective that highlights overall movement.

We need cameras to capture the user’s movements. The number and placement of these cameras requires consideration.

  • Ideal recording angle and distance. For each possible posture, we determine the best angle(s) from which to view the entire body. Figure 6 illustrates that for a seated posture an ideal camera location is a 90 degree angle at approximately 1200mm height. The x distance depends on where other objects may be located in the vicinity of the user.

 

Figure 6. Camera location for ideal perspective.

As illustrated in Figure 7, the ideal recording angle could be obtained from a camera located on either side (B or C) or above the user (A) depending on the user’s posture.

Figure 7. Multiple camera locations providing alternative recording angles.
  • Camera locations. Reviewing the overall studio layout, we can locate cameras where they facilitate ideal recording angles for each pilates reformer machine. One thing to note is cameras need to be fixed to something. In this case study, I’ve attached them to the wall, ceiling and standing objects (e.g. weight machine, trapeze table).
Figure 8. Layout of pilates reformer studio with potential camera locations.

Camera feeds and content display feeds

Now that multiple cameras are capturing feeds from different angles, we can display these feeds to the user. But which one?

One solution is to allow the user to scroll through feeds and select a preferred view as demonstrated in Figure 9.  (Although this basic mockup uses images of different users it illustrates how the user can see themselves from different angles and make the selection through a rotating carousel format.)

Figure 9. Mockup of basic UI for a mixed reality display. Right and left arrows facilitate navigation through a rotating carousel of live camera feeds.

Interaction design

It’s important to note that in most pilates’ postures, both hands are occupied during exercise. So what types of interactions are possible while wearing a mixed reality headset?

  • Gesture. A gesture-based user interface could be used to access the menu before and after exercise. The user could open the system, navigate through menus and commence “exercise mode”. Once this mode has been engaged, the user must rely on other system inputs.
  • Gaze tracking. This may not be entirely useful during exercise mode as the user’s head will move during exercise. However, like gesture inputs, gaze tracking could be used to open and navigate menus, before and after exercise mode.
  • Voice commands. As long as the interface design is quite simple, voice commands can facilitate navigation through camera feeds and open/close “exercise mode”. In the mockup illustrated in Figure 9, the user could say “next feed” to view the next camera feed (i.e click the right arrow) or “previous feed” to view the previous camera feed (i.e click the left arrow). One drawback is that some users may feel uncomfortable about issuing voice commands in a public space. My experience in pilates reformer classes and gyms is that there is often music playing or people chatting so it might not be too awkward.

Potential issues

  • Unobstructed feeds. Clear camera angles may be difficult to obtain in a busy studio. People and equipment are constantly moving. When reformers are side by side, it isn’t always possible for the user to have an unobstructed view of themselves in a mirror or a camera feed. This may require rearranging the location of machines within a studio.
  • Which camera feeds? The system must be set up in such a way or “smart” enough to know which camera feeds to display to each user. Thus, the user is only presented with angles of themselves. One solution is that the system recognises which reformer machine is in use and therefore which cameras will provide the ideal recording angles.
  • Privacy. Capturing live video feed presents an opportunity to record these feeds for later viewing. This may be useful to individual users to improve their practice. However, all studio clients should consent to recording.
  • Safety. Overlaying a mixed reality display over the real world is a potential safety hazard. Display feeds should not obstruct vision. One solution is for the system to recognise when objects enter the gaze area or within prescribed distance around the user. For example, someone in the real world  “walks through” the display and appears to stand in front of it.

Additional features

The rapidly expanding field of Artificial Intelligence (AI) could take these live camera feeds and provide the user with additional information.

  • Live personal instructor. Monitor a user’s movements and alert them if they move beyond an “ideal” range. For example, when the user’s posture is out of alignment or limbs are in an incorrect position which could lead to discomfort or injury. The alert could be displayed through a visual alert and/or audio message.
  • Guide attention. Highlight within the live camera feed areas on the user’s body which muscles should be engaged for a particular posture. For example, using hamstring muscles rather than back or abdominal muscles.

Future applications

Pilates may seem like a quirky use case to explore but there are serious applications that could benefit from mixed reality assisted proprioception.

Stroke is one of Australia’s biggest killers and a leading cause of disability. One particular problem stroke victims can experience is difficulty planning or coordinating movement known as apraxia. They can also feel slow or clumsy when coordinating movements which is known as ataxia.

In conjunction with a physiotherapist a mixed reality system could become part of a Proprioceptive Neuromuscular Facilitation (PNF) program:

  • Real time feedback. Showing a stroke victim how they walk from different angles as they walk. The visual input helps them make a mental association between what they are seeing with what they are feeling.
  • Mixed reality displays versus mirrors.  Stroke victims may find it difficult to turn their head and look into a mirror while trying to coordinate their own walking movements at the same time. Additionally, it is not always possible to locate large mirrors in many positions around a rehabilitation centre.
  • Voice commands. Stroke victims rely on their hands and arms to steady themselves while learning to walk. Thus their hands are not available to operate a smartphone or other device. While someone else could operate a system on their behalf, voice commands provide stroke victims with autonomy and independence to use the system themselves.
  • Record and analysis. Later viewing can assist a stroke victim to see and understand their own movements. Working with a physiotherapist they can plan future therapy sessions collaboratively.

How to make a rapid prototyping kit

Years ago, I bought a plastic tool box and started collecting useful bits and pieces for creating prototypes. Good rapid prototyping should be 3 things: cheap, fast and easy. So I procured items that would help encourage this approach.

protoypingkit02

Here’s a list of what’s in my tool kit. (Note, not everything fits in my tool box at one time but it’s very useful for carrying stuff to a prototyping session.)

  • Pens, pencils and markers/ textas.
  • Simple stickers, different shapes and colours.
  • Plain white paper for sketching. Long rolls of butcher’s paper or kids craft paper are great for when you need large sections of paper. Coloured paper is good too.
  • Post it notes. I have a love/hate relationship with these. They are very useful but can be over relied on during the brainstorming stage at the expense of moving on to create actual prototypes.
  • Graph paper is ideal for sketching levels, layouts and journeys. Hexagonal paper in particular is very useful for mapping movement because your characters can move diagonally. If you can’t find any locally, you can find these online and print them.
  • Cardboard – all kinds! They come in so many different thicknesses and are essential for 3D prototyping.
  • Adhesives. Tape, double sided tape, glue sticks, blu tack, hot glue gun. Depending on whether you need to stick things together temporarily or permanently.
  • Small figurines to represent players and NPCs (non-playing characters) preferably in different colours. For example very small toys or game tokens.
  • Cutting blade/scalpel.
  • Steel ruler.
  • Cutting mat.
  • Larger items that are useful for role playing like plastic cones, sticker labels, string, blu tack, larger sections of paper/cardboard etc

Where do I find all these amazing items?

Prototyping materials don’t have to be expensive. In fact, they shouldn’t be. Plus, the added benefit of cheap (or free) items is that you aren’t so precious with supplies and are more likely to discard prototypes for the next iteration. But I do have a few tips on how to creatively source items for your kit.

Your local Daiso, Poundland, reject or two dollar store. These stores stock basic household items and stationary for very low prices. When you start developing more complex 3D prototypes, get creative and think about repurposing items beyond their original design. For example, if you need a pen holder you don’t have to buy a special container marked “pen holder”. You could use a glass jar, teapot, vase, chopstick storage box, plastic food storage container etc

Recycling. When I was a kid I had a craft box. I saved every bit of cardboard, ribbon, and egg carton I could get my hands on. Even now as a designer I have one. And why not? It’s free and once you get started, you’ll start looking differently at objects you might once have just thrown out.

Op shop/ Thrift shop. Ok, I’m the first to admit it. I love visiting Op Shops. I love browsing racks, picking through shelves, and rummaging in boxes. I wonder about these objects that get left behind. For what purpose were they originally bought and why weren’t they needed any more? I love to recycle rather than buy something new. I also love repurposing items for uses different to what their designers may have intended. Mostly, I love the random serendipity of it all. There are hidden delights for one and all! So when you deck out your protoyping kit, be sure to check out all the various sections of your local Op Shop. For example, the pieces that come in board games are perfect as player figurines or tokens and the board itself may come in handy.

I hope this information helps to whet your appetite so you can go forth and create your own rapid prototyping kit. You can start collecting bits and pieces today! The more you prototype the more you’ll start to see everyday common objects as potential tools and parts for your next prototype. And that’s another benefit to creating a kit, it gets your brain working and thinking about how to create all kinds of prototypes.