Mixed reality for the perfect pilates work out

One of the most obvious use cases for mixed reality is the ability to locate two dimensional displays in space. These days digital screens seem to be everywhere: airport lounges, bus shelters, highway billboards, fixed to the ceiling in dentist rooms. But there are a number of drawbacks including:

  • Cost (initial purchase and electricity).
  • Installation.
  • Fragility.
  • Maintenance.
  • Obsolescence and need for replacement.

So the ability to place a virtual screen wherever you like and at little or no cost, presents an almost infinite range of potential use cases.

But there’s one particular situation I’ve been considering for a while: how mixed reality displays could improve exercise.

Proprioception is the sense of one’s own body. Where our body is located in space, how it moves and the relative position of parts of the body. For example, close your eyes, stretch out an arm, then try to touch your nose. In order to complete this task you need to have an idea of the length of your arm, the angle at which your arm is bent and the location of your nose. But this ability is not just for touching your nose in the dark.

“The ability to sense force, which is known as mechanosensation, provides humans and other animals with important information about the environment; it is crucial for social interactions, such as comforting or caressing, and is required for motor coordination… Similarly, proprioception is considered to be essential for posture and controlled movement, but little is known about the underlying mechanisms and the precise role of this sense.” (From The New England Journal of Medicine.)

Undertaking any new physical activity provides a real work out for not just the body but also the brain. If you’ve ever taken up a new exercise that requires a lot of coordination, you know what I’m talking about. There’s a clumsy dance between your brain and body, as they both try to talk to each other about how to coordinate these new movements.

How can we help this body/brain connection while learning new movements?

We can assist this process by showing the body what it’s doing in real time. Once we’ve been shown how to do something (by an instructor or video) we understand what the correct movement looks like: we just don’t understand what it should feel like. So it makes sense that if we could see what we’re doing, we can compare that image with a mental map of what we should be doing and then make the necessary adjustments.

How do we see what we’re doing? Easy. We can use mirrors. We could also record ourselves but we would’t receive the real time feedback comparing what we see with what we feel.

As humans we’ve long understood the use of this aid. It’s why dance studios have mirrors along their walls. As well as gyms and yoga studios.

I am relatively new to pilates reformer classes but have practiced pilates and yoga on and off for a while. For those who aren’t familiar this is what pilates reformer equipment looks like and some of the associated postures.

Figure 1. Various postures on a pilates reformer machine. (Original images from http://www.pilates-exercises.info)

I know. Looks like medieval torture. But I’ve gained a lot of health benefits like strength and flexibility. I also like the way the exercises force me to use my brain. I have to really concentrate. Part of this focus comes from watching my body in mirrors around the studio, adjusting my posture and movements with what I’ve been shown by the instructor. But this has some difficulties. Let’s take a look at the layout of my pilates studio including the placement of mirrors.

Figure 2. Layout of a pilates reformer studio.

Mirrors are located as follows:

  • One mirror in front of each reformer machine on the North wall.
  • A large mirror on the East wall.
  • A large wall mirror on the South wall located behind a trapeze table.
  • No mirrors on the West wall.

There are a lot of mirrors in the studio but it’s not always possible to see my reflection. This depends on my body position, posture, head angle, location of nearby mirrors and other people/objects in the studio.

In Figure 3, I’ve highlighted a rough estimate of a user’s central gaze in each posture (“near peripheral vision” approximately 30° either side of straight ahead, 60° total field of view).

  • A blue mirror indicates the user can see her reflection.
  • A red mirror indicates they cannot.


Figure 3. Central eye gaze of pilates reformer user and visibility of her own reflection in a nearby mirror.

As illustrated, the user can only see her reflection in three postures: A, E and F. Even then, it’s not possible to see her whole body at once.

Another complication is that these postures are not static as the user will move in different directions as per Figure 4.


Figure 4. Directional movement of pilates reformer user.

Ok, so today’s pilates reformer setup isn’t ideal.

But how could mixed reality help a student see how her body moves while she works out?

A mixed reality design facilitating proprioception during exercise

(Note, the following designs are based on the user wearing a mixed reality headset/glasses. Ideally the device should not impede movement or cause discomfort during the activity. Realistically, no current mixed reality hardware would be useful in this context as they are either tethered and/or too heavy for prolonged use during exercise. When mixed reality content can be displayed on to light weight glasses or contact lenses, this use case will be more likely.)

Mixed reality displays

Mixed reality offers the use of dynamic displays. Basically, these look like a two-dimensional screen hanging in space. This presents many advantages over a mirror or any type of digital screen in the real world.

A mixed reality display can be:

  • Displayed anywhere. It could be fixed to any plane/flat surface in the real world e.g. wall, table, floor or ceiling. If there are no planes, the display could “hover” in the air at a fixed and comfortable distance from the user within their central eye gaze.
  • Displayed at any size or ratio. The display could fill an entire wall of the studio or be the size of a smartphone screen.
  • Adjusted to move with the user. During exercise the user may move their head and/or body in different directions. The mixed reality display can move with the user to ensure continuity.

For our pilates reformer student, we can position a display at a comfortable distance and within their central eye gaze – no matter what position they are in or which direction they look as illustrated in Figure 5.

Figure 5. Mixed reality displays located within central eye gaze of pilates reformer user.

Capturing the ideal perspective

A mirror can only show a user’s reflection. Mixed reality can show much more. During a pilates class a user would ideally like to see their body from multiple angles without having to turn their head (which in itself could disrupt their balance, posture or movement).  Thus, in an optimal situation we should capture a user’s movement from different angles then select the ideal perspective that highlights overall movement.

We need cameras to capture the user’s movements. The number and placement of these cameras requires consideration.

  • Ideal recording angle and distance. For each possible posture, we determine the best angle(s) from which to view the entire body. Figure 6 illustrates that for a seated posture an ideal camera location is a 90 degree angle at approximately 1200mm height. The x distance depends on where other objects may be located in the vicinity of the user.


Figure 6. Camera location for ideal perspective.

As illustrated in Figure 7, the ideal recording angle could be obtained from a camera located on either side (B or C) or above the user (A) depending on the user’s posture.

Figure 7. Multiple camera locations providing alternative recording angles.
  • Camera locations. Reviewing the overall studio layout, we can locate cameras where they facilitate ideal recording angles for each pilates reformer machine. One thing to note is cameras need to be fixed to something. In this case study, I’ve attached them to the wall, ceiling and standing objects (e.g. weight machine, trapeze table).
Figure 8. Layout of pilates reformer studio with potential camera locations.

Camera feeds and content display feeds

Now that multiple cameras are capturing feeds from different angles, we can display these feeds to the user. But which one?

One solution is to allow the user to scroll through feeds and select a preferred view as demonstrated in Figure 9.  (Although this basic mockup uses images of different users it illustrates how the user can see themselves from different angles and make the selection through a rotating carousel format.)

Figure 9. Mockup of basic UI for a mixed reality display. Right and left arrows facilitate navigation through a rotating carousel of live camera feeds.

Interaction design

It’s important to note that in most pilates’ postures, both hands are occupied during exercise. So what types of interactions are possible while wearing a mixed reality headset?

  • Gesture. A gesture-based user interface could be used to access the menu before and after exercise. The user could open the system, navigate through menus and commence “exercise mode”. Once this mode has been engaged, the user must rely on other system inputs.
  • Gaze tracking. This may not be entirely useful during exercise mode as the user’s head will move during exercise. However, like gesture inputs, gaze tracking could be used to open and navigate menus, before and after exercise mode.
  • Voice commands. As long as the interface design is quite simple, voice commands can facilitate navigation through camera feeds and open/close “exercise mode”. In the mockup illustrated in Figure 9, the user could say “next feed” to view the next camera feed (i.e click the right arrow) or “previous feed” to view the previous camera feed (i.e click the left arrow). One drawback is that some users may feel uncomfortable about issuing voice commands in a public space. My experience in pilates reformer classes and gyms is that there is often music playing or people chatting so it might not be too awkward.

Potential issues

  • Unobstructed feeds. Clear camera angles may be difficult to obtain in a busy studio. People and equipment are constantly moving. When reformers are side by side, it isn’t always possible for the user to have an unobstructed view of themselves in a mirror or a camera feed. This may require rearranging the location of machines within a studio.
  • Which camera feeds? The system must be set up in such a way or “smart” enough to know which camera feeds to display to each user. Thus, the user is only presented with angles of themselves. One solution is that the system recognises which reformer machine is in use and therefore which cameras will provide the ideal recording angles.
  • Privacy. Capturing live video feed presents an opportunity to record these feeds for later viewing. This may be useful to individual users to improve their practice. However, all studio clients should consent to recording.
  • Safety. Overlaying a mixed reality display over the real world is a potential safety hazard. Display feeds should not obstruct vision. One solution is for the system to recognise when objects enter the gaze area or within prescribed distance around the user. For example, someone in the real world  “walks through” the display and appears to stand in front of it.

Additional features

The rapidly expanding field of Artificial Intelligence (AI) could take these live camera feeds and provide the user with additional information.

  • Live personal instructor. Monitor a user’s movements and alert them if they move beyond an “ideal” range. For example, when the user’s posture is out of alignment or limbs are in an incorrect position which could lead to discomfort or injury. The alert could be displayed through a visual alert and/or audio message.
  • Guide attention. Highlight within the live camera feed areas on the user’s body which muscles should be engaged for a particular posture. For example, using hamstring muscles rather than back or abdominal muscles.

Future applications

Pilates may seem like a quirky use case to explore but there are serious applications that could benefit from mixed reality assisted proprioception.

Stroke is one of Australia’s biggest killers and a leading cause of disability. One particular problem stroke victims can experience is difficulty planning or coordinating movement known as apraxia. They can also feel slow or clumsy when coordinating movements which is known as ataxia.

In conjunction with a physiotherapist a mixed reality system could become part of a Proprioceptive Neuromuscular Facilitation (PNF) program:

  • Real time feedback. Showing a stroke victim how they walk from different angles as they walk. The visual input helps them make a mental association between what they are seeing with what they are feeling.
  • Mixed reality displays versus mirrors.  Stroke victims may find it difficult to turn their head and look into a mirror while trying to coordinate their own walking movements at the same time. Additionally, it is not always possible to locate large mirrors in many positions around a rehabilitation centre.
  • Voice commands. Stroke victims rely on their hands and arms to steady themselves while learning to walk. Thus their hands are not available to operate a smartphone or other device. While someone else could operate a system on their behalf, voice commands provide stroke victims with autonomy and independence to use the system themselves.
  • Record and analysis. Later viewing can assist a stroke victim to see and understand their own movements. Working with a physiotherapist they can plan future therapy sessions collaboratively.

Serious Games in Singapore

Serious Games Conference (Image courtesy of Asia Events)

It was a real pleasure to attend and speak at the third annual Serious Games Conference in Singapore last week. The third instalment of this annual event was well attended by a variety of speakers and industry representatives from across the globe and local Singapore developers were out in force. A range of speakers including designers, developers and domain experts shared their experience, current developments and where they believe serious games are heading. Companies included Crytek (South Korea), Digitalmill (US), Eduwealth (Singapore)Littleloud Studios (UK), MOH Holdings (Singapore)National Institute Education (Singapore)Playware Studios (Singapore), Ranj Serious Games (Netherlands), Rockmoon Pte Ltd (Singapore), Serious Games International (UK) and yours truly from Hummingbird Interactive.

Education, e-learning and training were popular topics as always, however there was a particular interest in serious games applications for health. Several speakers including Ben Sawyer (co-founder of Serious Games Initiative and DigitalMill who also recently spoke at Games for Health 2012 US) and a representative from MOH Holdings both noted that the health sector is experiencing a major paradigm shift.

Day One Panel (L-R): Michael Bas (Ranj Serious Games), Tim Luft (Serious Games International UK), Dr Koh Noi Keng (NIE), Natalie Marinho (Hummingbird Interactive), Jun Magata (Rockmoon). (Image courtesy of Asia Events)

Instead of focusing primarily on treatment and disease management, health care providers are looking towards prevention and a more holistic approach to health and wellness. Some surprising statistics were mentioned including the fact that the percentage of Singapore residents with diabetes aged between 18 and 69 years old has increased from 8.2% in year 2004 to 11.3% in year 2010 (National Registry of Diseases Office, 2011). Obviously there is huge potential for serious games to contribute to behavioural and attitude change with regard to awareness and prevention of chronic disease such as obesity and diabetes, as well as in the area of mental health.

Having fun while learning about financial literacy through a participatory exercise during Dr Koh’s presentation “Serious Games as a Powerful Pedagogical Tool for Learning”. (Image courtesy of Asia Events)

Gamification was also hot topic, mentioned in several presentations besides my own and it was encouraging to hear that other delegates believe there are potential applications beyond points, badges and leader boards that are most common at this point in time. After describing the current landscape of gamification and its relationship with serious games, I enjoyed some very healthy and enjoyable debate during the networking drinks! However, I strongly believe the future of such games lies in encouraging real world action and participation. Specifically, integrating the game world with the physical world through alternate reality games and rapidly evolving technology in the mobile and augmented realty spheres.

It was energising to meet with so many passionate designers and developers working in the area of serious games and I look forward to more exciting projects being launched out of Singapore in the near future.

This post was first published on RecognitionPattern.com

Nike+ FuelBand: the next evolution

Nike has launched a new product designed to encourage participation in all types of physical activity.

At an unveiling in New York last week, Nike CEO Mark Parker stated, “The Nike+ FuelBand is a way for Nike to further evolve the exciting possibilities of merging the physical and digital worlds. Nike has always been about inspiring athletes, and the Nike+ FuelBand will help motivate them in a simple, fun and intuitive way.”

The Nike+ FuelBand

The FuelBand will no doubt capitalise on the success of the existing Nike+ platform which this blog has covered previously: How Nike Turned Running in to a Game. Whereas Nike+ focused exclusively on runners, FuelBand will open the door to individuals who participate in other activities such as walking, dancing, athletics and team sport.

FuelBand is an accelerometry-based motion tracking system embedded within a wristband. The band measures time, calories, steps and a new type of currency: NikeFuel, a normalised score that awards equal points for the same activity regardless of the individual’s physical makeup. It’s an interesting step away from calorie counting which is a substractive mentality where you want to get rid of something. NikeFuel is an additive notion where the individual earns points through activity.

Data recorded by the band is synced with the Nike+ website (via built-in USB or bluetooth to a free iPhone app) with an interface that facilitates goal setting and review. Professional cyclist Lance Armstrong, also present at the event, said that NikeFuel provides “real information and numbers to show how much people are doing all day, every day. That’s what will get people challenging themselves to do more and better their own scores. It’s a tool to get people more active.”

Information is vitally important to monitoring one’s training progress whether its losing weight, increasing speed or stamina. However, just being able to monitor and review general activity consistently will be a huge step forward. Currently gyms are full of machines that already track your heart rate, calories burned, distance travelled etc. But to export that data you have to use similar branded devices, insert a USB or write it down and then manually add it to a personal spreadsheet or online tracking system. What is missing is a simple process/interface to measure, record, view and then interact with that data. FuelBand may provide that solution.

One can assume that integrating Nike Fuel to the existing Nike+ platform will allow users to utilise social networks, a feature that remains very popular with its runners. With rates of obesity rising in many cities around the world NikeFuel has an opportunity to create a very positive environment where a community of users can motivate, support and encourage each other as they work towards their individual and perhaps collective goals.

The following is an entertaining video from Nike about what activities count and what don’t, when trying to earn Nike Fuel, and features the campaign tag line: Life is a sport, make it count.

You’re reading a post written by Natalie Marinho, originally posted for www.recognitionpattern.com 

Singapore’s Health Promotion Board releases free health app

Singapore’s Health Promotion Board hopes that the release of a new mobile app will help transform the health habits of its citizens for the better.

Released in late October, the Interactive Diet and Activity Tracker or iDAT is a free mobile app that assists users in monitoring their diet and exercise activity. There are already a number of diet and exercise monitoring apps available in the mobile market, however iDAT is the first application in the world to be programmed with a scientifically based algorithm specially designed for Singaporeans. Its calorie recommendations take into consideration the body types of Asians, diet preferences (including bubble tea and char kway teow) and as well as the working lifestyle of Singaporeans.

Available for both OS and Android platforms, iDAT is also linked with a customised website, where organisations and community groups can create their own online campaigns or incentive-based programs and people can challenge each other. “Challenges” are a game mechanic that has been successfully utilised by services such as Nike+, encouraging users to track their progress against others and providing further motivation to reach their goals. It will be interesting to see how iDAT users interact with challenges and each other, and whether the mechanic is fully utilised to support community engagement.

Now you can slice and dance your way through physical rehabilitation

Casual gaming and related technologies are providing medical researchers with creative opportunities to assist patients.

Physical rehabilitation is an essential tool in the recovery of patients following serious injury. However, it can also be a very monotonous and solitary activity. Neuroscience Research Australia teamed up with Half Brick studios, developer of Fruit Ninja, to design a modification to the popular game that allowed patients with stroke and spinal chord injuries to engage in physical therapy through the use of their arms while interacting with the Sony PlayStation Move.

NRA’s work exploring the use of video games and rehabilitation was highlighted earlier this year during an episode of the Australian TV video game review program, Good Game. As Senior Research Officer Dr Stuart Smith says, “Games are going to enable us to really radically transform the way that we deliver health services.” The full clip is embedded below.

Another popular game, Dance Dance Revolution, was modified for seniors in a bid to reduce the number of falls, injuries and fatal accidents suffered by older people. In a paper entitled “Electronic Games for Aged Care and Rehabilitation” (presented at eHealth Networking Applications and Services, Healthcom 2009) researchers described how patients using the dance mat were guided through a series of steps on a video screen with music in order to train them to regain or improve their balance.

Although both examples utilise fully developed games, I believe their use in non-game contexts is a good example of gamification. Hopefully other researchers will also see the potential for applying game elements within their own fields.

How Nike turned running into a game

Established in 2006 with a sneaker embedded sensor, Nike+ has grown to be one of the most recognised examples of gamification to date. With over 750 million miles covered so far, this community of runners has demonstrated how game mechanics can be used to turn a simple activity into a world-wide collaboration.

How it works

While running, the sensor communicates with an iPod Touch or Nano, recording relevant information. The device is then synced to the runner’s profile on the Nike+ website. The system utilizes many game features and mechanics. Runners can compete against themselves (personal best) or compete against others through leader boards, challenges and friendly trash talk. They can monitor and balance the increasing difficulty of their runs through moderated difficulty curves. All the while, runners monitor their progress against an overall goal of their choice (just like a quest) to be achieved through the completion of individual activities (missions).

Why its effective

There are three key lessons for designers who want to utilise game features or mechanics to enhance user experiences.

Understand your users. Nike+ knows what runners are like and what they want. Running is an ongoing activity and improvement can occur over a long period of time. Providing relevant real-time data helps runners to situate each individual run within an overall context of improvement. Moreover, Nike+ recognises that although users may be engaging in the same activity, they may be doing it for very different reasons. The types of goals offered include running more frequently, running further, running fast or burning more calories. Thus the same system can appeal to different kinds of users much like an RPG (role-playing game) can appeal to different players (merchants, warriors, politicians).

Ensure the activity is meaningful and context appropriate. Rather than just rewarding the user with points for every run, the system provides practical data: how far did they run, how fast, is that better than last time? Runners can track progress against their own self-prescribed goals, which have an intrinsic meaning to them rather than abstract badges that have no value outside the system. There’s no confusion here: running and its associated health benefits are the reward, not points.

Develop and engage your community. Nike+ has created a place (albeit virtual) where like-minded people can come together and share their experiences. Moreover this interaction can take place through competitive (leader boards and challenges) or collaborative (sharing information and encouragement) platforms. A runner may train alone but can still feel that they are part of something bigger.

Understanding users and the context in which they will interact with a service, process or product is essential when designing game-based interactions. It allows designers to create a much deeper sense of engagement and richer user experience than points and badges can ever achieve alone.