How Gesture Control Will Become a Reality in Healthcare

With the popularity of gesture controlled devices like the Nintendo Wii, Microsoft Kinect, and Playstation Move showing no signs of slowing down, it’s clear that individuals playing games want to be more physically involved and interact in ways like never before. This increase in physical activity has helped to break the stigma of some games turning us into “couch potatoes,” but is there more of a therapeutic benefit to this type of interaction? I believe there is.

Many assisted-living homes across North America have already adopted programs which utilize the Nintendo Wii to increase physical activity amongst the seniors’ population. This has not only shown improved vitality to those involved, but also their quality of life radically improves in most cases. As a person with a physical disability myself (Cerebral Palsy or CP) and a former game industry employee, video games have always intrigued me, namely for two reasons. The first is because I could have powers and abilities far beyond my own. The second is because I could use my existing abilities, and in my case, increase my fine motor functions through repetitive actions. This is something that both my physiotherapist and occupational therapist had taught me over the years. The more I train myself to do daily processes like tie my shoes or do up buttons, the easier it would be, and yet, it was still never as fun as when I was engaging similar muscle groups in a gaming environment.

Throughout my time in the game industry, I often wondered why software developers didn’t capitalize on the therapeutic benefits of games, but if I could just show them, perhaps they would see things in a new light? By a chance encounter in August of 2012, I was able to do just that. That August, I came across Reality Controls, a software development company based locally out of Vancouver, Canada. What started out as a simple review of their software for AbleGamers.com (a charity which has game reviews for persons with a disability) turned into so much more. I recognized that the software they were developing could be used in therapy and rehabilitation, and as a result, I was brought on as their Director of Communications to help realize this vision.

Marco Pasqua (left) and Reality Controls CEO Sean Sibbet giving their TEDx talk.

One of our projects is called control:mapper. Control:mapper enables a person with limited mobility to create custom templates for mapping their natural body motions and verbal input to any arrangement of keyboard and mouse commands needed for their favorite Windows software. This software would serve as the perfect framework to cater to individuals with a plethora of varying abilities. One such individual was Darrell Wyatt. Like me, Darrell has CP, and after sitting down for only a few minutes with control:mapper to control a game, he had this to say [video]:

It’s more interactive … actually being able to do the movements and make your characters move and do whatever is involved in the game, at least for me, would give me a whole different feel like I’m actually a part of the game. It’s not just a game; you actually have to do something. This would be a way to get exercise that I don’t normally get.

This proved that we could truly impact a lot of people, but now, how were we going to reach them all? In order to do that, we looked into the area of telerehabilitation. We’ve made big strides in this arena, and our work has helped to enhance accessibility and rehabilitation to make a mark on Vancouver’s tech scene.

At the 2013 Vancouver Mini Maker Faire, we illustrated how this is possible, with two networked Kinect workstations enabling individuals to interact in an immersive environment. This is a simple example to show how a client and their therapist interact, using virtual avatars to track the client’s progress, regardless of where they were physically located. Not only would this be ideal for individuals in rural communities, it will support ongoing care for clients who have undergone a stroke or spinal cord injury, without the need to physically travel to a facility for every rehabilitation session.

Marco and Sean at the Reality Controls booth at Vancouver Mini Maker Faire.

We’ve had the opportunity to work with the University of British Columbia (UBC) on a project called “FEATHERS” (Functional Engagement in Assisted Therapy through Exercise Robotics). The aim of the FEATHERS project is to help motivate children/adolescents and adults who’ve had a stroke (or other upper extremity limitations such as Cerebral Palsy) to continue with an exercise program by using social media and online games and robotic interfaces. Having personally gone through 18 years of physical therapy, I know how beneficial it would have been to have a fun and engaging application, which enhanced my therapy sessions. At the same time, my physiotherapist would have loved the tangible results, to track my progress.

This is only the beginning for where we envision Reality Controls is going. We imagine a world where clients don’t have to go into their therapists’ office each visit to get their progress results, but instead, can do their assigned program from the comfort of their own home and have their practitioner monitor these results remotely. Furthermore, with the Kinect 2.0 for Windows coming in 2014 and rumored to be able to pick up not only macro-motions (e.g. arm and leg joints) but also micro-motions (e.g. finger and eye movements) along with voice-control, we’ll be able to reach and support more individuals regardless of their abilities.

By building strong relationships with some of today’s leading educators and healthcare providers, we feel it is only a matter of time before this is not just a possibility, but a reality.