Gestural Interaction in the Operating Room with a Wii Remote and Kinect

Computers & Mobile
Gestural Interaction in the Operating Room with a Wii Remote and Kinect

3D Thursday Banner
Future OR
A rendering of an operating room of the future, with Tamara’s devices installed.

Student Tamara Worst has completed a fascinating project with Interaction Design group IxD Hof, in cooperation with Siemens Healthcare, that frees doctors from using their hands for menial tasks during operations. Siemens was looking for a non-contact interaction solution for surgeons in the operating room, so Tamara used two 3D sensors, a Wii remote and Microsoft Kinect, to detect the orientation and position of the surgeon’s right foot. She then wrote software in Processing to allow them to view and manipulate images and 3D models without the use of their hands.

Shoes + Wiimote
A Wii Remote attached to the right shoe. In the future Tamara hopes to replace the bulky remote with a single accelerometer.

The Wii remote and Kinect sensor each accomplish a different task – the Wii remote measures the orientation of the doctor’s foot, allowing precise control over the view and orientation of images and models. The Kinect sensor detects the position of the surgeon’s right foot, allowing him or her to accomplish different tasks, depending on where the foot lies.

Image Actions:

  1. Browse catalog of images – lean foot forward and back to scroll forward and back.
  2. Change contrast of images – lean foot left and right to increase and decrease contrast.
  3. Zoom – zoom in by stepping once forward, zoom out by stepping once backward.

Floor Setup

The doctor can switch between 3D and 2D views by moving his or her foot one step to the right and leaning forward and back. Moving back to the “home” position will select the current mode.

3D Model Actions:

  1. Rotate object – lean foot left and right.
  2. Tilt object – lean foot forward to rotate away and back to rotate towards.

3D Part Manipulation
Manipulating an object in 3D.

In this lengthy post, Tamara goes in depth into her prototyping process. She initially used simple pressure sensors to detect different foot movements and orientations but found that the wires required to connect to the sensors were clunky, annoying, and restricted movement of the surgeon in the operating room.

I particularly like one of the possible improvements she suggests: using a wireless multitouch system in the shoe to detect different doctors based on the force profile created by their feet. This would allow each doctor to pair specific functions and foot orientations at will; the device could then recognize different doctors and change the computer interface and features automatically.

How would you use these two powerful sensors to accomplish a helpful task in the OR? In what other fields would a 3D model viewer like this be useful? Share your thoughts with a comment below.

14 thoughts on “Gestural Interaction in the Operating Room with a Wii Remote and Kinect

  1. James Bryant says:

    A two- or three-axis accelerometer, with rechargeable battery and radio interface, could easily be made small enough and tough enough to embed in the heel of a shoe.

    1. Eric Weinhoffer says:

      Yeah, and she could probably fit a whole Xbee and breadboard in a shoe if it has a big enough heel :)

  2. designporvida (@designporvida) says:

    Fantastic idea! Did Tamara and team get any feedback from real doctors? I advanced to the links embedded but didn’t see a mention of feedback from doctors and didn’t want to try to translate her blog entry :)

    1. Eric Weinhoffer says:

      I translated her post with Google Translate and was able to understand 95% of it. And as far as I can tell, she didn’t mention any feedback from doctors. I hope that’s what she’s working on now :)

    2. tamara worst says:

      I got some feedbacks from doctors:

      Feedback 1:
      The doctor said that the footprints on the floor and the control catalogue on the monitor is very useful.

      Feedback 2:
      The distance between the mainposition and the zoomposition should be very small.
      You can minimize the distance if you use colored footprints.
      The Kinect could differentiate them really easy.

      Feedback 3:
      The Kinect doesn’t divice between wanted and unwanted movements.
      To deal with this problem you can install a button in the left shoe.
      If you push the button the Kinect and the Wii Remote start to read out the movements.

      There you can get the English documentation.

      Thanks for every post :)
      If you have some more questions, then let me know them.

      Kind regards


      1. Eric Weinhoffer says:

        Thanks for this, Tamara! I’m glad you got some helpful feedback from Doctors :)

      2. designporvida (@designporvida) says:

        Great to know Tamara and thanks for the reply. My first inclination was that this might be a lot of movement (stepping up-down-side) for a surgeon, but I’m not one. It seems like feedback two referenced above starts to address that.

        It would be really great to see a qualitative study with doctors with all ranges of ages. I wonder if the younger doctors would respond more favorably vs less vs neutral. (Not to draw conclusions :) Really great exploration, I’m fascinated by interaction/software solutions for medicine– we need more!

Comments are closed.

Discuss this article with the rest of the community on our Discord server!

Eric is a Mechanical Engineer with interests in machining, mass manufacturing, product design and kinetic art. While not building things, he enjoys skiing, cycling, and juggling.

View more articles by Eric Weinhoffer
Maker Faire Bay Area 2023 - Mare Island, CA

Escape to an island of imagination + innovation as Maker Faire Bay Area returns for its 15th iteration!

Buy Tickets today! SAVE 15% and lock-in your preferred date(s).