Find all your DIY electronics in the MakerShed. 3D Printing, Kits, Arduino, Raspberry Pi, Books & more!

Tangible interface designer and inventor Andrea Bianchi, along with his colleague, Ian Oakley (University of Madeira / Carnegie Mellon Europe), have come with a novel approach to interacting with a mobile device. Using the magnetometer built into most modern smartphones, Bianchi and Oakley have created a series of tangible user interface demonstrations that go beyond what’s achievable with capacitive touch displays.

We caught up with Andrea over the weekend as he prepares to deliver a presentation at the upcoming ACM TEI 2013 conference in Barcelona to ask him a few questions about his technique.

MAKE: How does this differ from capacitive touch tokens?

Bianchi: First of all, capacitive tokens need to occupy (often relatively large) portions of the screen, while magnetic tokens can be located anywhere around the screen. In the video many tokens are located on the screen to simplify the calibration process: since the location of the token is know in advance (there is a “put the token here” marker on the GUI), then it is trivial to detect the position/orientation of the magnet. However, all the techniques shown in the video can work equally well off screen (for example assuming that we put all the tokens on the left or right of the screen).

Moreover, the capacitive tokens cannot be passively sensed, requiring either human contact or active electrical components to simulate finger touches (here an example). However, magnetic tokens do not require the user to keep touching them, nor they are active components (not batteries, just magnets…)

MAKE: Assuming you don’t have to touch the device for this technique to work, what is the practical distance it can be used?

Bianchi: It depends on the strength of the magnets and the interferences of other magnetic fields. Assuming there are not other strong magnets around, usually the device will sense the Earth magnetic field (0.25 to 0.65 gauss). The 3 magnets we used were very strong in comparison (Small: thickness 2mm, diameter 0.5 cm, 400 gauss; Medium: thickness 2mm, diameter 1 cm, 1000 gauss; Large: thickness 2mm, diameter 2 cm, 1500 gauss) so we had no problem detecting them, and you can imagine that we could have used a much richer set of identifiable magnets. To avoid noise and keep detection reliable, I would say that empirically we found that 10cm from the device border is about as much as you want to get, but this can be probably improved.

MAKE: What is the smallest magnet you have used? How about the largest? Does physical size matter?

Bianchi: Magnetic fields have two properties: strength and direction. Strength is the intensity of a magnetic field and it varies from magnet to magnet and with distance. Direction reflects the fact that magnets have a north and south pole. Flipping a magnet inverts the poles, causing substantial changes to the magnetic field. Both strength and direction can be measured with a gauss-meter or via magneto-meters and compass sensors. Hence, the magnet physical size (see previous question) matters. The strength of a magnetic field is affected by the size of a magnet. Our software simply sense measurable changes on the magnetic field and try to use them to build novel interactions for exploring the design space. We could have used more than three magnets, but we just adopted a simple approach for the demo.

MAKE: How many magnets can you use at once?

Bianchi: Our techniques leverage on the detection of the magnetic field strength (e.g., detecting position, or size, linear movement), or orientation (e.g, flipping, rotational movement), or both (e.g., orientation). Since the software only reads a cumulative value of the magnetic field strength and direction, it cannot know if for instance a “small intensity” is due to a small magnet or a strong magnet that is far away. Generally though you can use multiple magnets if you measure different (orthogonal) properties (e.g., one magnet will be used for the strength, one for the direction) or if they are used together to achieve a combined effect (e.g, snapping two magnets together makes a stronger field, so we can identify this action). I attach a table that show you how these techniques can be combined together. So for instance, “flipping and position” or “flipping and identification” leverages on orthogonal properties, so we can use 2 magnets at the same time. “Position and identification” leverages both on the magnetic strenghts so only one token at a time can be used. This problem can be solved either introducing a more complex calibration, or constraints on the movement (e.g., only few targets) or using actively powered magnets (e.g, solenoid) which could pulse at different identifiable frequencies.

MAKE: Can this break my device?

Bianchi: It is usually good to keep (strong) magnets away from electronic devices. In practice though, I did not find any damage or malfunctioning of my devices (phone and tablets) during or after the usage of magnetic appcessories. 

MAKE: Are you using a particular platform to develop and if so, why?

Bianchi: We used Android on a Samsung Galaxy Tab simply because developing a prototype for Android is extremely simple. All this work was basically built in few days.

MAKE: How can I get started using this in my app? 

Bianchi: We have not released an app yet, but we are considering working on a open source toolbox to help other developers working with magnets. This work is however still very young and require some few iterations. Extensions of this work will investigate better ways to calibrate magnets, tokens that snap together (creating recognizably stronger magnets), explore the potential of active magnetic tokens (e.g., electromagnets pulsing at different frequencies) to create larger sets of uniquely identifiable tokens and combine magnetic sensing with capacitive sensing.

MAKE: What are some of your favorite examples of this technique in practice?

Bianchi: When I was designing these tokens I was thinking to use them for DJing. So, that’s the inspiration for sliders (faders) and wheels (volume and gain controllers) or even menu selections. The main idea is that we could use tangible interaction with commonplaces devices for some activities (e.g., DJing) that seem to work much better with physical widgets than not “beyond-the-glass” graphical interfaces. I also can imagine how magnetic appcessories could be used for making toys.

Adam Flaherty

I make cool stuff and write about other people making cool stuff on makezine.com. If you have something you think I should see, send me a tip.


Related

Comments

  1. Trav says:

    I could also see this being used for someone with disabilities. For example someone with motor control issues who would have hand dexterity problems.

    Q. would the magnets effect the calibration of the sensor? would it have to be re-calibrated after using the magnets before you could use it for navigation?

  2. David says:

    EXCELLENT and VERY useful for Disabled.

    Consider: ONE IN FIVE Americans, twenty percent, minimum, have a hearing problem whether they know it or not. This could be YOU and IS someone you know or DO contact.

    So my comment in forwarding this story was:
    Good visual demonstration of various interactions.
    Audio is in heavy foreign language accent. Captions have poor-to-funny word capture.

    { cc: Andrea Bianchi via his website.}

  3. Bob says:

    The problem I see here is that putting a strong magnet on the device might magnetize seel parts inside it. These wouldl then cause problems with the magnetometer accuracy.

  4. Dogger says:

    Imagine if a tablet contained 3 well-spaced magnetometers, chess could be played on it, where each chess piece contained a different strength+orientation magnet.

  5. Jeffeb3 says:

    The first thing I would do is to create some real dice that could be used with a digital board game (If you haven’t played Settlers of Catan for Android, I recommend it). I wonder how you could use this to measure the orientation of two dice, thrown in a random location on or near the tablet…

  6. redfood says:

    You can do everything except for the first and last demo w/o magnets. Just put carefully spaced conductive pads on your objects. You can get the distance between them (since they look like fingers to the screen) and presto – you know the object and its orientation. And it can move around the screen too. This is how disney does their app mates -

    http://www.engadget.com/2011/09/27/disney-appmates-cars-toys-mean-parents-may-never-see-their-ipa/

    The last demo can easily be replicated using the rear facing camera.

    The first demo waving around the x,y,z is the most impressive and the only one that adds real functionality. I suspect that it isn’t that accurate or I think we would have seen more than some quick hand waiving.

  7. john brohan says:

    Ultra Super Cool
    I work with Android Tablets for the elderly, and one of our problems is to simplify the user interface to null, so that it cannot go wrong (there are NO wrong buttons). So far we have made some progress by just showing events for a limited time, so no interaction is needed. For example in a text message for the hard of hearing, it just pops up for 10 minutes and the old person picks up the phone and talks to the sender, who sends another text.
    We are looking at Brrds http://www.robotgrrl.com as a project for grandchildren to MAKE for their grandmas, using a DTMF interface which more or less works (DTMF communications I mean). The DTMF sends 3 tones which can be decoded in the Android and used to select an activity just like the magnets. We could look at the magnets too. I’d love to see some code!

    Check http://www.medstartr.com/projects/75-cook-safe and our main site http://www.stay-in-touch.ca

  8. Majakk says:

    Nice! Presented a demonstrator using this technique this spring at ACM CHI 2012 – Mobile ActDresses. http://dl.lirec.org/papers/Jacobsson_CHI2012_Interactivity.pdf

  9. john brohan says:

    This is super-cool. I write apps for aging-at-home, and a continuous problem is to provide ease of operation with avoidance of “The Wrong Button”. To a large extent not requiring any user interaction is feasible (It does family pictures, Hard-of-Hearing texts and pill-reminders all by itself, setup remotely.
    What the magnetic interface shown here offers is an application specific user interface. The magnets only work for the Stay-in-Touch app and will not lead to unexpected places like the clicking of uncertain buttons. I would love to see the code and try and bring in some of this functionality to get to “Phone Peter” or “What’s on Youtube” . Thank you Mm Bianchi it’s great!

    In the same vein we got DTMF to work eh, more-or-less. There is a DTMF receiver on the tablet and it sends a message (BroadcastIntent) to trigger these actions on the Stay-in-Touch program. I have in mind to use Brrds (www.robotGrrl.com) to build a fun interface that cannot go wrong!
    Main site for app http://www.stay-in-touch.ca
    Two minute Video at http://vimeo.com/50024262 (Starring Auntie Ollie!)
    http://www.medstartr.com/projects/75-cook-safe (no User Interface) Did I leave the stove on?

  10. [...] Multi-touch screens are just amazing. The early demos of Microsoft’s Surface computing environment included some intriguing games.  Andrea Bianchi and Ian Oakley have demonstrated a series of magnetic “appcessories” (ugh)… interface elements based on magnets. [...]

  11. Simon says:

    Nice project,

    I wonder if you still can use the compas, while using the magnets.
    and was wondering if you have found new ideas or problems these last months.
    Thanks

In the Maker Shed