Air Guitar Hero

Step #1: Modify the guitar controller.

Our system needs to send computer-generated commands to the game console. We do this by sending digital output commands from the USB-1208FS data acquisition board. Our quick-and-dirty approach is to modify the guitar controller and override the manual buttons you’d normally press.
Make sure your Wii Remote is disconnected from the guitar controller. Unlock and remove the guitar neck and the front faceplate. Remove the four #0 screws on the front of the guitar and the eight T-10 Torx screws on the back of the guitar to open the controller. (Don’t forget the gratifying removal of the “Void Warranty” sticker to get to the last Torx screw).
Using a Dremel tool, bore out a small D-shaped hole to fit the DB9 solder pot connector. Score along the outline of the connector, incrementally removing material and checking fit with the connector. Be sure to leave enough material on the sides of the connector for the mounting screws.
Once the connector fits, score the center location of the mounting screws, drill tiny pilot holes, and insert 2 small screws (or use 2 of the 8 Torx screws you removed), ensuring that everything fits.
Remove the solder pot connector for wiring. Add seven 12" leads to solder pot pins 1–7 (one ground, plus 5 note buttons, plus the Strum button). Connect pins 1–6 to the 6 through-hole pads at the base of the guitar neck, from right to left if the push pins are on the bottom. Connect pin 7 to one of the Strum switch contacts; either Up or Down will suffice. Ensure all wires are secure and close up the guitar.
Check your work! Use a multimeter to perform a continuity check between pin 1 (Ground) and each of pins 2–7 (Green, Red, Yellow, Blue, and Orange buttons, and Strum Up/Down). There should be no connection until each corresponding button is pressed.

Step #2: Configure the data acquisition board.

Next, use a 3' section of serial cable and wire it to the digital I/O pins of the USB-1208FS data acquisition board. You only need half of the 6' serial cable, so cut off the male-pin half and use it to wire to the data acquisition board.
Connect pin 1 of the serial cable to terminal 29 (GND) of the data acquisition board. Connect pins 2–6 of the cable to digital ports.
A0–A4 (terminals 21–25) respectively, and pin 7 of the cable to digital port A7 (terminal 28).

Step #3: Verify computer control of the game.

At this point, a good intermediate checkpoint is to connect the data acquisition device and test Guitar Hero in Auto-Play mode. By routing the video signal to your PC, Matlab can detect each oncoming note from the game and then automatically send the digital signal to play that note for you. The software looks for the bright pixel intensities above the fret- board within Guitar Hero and then plays the corresponding note. (This is especially useful if there’s a song in Expert mode that you just can’t beat!)
Connect the USB video capture device to the Wii and to your computer, and install the drivers on your computer. This will allow the game to run on your PC.
Optional: Download and install Audacity (http://audacity.sourceforge.net) to allow “Software Play-through” of the game’s audio channels through your laptop speakers.
Attach the USB-1208FS to your computer and install the drivers from http://makezine.com/go/usb-data. Once the drivers are installed, open the configuration utility (Start → All Programs → Measurement Computing → InstaCal), and ensure that the device appears in the list of detected PC boards and that it’s configured for 8 single-ended analog inputs (as opposed to 4 differential inputs).
Download the MiniVIE package from the Subversion repository at http://makezine.com/go/minivie. This is a package library based on the Johns Hopkins University Applied Physics Laboratory Virtual Integration Environment (JHU/APL VIE). It has a basic signal simulator that you’ll use to test the functionality of your modified guitar controller.
Open Matlab and navigate to the MiniVIE directory. Type the following commands at the Matlab prompt:
>>MiniVIE.configurePath >>Presentation.AirGuitarHero.AGH

Step #4:

The Wii console will display on your PC within Matlab. Use the guitar controller to select and start the game. Once the song begins, click Auto-Play. The MiniVIE will graphically detect notes and then play them using the digital output pins automatically!

Step #5: Train the pattern recognition system.

Before you hook up electrodes and do anything with the EMG signal, it’s worthwhile to get a feel for how the system operates. This can be done with the EMG Signal Simulator that’s built into the MiniVIE system. You’ll use the simulator to walk through each step of acquiring, processing, classifying, and displaying signal outputs.
Start the MiniVIE by typing in Matlab:
>>MiniVIE
In the user interface that appears, select the EMG Simulator from the Inputs drop-down dialog. A small window will pop up in the bottom left side of the screen. When the window is selected, typing keyboard keys a, s, d, f, etc., will generate simulated signals as if a real DAQ system was present and connected to electrodes.
Preview the signal patterns by selecting the SignalViewer button. Each time a valid key (a, s, d, f) is pressed in the Signal Simulator window, a new pattern of simulated EMG activity will appear on the screen. Typically one of the selected channels will get “noisier” in both frequency and amplitude for a selected pattern.

Step #6:

Next, choose linear discriminant analysis (LDA) as the pattern recognition algorithm to use for decoding signals by selecting LDA Classifier from the Signal Analysis drop-down menu. The classifier will be used in the following steps to attempt to recognize patterns from the input signals and label them as one of several discrete classes. Click on Select Channels and select Index, Middle, Ring, Little, and No Movement as the classifier’s available choices.
To use this pattern classifier, you need to create a training data set. From the Training drop-down menu, select Simple Trainer. Click the Begin Training button and a series of prompts will begin asking you to perform each of the movements you selected for the classifier (i.e. Index, Middle, Ring, Little). Each prompt begins by displaying the current class as text while a red bar moves from left to right.
When the bar turns green (or slightly before), recording begins and you should perform the requested motion. In this case, while the green bar indicates data is recording, press the f key with your index finger. When the next prompt appears for Middle, press the d key, then s during Ring, and finally a during Little. In each case you’re indicating to the software to generate the EMG pattern that will appear once you attach real electrodes.
After completion, you can save your training data in the dialog that appears. Resulting accuracy is displayed in the console window. For example:
Training LDA with 551 Samples (1=105; 2=109; 3=111; 4=112; 5=114;) Percent correctly classified: 100.0% Index Class accuracy: 100.0% Middle Class accuracy: 100.0% Ring Class accuracy: 100.0% Little Class accuracy: 100.0% No Movement Class accuracy: 100.0%
Once the training set has been created and the pattern recognition algorithm has been trained with output data, you can visualize these effects in a number of ways. From the Presentation drop-down menu, select the MiniV visualization utility. Pressing the keys a, s, d, and f will cause the system to recognize simulated finger contraction patterns and display the corresponding finger movement in the visualization.

Step #7: Acquire EMG signals from finger movement.

Air Guitar Hero processes electromyogram (EMG) signals to decode your finger movements. To acquire these signals, you’ll connect the input side of the electrodes to your arm via stick-on electrodes and a snap connector.
The EMG signal is a high-frequency white noise signal of several millivolts, with the most frequency energy in the 100Hz–400Hz range. The amplifiers act as an instrumentation (differential) amplifier with a variable gain (approximately 1,000x), which rejects common-mode signal (e.g., the 60Hz line noise coupled into our bodies) and amplifies local biopotential differences caused by muscle contractions (i.e., the EMG signal).
The USB-1208FS data acquisition system has a +5V power connection that can be used to power the electrode amplifiers.
The Otto Bock electrode amplifiers have 2 wires plus a shield on the input side (skin- side, unamplified), and 3 wires on the output side (amplified): +5V (power), ground, and signal. The output signal is centered at 1.2V.
If you build the electocardiogram (ECG) circuit from the schematic in Medical Instrumentation: Application and Design, note that its voltage and frequency ranges are slightly different than EMG amplifiers. EMG amplifiers have wider frequency response than ECG amplifiers, but do not cover so low a frequency range. The EMG amplifier must have higher gain than the ECG amplifier for the same output-signal range.
Disconnect your laptop from the wall supply. Although there’s minimal risk here, with electrodes attached directly to your skin it’s best to take every precaution to ensure you’re isolated from any power fault your building might experience. This also helps to ensure that line noise isn’t coupled into the data acquisition board.
Solder 2 snap-connectors to the input end of each electrode amplifier. Additionally, solder a third female snap-connector to the ground shield of one of the electrode amplifiers.

Step #8:

Connect all the electrode amplifier output ground wires to the data acquisition board’s digital ground (terminal 31) and all the electrode power wires to the board’s +5V (terminal 30). Connect electrode signal outputs indi- vidually to analog input channels CH0–CH3.
Apply stick-on electrodes equally spaced around your forearm, about 1" below your elbow. Place the stick-on ground somewhere in the middle of your arm about 1" away from other electrode contacts. Snap on each amplifier lead wire to the electrodes.
Start the MiniVIE program again by entering >> MiniVIE at the Matlab com- mand prompt. In the pop-up GUI, select DaqHwDevice from the drop-down menu, then press the Signal Viewer button to preview and display the signals. Verify that the signals displayed correspond to the contraction of your forearm muscles by moving your wrist and pinching each of your Index, Middle, Ring, and Little fingers.

Step #9: Rock out!

Once you’ve tested the individual components, it’s time to put it all together to check out your Air Guitar Hero chops.
Disconnect your laptop from the wall supply. Ensure that the EMG electrodes are well attached to your forearm (too much electrode gel on the adhesive can make them fall off).
Open MiniVIE and select your input hardware device. Use the Signal Viewer to verify that the muscle signals are at rest when you’re at rest; you should only see EMG activity when you’re actively pinching your fingers.
Select the LDA classifier and choose Index, Middle, Ring, and No Movement, initially for beginner mode. Train each of these classes using the Simple Trainer interface. Verify that your accuracy is at least 80% or higher.
Start your Wii console and select a song to play using the guitar controller, then select AGH from the Presentation drop-down and run it. In the default configuration, you don’t have to use the Strum button. Just press your fingers when it’s time to play the note, and it will be played with a strum added.

Step #10:

That’s it! Now you can play Guitar Hero simply by making muscle contractions and thinking about playing the desired note.
And after you become a Rock God, you can modify your Wiimote — see our WiiEMG hack at http://makezine.com/go/wiiemg — to play games like Wii Sports Tennis or Mario Kart!