Codebox: Lose your head with OpenCV

OpenCV is a “library of programming functions for real time computer vision.” An open source project supported by Willow Garage, the library contains over 500 algorithms for image manipulation, object detection, feature extraction, and a variety of other tools you can use to write programs that can “see.”

The OPENCV Processing and Java Library, a project maintained by Stéphane Cousot and Douglas Edric Stanley at the École Supérieure d’Art d’Aix-en-Provence, provides a wrapper that allows you to use a subset of OpenCV in Processing. In honor of Halloween, this Codebox will show you how use OpenCV to remove people’s heads in real-time from a video feed. For example:

Future posts will cover less gruesome applications for this awesome library.

Installing OpenCV Processing and Java Library

The fist step in using the library is to download and install it from http://ubaa.net/shared/processing/opencv/. The site’s documentation is great — just follow the steps listed on the “Installation Instructions” on the main page. Once you’ve followed the steps, fire up Processing, and then try out the code in the detect() example, which puts a red rectangle around any faces (frontal view) detected in the webcam. You should see something like this:

Note for Windows users: you may have to installQuicktime and WinVDIG to use the webcam in Processing.

Setting up the Sketch

Once you’ve got the basic detect() example working, you can use the following sketch. As with other posts in this series, you can either highlight all the text in the box below and press ctrl+c, or go directly to the Makezine code repository and copy headless_cam.pde.

When you start the program, you should see a message to “step out of the scene and press any key.” Once the coast is clear, press any key and walk back into the image. You (and anyone else) should, thanks to OpenCV, be headless. As you move around you’ll notice that your head will wink in and out — this happens when something interferes with OpenCV’s detection algorithm. For example, if you put your hand in front of your face, this will “break” the OpenCV detection algorithm.

Discussion

So, how does this thing work? It’s pretty simple, actually — all the sketch does is search for faces using OpenCV’s detect() method and then replace the detected faces with the same (but slightly bigger) pixels from the background image saved at the start of the sketch. This makes the face “disappear,” mostly. Slight differences in lighting create small changes between the new and old images, making the replaced face appear a bit boxy. In theory, you should be able to smooth some of these out with other OpenCV functions, but that’s for future projects.

The first step in the sketch is to save a picture of the empty background scene. This is done by the copy() command in the keyPressed() method:

 bg.copy(cam,0,0,width,height,0,0,width,height);

The command requires two images: a destination image, which is where the pixels will be copied to, and a source image, which is where the pixels will be copied from. To use it, you call the copy() method on the destination image (in this case, the variable bg) and supply the nine (!) required arguments, which are:

• The source image from which we want to make a copy. In this case we’re using the variable cam, which contains the current frame from the webcam.
• The area in the source image that is to be copied. This area is described by the next four arguments: the x and y coordinates of the upper left-hand corner of the region, and the region’s width and height.
• The area in the destination image where the copied pixels will go. These are similar to the prior four arguments, and consist of the x and y coordinates, along with the area’s width and height. If the dimensions are different between the source and destination areas, Processing will scale the image automatically.

After this command completes, you’ll have a copy of the background scene saved in the variable bg. The next step is to use OpenCV to search for faces. As described earlier, this is done using the detect() method, which returns an array of Rectangles. A Rectangle is a class that is included in Java, the base language from which Processing is derived. (One of the cool things about Processing is that it can use any command that you can use in Java, making it a very powerful language indeed.) A Rectangle has a four key fields that concern us: the x and y coordinate of its upper left hand corner, and a width and height. (Sound familiar?)

If all we wanted to do is replace just the face, we can simply use the copy() command to copy over the equivalent regions. (Note: the loadPixels() and updatePixels() method are required whenever we want to access or update the pixels in an image.) Here’s the code:

     fg.copy(cam,0,0,width,height,0,0,width,height);  //Copy the camera image into  fg
     opencv.copy(cam);  //Copy the image into openCV's buffer
     Rectangle[] faces = opencv.detect();  // detect anything ressembling a FRONTALFACe
     for( int i=0; i<faces.length; i++ ) {
         fg.loadPixels();
         fg.copy(bg, faces[i].x, faces[i].y, faces[i].width, faces[i].height, faces[i].x, faces[i].y, faces[i].width, faces[i].height);
         fg.updatePixels();
      }

Here’s the output:

As you can see, removing only the face is not quite the right because a lot of the head remains visible. So, we’ll also need a way to increase the size of the box. This is the job enlargeFaceBox(), which uses a bit of trigonometry enlarge the box by a given percentage and translate it along its diagonal. The following diagram shows what’s happening:

The trig comes into play when calculating the correct displacements that will slide the rectangle up the diagonal. The key function is atan2(), which calculates the angle between two points relative to the x-axis. ( This angle is often called theta by hoary old mathematicians). Once you have theta, you can use the cos() and sin() functions to compute the x and y displacements dx and dy. Here’s the code:

Rectangle enlargeFaceBox (float incPct, int x, int y, int w,int h) {
    float r = dist(0,0,w,h) / 2;  //Computes radius of the center diagonal
    float theta = atan2(h,w);  //Computes the angle of the diagonal
    float dx = r*incPct*cos(theta); //Finds
    float dy = r*incPct*sin(theta);
    return new Rectangle( (int) (x - dx), (int) (y - dy), (int) (w + 2*dx), (int) (h + 2*dy));
}

Finally, since we’re enlarging the box, we need to make sure that it remains within the visible areas of the screen, just like we did in the Swat an (arraylist) of targets project, which is all done here:

    faceBox = enlargeFaceBox(0.75, faces[i].x, faces[i].y, faces[i].width, faces[i].height);
    if (faceBox.x < 0) {faceBox.x = 0; }
    if (faceBox.x + faceBox.width > width) { faceBox.width = width - faceBox.x; }
    if (faceBox.y < 0) { faceBox.y = 0; }
    if (faceBox.y + faceBox.height >  height) { faceBox.height = height - faceBox.y; }
    ...

And, that’s it. Hope you can find a creepy use for this on Halloween.

PS: You can learn the full details of how to work with images from Daniel Shiffman’s excellent Images and Pixels tutorial on the Processing.org site. (The chapter is an excerpt of his wonderful book, Learning Processing: A Beginner’s Guide to Programming Images, Animation, and Interaction, which I highly recommend you pick up if you want to really master Processing.)

In the Maker Shed:

Getting Started with Processing
Learn computer programming the easy way with Processing, a simple language that lets you use code to create drawings, animation, and interactive graphics. Programming courses usually start with theory,but this book lets you jump right into creative and fun projects. It’s ideal for anyone who wants to learn basic programming, and serves as a simple introduction to graphics for people with some programming skills.