Codebox: Amuse yourself with Google Autocomplete

Autocomplete is one Google’s many interesting features. For example, if you start typing “How do I make” into the search box, Google presents a top-10 list of things that it thinks that you might be searching for. Generated by analyzing millions of similar searches, this collective “wisdom of the crowds” can be hilarious, tragic, or deeply indicative of the human condition. Often, it’s all of these at once.

This sketch lets you explore the Google zeitgeist of a phrase you select. It works like this. When you enter a phrase like “How do I make,” the sketch creates 26 variations by adding each letter of the alphabet at the end: “how do i make a,” “how do i make b,” “how do i make c,” and so on. It then queries Google for each variation and adds the ten results to a running list. Befitting the inherent drama of seeing the hopes, dreams, and aspirations of millions, the 260 results are presented as an iconic “Star Wars” scroll:

The Autocomplete API

Autocomplete is powered by a relatively simple URL that looks like this.

http://google.com/complete/search?output=toolbar&q=encoded search term

Unlike most pages you visit, which present pretty HTML, this one returns XML data that is meant for machines to read. Known as an Application Programming Interface (API), this incredibly powerful technique allows programs to easily communicate across a network. Without an API, we’d have to go to Google, type in our phrase 26 times, and then collect all the results by hand. The API lets us do this automatically.

To get the underlying XML data for the “How do i make” suggestions, all you have to do is visit this URL:

http://google.com/complete/search?output=toolbar&q=how+do+i+make

If you “View Source” on your web browser, you’ll see some XML that looks something like this (although I’ve formatted this example to be more readable):

<?xml version="1.0"?>
<toplevel>
  <CompleteSuggestion>
    <suggestion data='how do i make my hair grow faster' />
    <num_queries int='300000' />
  </CompleteSuggestion>
  <CompleteSuggestion>
    <suggestion data='how do i make a heart on facebook' />
    <num_queries int='200000000' />
  </CompleteSuggestion>
  ...
  <CompleteSuggestion>
    <suggestion data='how do i make a group on facebook' />
    <num_queries int='970000000' />
  </CompleteSuggestion>
</toplevel>

We’ll use Processing’s XMLElement() library to read this XML data to create the scrolling text.

Set up the Sketch

First, though, let’s get the sketch running. Since we’re using the controlP5 library to get the query phrase, you’ll need to have that library installed and configured. (If you don’t, the controlP5 section of Codebox: Create a fractal menagerie will tell you how to set it up.) You can grab the sketch’s source code from the box below or from the zetigeist.pde file.

Once you’ve gotten the source code into Processing, start the sketch and type in a phrase. After a few seconds, you should see the results start scrolling across the screen a la Star Wars. (If you don’t enter any text, the sketch will simply present the most popular results for all general queries.)

Discussion

So, how does this thing work? There are a few key elements:

1. It grabs a batch of data from the Autocomplete API
2. It parses the XML with XMLElement() and save the results
3. It displays the results as a scrolling list

So, let’s talk about how to grab the data from the API. Processing’s loadStrings() function allows you to grab the text of any URL, so all we have to do is present the Autocomplete API with a properly formatted query. To do this, we’ll use Java’s URLEncoder library (remember, everything you can use in Java is fair game for Processing) to write the encode() function, which encodes the text entered into the userQuery box.

The getSuggestions() function uses our encode() function to fetch the page and parse the XML results. The function uses the XMLElement() object to work with the results. Like most XML libraries, this one uses a “family tree” like syntax to allow you to access the various data elements. As shown in the next figure, the suggestion data is the first child element (<suggestion>) under of root element’s children (<CompleteSuggestion>).

Since the actual text is help in the attribute, we use the getStringAttribute() function to pull out the data we want.

[Note: that this syntax is different than the official documentation, but this post on Processing’s community forum, XMLElement problem: function getInt/getString ‘does not exist’ provides the details.] Once we’ve grabbed the text of the suggestion, we’ll append it to the suggestionList ArrayList.

Finally, it’s worth explaining a bit about how we actually call the getSuggestions() function. As you’ll see, it’s actually called in the draw() method, which is a bit unusual. Initially, I tried to slurp up all 26 query variants at once by making a series of rapid hits against Autocomplete. I quickly hit the API’s rate limit, which is the maximum number of times per second that you can use it at any given time. In addition, there was a long pause at the start of the sketch that I didn’t like.

So, instead, I added a timer called timeBetweenFetches so that the API was only called every 5 seconds. Since the text is scrolling by rather slowly, this allowed the results to buffer up over time, leading to a smoother overall look without violating Google’s requirements. Each time we hit the API, we advance a counter that tells us which letter of the alphabet to add to the base query. Once we get the results, we reset the timer. There’s also a bit of logic in there to test if the user has typed in a new query term and pressed the findButton.

The other piece of the draw() method handles the scrolling text. To do this, we use a variable called y to hold the y-coordinate of the first string in suggestionList. We then loop through each element in the list, and write the text on the screen in reference to the variable y, which is decremented on each pass through draw(). This gives us the nice, scrolling effect. (We also center the text on the x-axis, which is done with a simple formula.)

Finally, the Star Wars effect is easily achieved by using Processing’s built-in 3D Graphics Library to rotate the plane of the X-Axis by a few degrees. This was a neat hack that I saw on Luis Gonzalez‘s “Star Wars” sketch on openprocesing.org. I highly recommend this site as a source for inspiration and ideas.

More:
Check out all of Andrew’s Codebox columns here.

In the Maker Shed:

Getting Started with Processing
Learn computer programming the easy way with Processing, a simple language that lets you use code to create drawings, animation, and interactive graphics. Programming courses usually start with theory,but this book lets you jump right into creative and fun projects. It’s ideal for anyone who wants to learn basic programming, and serves as a simple introduction to graphics for people with some programming skills.