Last October, a few of us were at the O’Reilly Strata conference in New York building a distributed sensor network — 40 bundles of sensors scattered around the conference venue, measuring and quantifying the environment, all connected together by a Digi XBee-based mesh network.
Coming out of a conversation between Edd Dumbill, Tim O’Reilly and myself at O’Reilly OSCON earlier in the year, this was the birth of what has become the Data Sensing Lab — hardware hacking for data scientists. It was also the start of a slippery path that would take Kipp Bradford, Julie Steele, Rob Faludi and me to Google I/O and possibly the most exhausting couple of weeks of my life.
But that was still months in the future when the original conspirators gathered in southern Rhode Island — myself, Brian Jepson, Shawn Wallace, Meghan Blanchette, Julie Steele, and Rob Faludi.
Despite the conversation with Tim, the whole thing was a skunkworks project, and with just a week to go before the conference in New York, selection of sensors was down to what off-the-shelf parts could be sourced and shipped in a few days.
With multiple boxes arriving throughout the first couple of days, we frantically worked to hand-solder 40 sensor boards, using the Arduino Wireless Proto Shield as a base for the work, which we dubbed “sensor motes” despite their size. We also worked Brian’s two 3D printers, especially his Printrbot Jr, from dawn to dusk to produce enclosures for the motes.
Loading the car at the end of the week, with a box of completed motes along with Brian’s Printrbot Jr so that we could continue to print enclosures for the motes at the conference, we drove from Rhode Island to New York for O’Reilly Strata.
Rolling the motes out in the hotel proved to be a difficult task. The hotel had thick, reinforced concrete walls, and it was a difficult space in which to deploy a mesh network. But we succeeded with the help of some extra hands (including Kipp Bradford) and got a lot of interesting data. Enough that the skunkworks project was brushed off, and we were invited back to the next Strata conference at the beginning of this year, in Santa Clara, Calif.
Ahead of the California conference, I flew into New York and camped out at ITP to build out a new feature for the Lab. In the space of a couple of days, I put together 15 “Awesome Buttons,” giant red buttons to go outside of each session room which the conference attendees were encouraged to push as the exited if they thought the talk they’d just seen was, well, sort of awesome.
We also added more sensor motes, and rebuilt some of the others that had been damaged in transit, for a total of about 50 sensor motes this time around. But we were also thinking ahead; I’d started thinking that if we were going to take this seriously, I should probably prototype some custom shields. So ahead of Strata I designed and ordered some boards from OSH Park to be delivered to ITP. They were waiting for me in New York, but only got soldered up in California on site at Strata.
This time around we even had real-time visualisation of our data, thanks to Kim Rees, and the team at Periscopic,
But then things got interesting. We’d talked briefly in New York, but after the Santa Clara conference, Michael Manoochehri, a Developer Programs Engineer with Google‘s Cloud Platform, asked us to bring the Data Sensing Lab to Google I/O.
It was immediately clear to all of us that, if we were going to do this crazy thing, the processes and hardware we’d been using up till now were going to have to change. The new custom shields I’d prototyped were a good start, but Google I/O is held in Moscone West. Compared to the sorts of spaces we’d been networking up to then, it is vast.
We’d need more motes, so many in fact that we were going to have to get them built and machine fabricated. After some back of the envelope calculations, I figured we needed somewhere between four and 500 sensor motes; we weren’t going to be able to hand-solder that many.
I’m a physicist, and only play at being an engineer on TV. We needed a real electrical engineer, and we needed one quickly, and that’s where Kipp Bradford came in. This time around we wanted more and better chosen sensors. Kipp took my initial through-hole design for a new sensor board and re-designed and prototyped something that could be mass produced, and actual product in other words.
Time, however, was not on our side. In fact, with only a month or so to Google I/O, everything about the project was a rush. Despite the general move to offshore manufacturing, often in China, our short time-frame meant that everything was manufactured on-shore in the United States. In this case, a lot of factors were in play: as well as being faster, it was also more cost-effective to manufacture in the States.
In the week before I/O, we ended up in a conference room at Google Boston. There were 500, not 50, motes — and despite coming from the factory with all the components soldered to the board, all 500 still had to be unpacked and attached to 500 Arduino boards, all of which had to be programmed. Then the XBee radios had to be associated with their ConnectPort gateways so the data could get out of our XBee mesh network and up into Digi‘s Device Cloud, and from there into the Google Cloud. It was all complicated by the fact that, due to budget limitations, we had four separate types of sensor motes.
This wasn’t a trivial exercise. It was a Kickstarter gone wrong.
In fact it was an intense and grueling week, despite extensive logistical planning ahead of time at Boston, and after working a week of 24 days, we arrived in San Francisco along with our hardware (which shipped next day air from Boston) a couple of days ahead of I/O with the rollout still ahead of us…
…but somehow we managed it, or almost. We managed to get 420 of the 500 sensor motes rolled out across Moscone West, and then we ran out of electrical sockets. But it was done, and the data started pour into the database. We collected over 150 million records over the course of the conference.
Kim Cameron and Amy Unruh, who worked along with Felipe Hoffa and Michael Manoochehri, on the integration with Google’s Cloud platform gave a talk at I/O discussing the software side of project.
Despite my worries, teardown on the Friday went pretty well, and Kipp, Julie, Rob and myself headed south for Maker Faire, which was kicking off the next day. Some of the sensor motes we deployed at Google I/O are on display at the Digi International booth at the Faire, and Kipp and I will be talking about our story later today at 11:45am on the “Meet the Makers” stage at Maker Faire. If you’re thinking about attending O’Reilly OSCON in July, Kipp and I will also be giving a tutorial where we’re talking about the Data Sensing Lab, our book Distributed Network Data, and what we did at I/O.
All the hardware and code from the project, including the backend database and pipeline work done by the team at Google, will be open sourced over the next couple of months. Finally (and pretty much because it was the most asked question when we were at I/O), the hardware we used is now available for pre-order.
ADVERTISEMENT
