Berlin-based artist Simonne Jones — a human being who embodies the fusion of art, science, and technology — asked me to engineer an interactive lighting system for her installation. Secrets of the Universe (SOTU) was a mixed media art installation and musical performance conceived by Simonne during her Artist’s residency at Platoon Kunsthalle in Berlin, Germany.
Simonne’s vision was to create a fully immersive experience that would encourage viewers to explore their curiosity about existence within a scientific context. She devised a novel combination of mixed media paintings that explored concepts in physics and cosmology (by Simonne Jones), a motion detecting, wirelessly controllable LED lighting system (by ArbitraryY… AKA me) and custom visuals+VJ performance (by Jem the Misfit) that resulted in a truly unique show.
Previously, I’d spent 15 years as a Software and Systems Engineer at Boeing, developing software and IT solutions and, while no one was looking, a fascination for the systems engineering process. Concepts of Operations, Requirements, Simulation, Testing, Design, oh my! Systems thinking makes me salivate as though I had pockets overflowing with Chick-O-Sticks.
SOTU was the type of project I never had the chance to be a part of in my career building big stuff. This project would satisfy my desire to engineer a system from start to finish. It was as magical as I thought it would be.
What did SOTU do?
Simply put, SOTU was a series of interactive, controllable, motion-detecting LED lit paintings. When triggered, the SOTU system would create a lighting experience for the observer that complimented the painting’s subject matter. How did they function?
First, during the exhibition people were allowed to traverse the stage and interact with the artwork. When they entered the “Activation Zone” (AZ, a configurable area in front of painting), its LEDs would illuminate and run through a programmed lighting scheme until they exited this zone. If an observer stepped within the (configurable) “Warning Zone” (WZ), the lights would erratically flash red (a cool Python threading application) until they exited this area. This is artwork that can tell observers to “Step away from the painting.” (The Mona Lisa needs a security system like this.)
During her performance, Simonne played a set of her music and used the paintings seamlessly as a performance tool. She activated the lights by dancing in front of them and triggering them with her arsenal of musical instruments.
SOTU System Overview
In this article I will outline the hardware involved and their associated functions. I will also touch on software subsystems that were essential to controlling the paintings with musical instruments. The painstaking system description (remember, I was a Software Engineer at Boeing for more than a minute, so haters of documentation be forewarned) can be found in the SOTU System Description Document.
A diagram and description of each component is given below. At the heart of SOTU lies a Raspberry Pi + Arduino mashup. Raspberry Pi’s computing power and Raspbian Linux operating system coupled with Arduino’s real time processing capabilities create a small, low cost, mighty computing duo that was ideal for this application.
Hardware and System Functions
Raspberry Pi
The Raspberry Pi pulled most of the processing weight:
- Communications Server – Open Sound Control (OSC) Server, processed all incoming OSC messages.
- Webserver – Apache httpd. For web based LED controls
- Remote Access – SSH and VNC. SSH for remote software development on the Raspberry Pi and VNC for development in the Arduino IDE.
- Wireless Connectivity – Connectivity to the SOTU wireless network
- LED Control – Software Pulse Width Modulation (PWM) for LED dimming
Note that each Raspberry Pi unit was connected to a private wireless network called “[PLTN]”.
LED Driver Board
I made two independent circuits of transistors and resistors arranged onto an Adafruit Pi-Plate to power the LEDs. This board was then attached to the Raspberry Pi I/O panel and was the connection point for the LED strips. Each painting was lined with two independently controllable strips.
Arduino and Protoshield
The Arduino and protoshield were in charge of the following:
- Range sensing and distance measurements – Measured the distance of an observer
- 12V and 5V power distribution – Powered the Arduino, Raspberry Pi, LEDs, LCD screen, and range sensor
- LCD Screen – Displayed the painting name
Range Sensor and LCD Screen
We cut holes into each canvas in order to mount the range sensor (center of painting) and LCD screen (bottom right). The range sensor was used to determine an observer’s distance from the painting, which was ultimately used to activate the LEDs. The LCD screen displayed the painting’s name.
Production
It took nearly two months of procurement, assembly and testing to get all six units ready. I had to purchase and receive all of the parts, solder like mad, install operating systems and SOTU software, functional test em, and finally label box.
Software
SOTU is a complex software system made up of open source modules, third-party software, custom Arduino programs, range sensor data processing algorithms, custom programmatic LED lighting controls and effects libraries, a communications layer, user interfaces, system health monitoring, maintenance scripts, OSS/COTS tools, test tools, and web controls. Breathe. I programmed all of the custom code in Python and PHP.
Painting Control Using Musical Instruments
The critical system function was to provide a mechanism for Simonne to illuminate the paintings with her musical instruments, namely, her MIDI keyboard and kick drum. Press a key or kick the drum and lights activate. I built multiple layers of abstraction to make it easier for me to program custom lighting functions and activate the lights from remote sources.
RGB LED Dimming (PWM)
Manipulating the RGB LEDs connected to the Raspberry Pi required Pulse Width Modulation (PWM). The model B only has one GPIO pin capable of hardware PWM. This limitation required me to use software PWM (hardware PWM emulation). When I built SOTU, the best libraries available were RPi.GPIO v0.5.2a (Python package, very outdated version now) and pi-blaster. Raspbian Wheezy’s kernel is not intended for real-time applications; jitter is likely when using software PWM. pi-blaster was much less jittery than this version of RPi.GPIO.
pi-blaster enables you to regulate the power output to a given GPIO pin (i.e. control the brightness) by simply writing a decimal value to a file on the OS filesystem like so:
echo "2=0.2" > /dev/pi-blaster
This would adjust the attached LED to 20% brightness.
Programmatic LED Controls with Python
Next I used pi-blaster to build an LED controls abstraction layer in Python. I built a series of classes that allowed me to programmatically:
- Set any (RGB) color on any LED strip on a painting
- Activate effects on any LED strip on a painting:
- fade – Fade between colors
- rotate – Rotate through an array of colors
- pulse – Switch back and forth between colors
- flashFade – Set a color then fade it to off
Using these functions I created custom lighting playback schemes that accompanied each painting by simulating the physical principles associated with the painting’s topic. Check out a video demonstration of the Big Bang: Birth of the Universe playback script running on my SOTU prototype.
Raspberry Pi Control with OSC
The next layer in abstraction in SOTU was to enable remote control. Enter Open Sound Control. OSC is a communications protocol that enables musical instruments, Digital Audio Workstations (DAWs), computers and other multimedia tools to communicate with one another on a network. The OSC protocol uses a simple client server mechanism; an “OSC client” sends “OSC messages” to an “OSC server” that receives and processes them. Typical processing involves triggering a system function based on the specific message received (known as a “Callback“). Once again the OSS community had my back…there is a great library called pyOSC available for all to use.
Using pyOSC’s callback mechanism I created OSC addresses that were mapped to the Python LED control functions (effects, set color, etc.). I could now perform all of the LED functions on a painting by simply sending OSC messages from ANY OSC client (See the SDD for details on other clients) like this:
/osc/led r1 1 solid /osc/led b2 0.3 solid
This would turn LED strip 1 to 100% red brightness and strip 2 to 30% blue brightness.
pyOSC’s callback mechanism had taffy-like flexibility. I was able to use it for many other things like manipulating and monitoring critical Linux services (httpd, ssh, rangeSensor, etc.), monitoring the systems health (service status, etc. using Processing) and shutting them down remotely (when the correct authorization key is provided of course).
Performance Integration
Simonne uses Ableton Live as her DAW along with an arsenal of MIDI controllers and instruments during her live performance. Ableton Live uses MIDI and OSC for its internal communications. For example, when a key is pressed on a connected external controller Ableton receives the MIDI event message and uses it to trigger an event configured in Ableton (e.g. a music sample or to play a note from a synthesized instrument). Ok, so we now have access to messages related to her keyboard presses. What about the kick drum?
We had to turn turned her kick drum into a MIDI enabled device. We attached a pressure sensor to the drum and connected it to another MIDI control device that was connected to her laptop. With each drum kick we then received MIDI on/off messages.
Ableton natively exposes its internal MIDI and OSC communications. I now had access to all MIDI event messages coming from the both her keyboard and the drum set. Ableton Live is the BizNiz.
Last step was to take these messages out of Ableton and route them to the different paintings to light them up.
Message Hungry Third Party Tools
The last integration required me to take these messages out of Ableton, translate them to SOTU system OSC messages, and route them to the paintings. No need to reinvent the wheel here. OSCulator is an amazing, reasonably priced MIDI/OSC routing tool that also boasts native integration with Ableton Live. It was now trivial to take MIDI event messages from her instruments and route them as OSC messages to send them along to the paintings.
Below is a sample of the OSCulator configuration for the SOTU show:
Here, OSCulator listens on port 9001 for incoming messages. If the key corresponding to MIDI message 99 is pressed on her keyboard, a value of 1 (on) is received. OSCulator is configured to send these OSC messages to painting 3 (“PLTN3”) to turn it red.
/osc/led r1 1 solid /osc/led r2 1 solid
When that key was released, OSCulator received a value of 0 and routed the following OSC message to turn the painting off (creating a flash effect):
/osc/led allOff
The lights on each painting could be customized by color and effect and controllable using her musical instruments.
So How’d It Go?
The paintings were displayed in a semi-circle on a stage made of milk crates that resembled a Q-bert playing field, which served as the layout for the exhibition and the stage for Simonne’s live musical performance. The show was high energy and the crowd loved what they saw:
Throughout the summer of 2013, Secrets of the Universe was also exhibited at the Berlin Remake Festival, at the Berlin Arts and Music Festival, and used during her performance in Bremen, Germany.
Conclusion
SOTU performed incredibly well in each of the exhibitions and performances. While the Raspberry Pi is typically touted as a hobby computer, I can attest to the fact that it is also excellent for real(ish)-time interactive art and performance applications. I attribute a large part of our success with this project the robustness of the Raspberry Pi.
Building a system with this level of complexity is a problem filled with interface, logistic, development, operational, testing, and deployment challenges. I endured all of these challenges while building SOTU with the result of me honing existing skills and acquiring many new ones. Well played Raspberry Pi Foundation, mission accomplished.
SOTU is the type of problem that unites creators from various disciplines and one that excites us as all engineers. I leveraged the collective knowledge of you all often during this project, my most valuable resource.
Con Amor de ArbitraryY
Extras
Simonne describes her philosophy and artistic vision for Secrets of the Universe in her interview for 3SAT TV.
ADVERTISEMENT
