How-to: Hacking RjDj with PD

RjDj is a sound application for the Apple iPhone* that allows users to access “scenes” that transform sound that comes in through the microphone and allows additional tweakage via the the iPhone’s accelerometer and touch screen. But did you know that RjDj’s scenes are made with PD (PureData), a free, open-source multimedia programming environment that uses a visual patch cable-style interface to create interactive applications for audio and multimedia? Yup. This makes RjDj scenes incredibly hackable, and it also makes it very easy to port RjDj scenes that you’ve made to other platforms. Because RjDj is built with PD under the hood, it isn’t difficult to modify the scenes that come bundled with RjDj, and with a little PD know-how, you can also create scenes from scratch (see my video below). Read on to see how.

RjDj and PD
RjDj is essentially a PD patch player that has been ported to the iPhone, minus the PD GUI (although you do get a touchscreen and accelerometer data that you can use to change parameters in your PD patch). The lack of GUI is not necessarily a bad thing though, as the minimalism of this application is precisely what has made it such a hit: All you’re really supposed to do is plug in the headphones, turn on the app and listen to how it transforms normally mundane sounds into ear-melting science fiction freak-outs. Although the main focus of the RjDj developers seems to be scenes that morph the sound that comes in through the microphone in real-time, the beauty of developing scenes for this app is that you don’t have to use it just for sound environment manipulation, you could also just use it to make generative music, brutal FM synthesis noises, or whatever else you might be able to coax out of a PD patch.

The Beauty of PD – RjDj and Beyond
Developing scenes for RjDj has applications beyond just the iPhone, because PD is so wonderfully open and portable (and by “portable” I mean able to ported to other platforms, not just “portable” like a Walkman, although that is certainly the case now as well). If you’re anti-iPhone (or just can’t afford one) but you like the idea of what RjDj does, you can run these scenes on virtually any computer simply by downloading the scenes from the RjDj developer repository, opening them in PD, and tweaking a few connections so that they’ll play on you computer (although without accelerometer and touchscreen support). RjDj scenes, without the iPhone contract! PD being the free and open environment that it is, I’m sure we can expect a port of PD for other mobile platforms (Android…?) soon as well. Heck, they even got PD running on old iPods back in the day, so anything is possible, right?

Developing for RjDj
The RjDj team has made it a point to encourage developers, musicians, anyone to write their own scenes by including a big button on their site that says “Create Your Scenes,” and has provided quite a bit of documentation at the RjDj developer’s wiki site. Having successfully made my own functional RjDj scene from scratch in just a few hours a few weekends ago, I’ll guide you through the process of downloading, editing, and uploading scenes to the iPhone.

Getting RjDj
First, the obvious: Grab an iPhone and download RjDj from the App Store. For the purposes of this how-to, the free “single” version will be fine, although there is certainly nothing wrong with supporting the hard work of these developers by downloading the paid “album” version.

Getting PD:
PD is a free and open source program that can be easily downloaded and installed on Mac, PC or Linux computer. For working with RjDj, the developers suggest that it’s best to use PD-Vanilla (linked above) as opposed to the slightly fancier PD-Extended build. This is because RjDj is built to read the objects found in pd-vanilla, and there are bells and whistles that are only found in pd-extended that you might accidentally include that would then not be understood when you open the patch in RjDj. So, just to be safe, forgo the anti-aliased GUI, nicer dock icon and Max/MSP-ported externals of the pd-extended build, and keep it simple with pd-vanilla. Or if you know what is vanilla and what isn’t (nerd!), you could probably get away with just testing your patch in pd-vanilla first.**

Setting PD up For RjDj
Once you’ve got PD downloaded and running, there are a few things you can do to make it work best for programming for RjDj. They suggest you set your audio settings to 22.05 kHz instead of the standard 44.1 kHz. I didn’t actually do this on my computer when I made my patch, and everything was fine, but if you’re working on something processor-intensive, this lower sampling setting could actually make a difference in minimizing processor load.*** If you are doing any work with samples that load in your patch, however, you will need to test your patch at 22.05 kHz and make sure that all of the samples you are using are also running at 22.05 kHz (See also: Damian’s comment in the comments section about why it’s important to test the patch at 22.05 kHz if you’re loading samples, and how it’s important to run at 22.05 kHz on your computer for evaluating the overall sound quality of the patch as it will play on the phone). Advanced PD’ers may also want to install RjLib to load up in PD at launch so that all of the necessary abstractions load on startup.

Abstractions – RjLib
Abstractions are handy sub-patches that can be used over and over again in PD patches to perform certain functions. All of the necessary abstractions for accessing sound input and output, accelerometer data, and touchscreen data are contained in a library called RjLib that can be found at the RjDj developer’s wiki here. Point your favorite FTP client to this link and you can download the whole RjLib as one folder, where you’ll find the abstractions that you’ll need to access the touchscreen, accelerometer, and sound in/out data so that you can write and hack your own scenes to use these functions.

RjDj Scenes – What They Are and How to Catch ’em All.
Scenes are the different “flavors” of RjDj that are available. As songs are to iTunes, scenes are to RjDj. If you downloaded the free version of RjDj, it came with one scene, Echelon. If you purchased the paid version of the application from the App Store, then you will have six scenes. A little-advertised feature of RjDj is that there are over thirty other scenes (ranging from developers’ utilities to sound games) that are publicly available at the scene repository on the RjDj developer’s wiki. Once you have them, if you have access to your iPhone’s file system via SSH, you can upload and run as many RjDj scenes as you can find or make yourself.

The Anatomy of an RjDj Scene
In the repository you will find folders that contain the RjDj scene contents, the folder (a folder that ends in “.rj” as in “yourscenenamehere.rj”) basically contains a PD patch and its associated files. Here’s what you’ll see in each folder:

• a .plist file that makes up the information that comes up when you hit “info” in the RjDj scene. This file will always be named “Info.plist” This is also where the name of the scene is set.
• a 320×320 pixel jpg that will make up the picture that appears on the screen when your scene is loaded.
• The main PD patch. This patch must be named _main.pd. You can also include abstractions (other related .pd files that you can use within the patch, a PD file within a PD file) that can be associated with your main patch.
• This folder could also include sound files that are played as samples in your patch.

Getting Additional Scenes
One of the main complaints people have about this app is that they like the scenes, but that there aren’t enough of them. So let’s go get some. Point your favorite browser or FTP program over to the scene repository and grab a few scenes with their .rj folders still in tact. Let’s start with the one that comes with the free version, the one most people are familiar with, Echelon.rj.

Looking Inside a Scene
Here’s what’s inside the “Echelon” scene:

This one is simple enough, there are no additional abstractions and there are no linked sound files, just the absolute bare minimum to make an RjDj scene. Now let’s launch PD and open Echelon’s _main.pd file. You’ve heard the scene, now see what it looks like as a PD patch:

This is the guts of the patch, and as you can see, there’s really not a lot to it, it’s quite simple. One of the things that’s nice about PD is that you can see the connections that go from one object to the other, and they help us to visualize the flow of the patch and how it does what it does.

How Echelon Does What it Does
What’s at the heart of the craziness in this patch is the “random 4” object that is picking a random number from 1-4 (well, actually 0-3, but it becomes 1-4 after the “+ 1” object is added directly after it), and that 1-4 number is multiplied by 333 (with the “* ” object with “333” in its argument) and then has another 1000 added to it (with the “+” object that has “1000” as its argument), which makes a ramp that changes the speed of the playback of the sample that is taken from the “delwrite~” object that records the delayed audio.

Huh?
Does that make sense to you? Don’t worry, it doesn’t have to make complete sense! Let’s modify this scene by taking a kind of experimental, circuitbending-style approach to hacking this patch. Like circuitbending, you don’t have to know exactly what it does to make it do something different. Part of the beaty of PD is that you can change things around in realtime and immediately hear the changes that you’ve made, so it’s a great way to learn while experimenting.

Hearing an RjDj PD Patch on a Regular Ol’ Computer:
First off, in order to tweak the patch, let’s set the patch up so that it’s audible on your computer. RjDj uses just a few special objects (well, technically abstractions) that aren’t found in regular PD to access the iPhone’s sound inputs and outputs. These abstractions are called “soundinput” and “soundoutput” and they show up as boxes with dotted lines when you load them up with PD Vanilla if you don’t have RjLib installed in your build of PD. It’s ok, because you can’t hear them anyway (they’re proprietary to RjDj and don’t input or output sound outside of the program) . In a regular PD patch, sound inputs and outputs are accessed with the adc~ (analog-digital converter, i.e., input) and dac~ (digital-analog converter, i.e., output) objects, so if you want the patch to play on your computer so that you can hear what it sounds like as you mess around with it, duplicate the connections to “soundinput” and “soundoutput” to adc~ and dac~ like so:

All of the connections that were going to the left and right outputs of “soundinput” should now also go to the respective left and right outputs of adc~. All of the connections that go to the left and right inputs of “soundoutput” should go to their respective L and R connections in dac~. Now you are set up to hear the patch on your computer, and it will also accept audio input if your computer has a microphone. You might want to use headphones, since a lot of the RjDj patches pump audio in and out at the same time, and this can cause some pretty awesome feedback.

Hacking an RjDj Patch:
Now let’s look at what is hackable about this patch. Take for example the “metro 250” object. The “metro” object is the metronome of this patch, and it’s connected to the random object. The number 250 that appears in the “metro” object is what’s called the argument, and it tells the “metro” object to send a bang (a “do your thing now” message in PD) to the “random” object once every 250 milliseconds, or four times a second. Say you wanted to speed up the rate at which the pitch changes happen in this patch, then you could change the argument to 100, and it would only generate that random number ten times per second.

The patch with the “metro” argument changed to 100:

Adding Accelerometer Control to PD Variables
Here’s where we can make an interesting modification. This patch doesn’t use any accelerometer data, so let’s make a hack to control the amount of time between pitch changes by using x axis of the iPhone. To do that, you would just need to write in a little bit of a patch to call on the accelerometer data and send it to the second inlet of the metro object (since accelerometer data goes from zero to one, I’ve also added “* 1000” to the accelerometer data output to pump the number up so that it goes up somewhere between zero and one second between bangs of the metro). I’m getting a bit ahead of myself in explaining how to access accelerometer data (the next section explains this in detail), but for now I’ll just tell you that you’ll need to paste in the “accelerate” abstraction from RjLib. From the top of the patch, it’d go a little something like this:

Now the the speed of RjDj’s pitch shifting can be manipulated by the iPhone’s position on the x-axis. This is a simple example of how to make a modification that uses RjDj-specific abstractions (the accelerometer and touchscreen abstractions found in RjLib) to control variables in the PD patch.

Other areas that would yield interesting modifications to this patch could include:

• Changing the values in the “delread~” objects. Keep the “test” argument in tact, but you should be able to step the delay of these all the way up to 6000 ms, since that is how long the delay is written with the “delwrite” object.
• Changing the argument in the “random” object to a higher number would result in a more profound fluctuation in the pitch shift.

RjLib – Accessing the accelerometer and touchscreen data in your patch:
Although you might be used to including faders, sliders, buttons and other GUI-related components into your PD patches, you’re only allowed a static .jpg in RjDj. Since there is no interactive GUI in RjDj, using the accelerometer and touch screen are a few ways of allowing the user to adjust certain parameters within your patch. To use these functionalities, first off, make sure that the accelerate.pd and touch.pd abstractions are placed in your patch. These abstractions can be found in RjLib. Once they are there, you can access any of their parameters. Let’s consider “accelerate”:

“Accelerate” gives you the option of accessing magnitude, pitch, roll, and the individual x, y, and z axes.

Here’s an example of a cheap, super-simple two-channel accelerometer “Theremin” that I made to test the responsiveness of the accelerometer, where x controls the pitch of left speaker and y controls the pitch of the right speaker. [Download RjDj file]

Here’s what it looks like:

As you can see, once you have the “accelerate” abstraction placed in the patch, you can receive info from accelerate anywhere by making an “r accelerate” box (“r” is short for “receive”) and sending its data out to “route” objects for x, y and z (z isn’t used here though).

This patch is incredibly lame, meant mostly to be used for reference, but here’s what it sounds like:

A note about this patch: The “* 1000” in this patch could also be making the accelerometer jumps more apparent, and you could add a “line” object either before or after the multiplied value of the accelerometer data to smooth the shift in pitches for either channel.

These same tricks can be applied to the “touch” set of actions, allowing you to control parameters of the patch via the touchscreen.

Specifically, What Can You Use to Control Variables in the PD Patches?
With the accelerometer, you have access to the following variables (taken from the RjDj wiki):

• pitch: pitch angle of the iphone in radian (-pi/2 to pi/2). 0=iphone lying on a table
• roll: roll angle of the iphone in radian.
• magnitude: amount of the overall acceleration
• x: acceleration on the x axis
• y: acceleration on the y axis
• z: acceleration on the z axis

With the touchscreen, you can use the following variables to control parameters in your patch:

• tap: this message is sent whenever a finger touches the pad
• speed: gives you the speed of which each finger is drawing on the touchpad
• updown: gives 1, when finger is on touchpad, 0 when it leaves
• direction: gives the direction angle of which the finger is moving (radian: 0 -> pi)

As you can see, despite not having access to regular PD GUI functions, there are still a lot of different ways that you can use the accelerometer and touchscreen to manipulate variables within a PD patch.

Uploading RjDj Scenes to the iPhone:
Now that you’ve made some monstrously wonderful PD creation that runs on your computer, it’s time to get it into RjDj so it can run there. Here’s where things get a tad tricky: Because RjDj does not currently support the ability to be able to change the contents of the folder that houses your scenes, you will have to enable access to your iPhone’s file system via SSH. This means that your device has to be “jailbroken” in order to be able to add, edit, test, and play the RjDj scenes other than the standard set that comes pre-loaded from the App Store. The internet is full of resources to help you enable SSH access to your iPhone’s file system, so we won’t cover that in detail here, but the RjDj deveopers perhaps said it best on their website: “If you know how the Iphone works, you will also find a way to get your scenes on the phone directly.” (I suspect that they’ll eventually update RjDj so that you can have web-based access to specific folders in the application, similar to the way that other existing iPhone apps already accomplish this without violating the iPhone SDK. But for now, you gotta get your hands a little bit dirty and work some basic SSH skills.)

Once you get SSH running you should be able to use an SFTP program like Cyberduck for the Macintosh or WinSCP for Windows to get into your iPhone’s root directory. (Hint: Unless you’ve changed it already, the default login is “root” and the default password is “alpine”). Once you’re into your iPhone’s file system, you need to get to where the RjDj scenes are kept. Be somewhat careful, don’t mess around with files in areas you aren’t familiar with, as this could potentially screw things up on your iPhone. That being said, there’s very little damage that can be done if you limit your experimentation to the rjdj_scenes folder. The worst that could happen is that the patch won’t run, and the application could freeze up – nothing a good old-fashioned restart won’t fix. The wiki says that the files should be here:

/var/mobile/Applications/*/RjDj.app/rjdj_scenes/

In this path, /*/ is going to be one of a bunch of cryptically numbered folders, one of which actually houses your RjDj application. All of the applications in the applications folder have cryptic, garbled numbers as their names instead of actual names, but they tend to go in order of when the application was installed, so you can set your SSH client to view the items by the time they were modified and just poke around until you find the one that contains the RjDj.app.

The cryptic “Applications” folder inside my iPhone:

Once you find the right folder, the contents of the RjDj application folder will look like this:

Inside the RjDj.app folder, (not to be confused with the RjDj folder in the Applications folder shown above) you will then see this:

In the “rjdj_scenes” folder you will find either one to six folders that end in “.rj”, depending on if you have the free or paid version of the application. If you like, go ahead and make a bookmark for this location so you don’t have to fumble around with the garbled number folders next time. If you are going to be messing with RjDj scenes, you will use this location a lot.

To add scenes to RjDj, all you have to do is drag your complete .rj folders into the “rjdj_scenes” folder and then fire up RjDj on your iPhone (it isn’t recommended to have the RjDj app running while you do this). If you are loading a scene that you have modified, you might notice that you have two of the same scene names when you load up a modified scene, even if you have changed the folder name. That’s because it reads the Info.plist file to find the name of the scene. If you want to edit the Info.plist file, you can do it as a straight text file or you can download any of a number of free, simple .plist editors for your computer platform of choice.

A sufficiently loaded rjdj_scenes folder, files downloaded from the repository, as viewed in SSH via Cyberduck:

So once you have your scenes placed in the “rjdj_scenes” folder, launch the application on your iPhone and go to the Scenes menu. You should then see the scenes that you have uploaded.

Behold, a fully-loaded RjDj scene list:

Hooray! Now you have a ton of (er, at least more than six) scenes that you can play on RjDj. Be the envy of your friends.

Making Your Own Scene From Scratch:
Of course, I can’t give a complete overview of PD in this article, or even a very sufficient introduction, but if you know PD already (or even its commercial sister app, Max/MSP), you are totally in luck, and if you don’t know anything about PD, you are also kind of in luck, because it’s not incredibly hard. Rock guitarists might benefit from thinking of it like a bunch of effects pedals and cables, where the objects (the boxes) are the stomp boxes and the lines that connect them are patch cables. But the stomp boxes (the objects) do one specific task, and they do it well. It’s of course not quite as simple as just plugging in a bunch of Boss Grunge Pedals, but I think it’s a pretty safe analogy. With this knowledge in mind, you are now ready to begin making a beautiful mess of boxes and patch chords that will become your own PD patch, which can then easily become your own RjDj scene. A good place to start is to just go to the Help browser in PD and look at parts 2 and 3, the Control Examples and Audio Examples:

The control examples give you all the info you’ll need to control MIDI and audio from within your patch. Like something you see there? Just copy and paste it into your own creation.

The audio examples give you recipes for just about every trick in the book for audio manipulation:

The HTML version of the PD documentation is also a great resource. Once you get a grasp of the basic concepts of these examples, you can start to mash up components from them to do your own thing. I would recommend that you play around with making patches on the computer and then load them into RjDj, but as I said before, remember to switch the adc~ and dac~ objects to soundinput and soundoutput, (and vice versa when you bring RjDj patches back onto the computer).

My RjDj Scene:
Here’s how I figured all of this stuff out anyway: I made an RjDj scene that samples a note that you sing into the microphone and then loops that note and adjusts the playback speed to change it to different notes, the notes being triggered by a simple generative music-making brain (just randomly selecting notes from a pool of predetermined likely-to-be-melodic notes). This patch also uses the “touch” abstraction to turn a touch of the touchscreen into a “bang” in the Pd patch to initiate sampling of the audio input. The scene is called Noble Choir, and it’s available from the RjDj repository. I sent it to the RjDj people, and they were kind enough to add it to the repository and give me a small shout-out on their web page the very next day. Download the patch from the repository, upload it and try it out!

Here’s my little demo of it. Wow, my desk is messy:

So, this is by no means a comprehensive how-to on hacking/creating RjDj scenes, but this should at least give you enough info to be able to download and upload RjDj scenes on the iPhone and make simple modifications to them in PD. This should hopefully also serve as an overview for seasoned PD’ers on how to set up pre-existing patches to run on RjDj.

Questions? Corrections? Comments? Please leave them in the comments below.

*Support for the 2G iPod Touch is reported to be on its way soon as well.

** There are a few exceptions to the rule that you can only use abstractions in pd-vanilla, and they are pretty awesome exceptions:

• You can use the fiddle~ audio to MIDI external. This lets you make patches that convert real-live singing, etc. into MIDI notes that you can then use for whatever evil MIDI purposes you desire.
• You can also use bonk~, an external that detects attacks in notes, good for setting up beat detection in raw audio.
• The few others mentioned here.

*** Speaking of processor load, they provide a patch in the repository called zzPerftest.rj that is designed to show you just how much RjDj can handle (it’s just a group of 256 oscillators all playing 220 Hz at the same time, same phase, same volume). Load the patch, check out how much load it has your computer, and then you know you can build about that much processing load into your own patch.

**** There’s technically no worry about the RjDj-specific abstractions not loading in PD on your computer, so don’t let all those dotted lines bother you too much, they will load again once the patch is put back into RjDj. Or alternately, you can put their abstractions in the same folder as your _main.pd file, and then they’ll get loaded up when the main patch is loaded. Or, if you’re really pro, you can have PD load up the folder of abstractions on startup. The makefile’s inside the RjLib folder.

One More Thing: Getting RjDj Recordings Off of the iPhone
A feature that a lot of people have requested but that has not been implemented yet in the application is the ability to get the recorded scenes off of your phone and onto your computer. While you’re in the RjDj app folder in SSH, you can find where these files are housed, under

/var/mobile/Applications/*/Documents/recordings

Find them in SSH, drag them on to your ‘puter, and enjoy.

See?!?