Embedding Pure Data into a Unity iOS App

UPDATE: I wrote a sample project here.  It uses the most up-to-date version of libPd as of 03/08/2013.

So you're making a Unity-based iOS game, and you want to use Pure Data as your audio engine? Great! Let me show you how it's done.  Before starting, I suggest you read and understand this post on embedding libPd into an iOS app.

 

Step one: Create a wrapper for PdBase

The first thing you'll have to do is create a native wrapper for PdBase.  As mentioned in my previous post, PdBase is the main interface between your application and libPd.  So essentially what you're doing here, is creating a way to send messages to PdBase, which in turn sends messages to your Pure Data patches.  You can find my example wrapper here, however be warned that this is just a proof-of-concept, it is not fully tested, nor production ready code.  I'll explain how these wrapper functions work with an example:

This function is used to send a symbol to a receiver in your Pd patch.  That means it needs to know two things, the symbol you're sending, and, the name of the receiver.  However, you'll notice the function takes four arguments.  That's because both symbols and receivers are strings of characters, and the most reliable way for our native c code to know when a string sent from Unity ends, is to just tell it the length. In other words, symLength is the length of the symbol string in bytes, and recLength is the length of the receiver string in bytes.  The reason we do it this way is because a c string is simply a pointer to the memory location of the first character of the string, whereas a Unity string is an object that contains the string and more.

The first thing you do with each of these strings is convert them into NSStrings, because that's what PdBase expects.  To do so, initialize a new NSString object with the data sent from Unity, and make sure to specify the string is encoded as NSUTF16LittleEndianStringEncoding because that's the encoding that Unity uses.

Next, you call the appropriate PdBase function with our newly created strings, and release those strings because you're done with them.

Step two: Create an interface in Unity

Now that you've wrapped all the PdBase functions you're going to use, it's time to interface with your wrapper from Unity.  My example interface can be seen here.  There isn't much to explain, this code can be used more or less as-is, but I have a few notes to mention.
First, lists of arguments are sent as one long string separated by colons.  For example, "arg1:arg2:arg3".
Second, I append %f to float arguments which are part of a list.  This is so our native interface can differentiate between float and string arguments.  I did this while debugging a particular issue, but I'm fairly sure it's unnecessary.
Finally, you'll notice when passing that when I pass the length of a string to the native code, I multiply it by two.  This is because our C code is interested in the number of bytes a string is, not the number of characters, and since Unity strings are UTF-16 encoded, every character is made up of two bytes.

Step three: Embed Pd into an iOS App

When you build a Unity project for iOS, Unity creates a new XCode project with your game embedded in it.  At this point you'll embed Pd into your project, the process of which is described in this post.  Secondly, you'll want to add your PdBase wrapper, the first file we created, to this project.

Step four: Dance Party!

At this point you should have a Unity project and libPd embedded into a single application running on your iOS device.  Try calling the functions from your Unity-to-native-code-interface and see if your patches react.  If they do, great!  Otherwise, leave a note in the comments so we can deduce what went wrong.

My DIY Controller at Montreal Maker Faire

I spent the weekend demoing my Play-doh controller this weekend at Montreal's first-ever Maker Faire!  Check out the photos:

Embedding Pure Data into an iOS App

So you want to embed Pure Data into an iOS App?  This process is both very simple, and frustratingly complicated, depending on your Pure Data needs.  However you’re in luck, because I’ve run into (almost) every possible issue, and I’m counting on you, the reader, to bring any new issues to me so I can deal with those too.

Requirements

• XCode
• an iOS Device
• Git

Step one: Get libPd for iOS

The first step is grabbing libPd from github.  For more information on libPD, refer to this post.  This is most easily done by cloning the the pd-for-ios repository with git.  Afterwards, follow the installation instructions on the pd-for-ios github page.
At the time of writing, these instructions are:

‘After cloning the pd-for-ios repository, please make sure to cd into the
pd-for-ios folder and say
 git submodule init
 git submodule update
These two commands install the dependencies from libpd.  After the initial
setup, say
 git pull
 git submodule update
whenever you want to sync with the GitHub repositories.’

Step two: Compile the sample projects

The first thing you’re going to want to test is whether or not you can compile the sample project.  So, open up your newly acquired test project from the repository you just cloned, which is aptly named PdTest01, and try to compile and run.  If that worked, you have my permission to give yourself a nice pat on the back before moving on.
Open up the next sample project, PdTest02.  This project has a more complicated structure because libPD is a sub-project inside of PdTest02.  This means that you can update or modify libPD as much as you want, and it’ll recompile it every time you compile the PdTest02 master project.  This is the same structure that your final iOS application should adopt.  Make sure that you select the PdTest02 scheme (pictured below), then compile and run.

Step three: Add libPd as a sub-project to your project

The next thing you'll do is add libPd as a sub-project to your own project.  I assume you've already created your project.  Open up pd-for-ios > libpd in Finder.  Here you’ll see a libpd.xcodeproj file.  Drag this file into your project explorer.  It should look something like this:

Step four: Add libPd as a dependency to your project

Now select your project from the project explorer in XCode, select your app under Targets, and select the Build Phase tab.  

Here you're going to add libPd as a dependency for your project, as well as link libPd to your project.  Expand ‘Target Dependencies’ and click ‘+’ to add the libpd-ios dependency.  Next, expand ‘Link With Libraries’ and click ‘+’ to add libpd-ios.a to link the library with your app.  When you're done, it should look something like this:

 Now when you compile your project, it will compile libpd-ios first and embed the library into your compiled project.

Step five: Compile and party!

Hurray! You’ve successfully embeded Pure Data into your iOS app.  Unfortunately this is only half the battle.  There are two significant issues left to address.

How do I use libPd?

I recommend you explore the sample projects to get an idea of how to use libPd.  This will show you how to initialize Pure Data and open a PD patch.  Also, PdTest02 shows the user how to setup the AppDelegate as a PdReceiverDelegate.  This will allow you to receive messages from PD.  If you want to learn more about interacting with libPd, explore the libPd source, with PdBase being especially useful.

How do I add extras?

Vanilla Pure Data contains as few objects as possible to run.  This applies to libPd as well.  As a result, libPd is as lean as possible, which makes it perfect for embedding.  However most (all) Pure Data programmers use objects that aren’t part of Vanilla PD/libPd.  The good news is that libPD supports these objects.  The bad news is this can be a huge source of frustration.

PD Objects come in two formats: .PD files and .c files.

.Pd Files

FX is an audio programmer I work with.  FX uses Pd-Extended to create patches.  Pd-Extended contains a lot of extra objects that FX depends on, for example tof/sample_granule~ and tof/sample_play~.  In order to accommodate FX, I had to add these objects to libPd, and here’s how.

First, download and install the Pd-Extended application from the Pure Data website.  Right-click the app, and select “Show Package Contents”.  Under Contents > Resources you’ll find a folder called extra.  This is where all the fancy “Extended” objects are hiding.  

Now copy the “tof” folder (or whatever objects you’re looking for) into your project folder.  Pure Data expects a certain folder structure.  What this means is when you create a “tof/sample_play~” object in your PD patch, it expects “sample_play~” to be contained in the folder “tof”.  However, by default when you create an iOS app, all your resources get dumped into the resources directory, and don’t retain any folder structure.  To get around this, when you drag the “tof” folder into your project explorer in XCode, select the option “Create folder references for any added folders”.  

Congratulations! You can now use all the tof objects with your embedded libPd application. (Note: I haven’t been able to make sample_shifft~ work, if anybody has any information on this, please let me know).

.c Files

C extras are easy to add.  Simply add the desired c files into your project, this can be done by dragging and dropping them into your project explorer in XCode, preferably into a group with a descriptive name like libpd extras.  

Next you’re going to initialize the new object before you use it.  Let’s pretend you’re trying to add the object lrshift~.  Every pd object has a setup function called [name_of_object]_setup().  For example, lrshift~ has a setup function lrshift_tilde_setup().  You’re going to call this function when you initialize Pure Data, however you have to tell the compiler that it exists first.  Open your AppDelegate.m file, and somewhere before your function definitions add extern void lrshift_tilde_setup(void);.  Now, before you initialize pure data, call lrshift_tilde_setup();.  Do this for any other c objects that you’re using.

Note about extras

I’ve noticed that Pure Data sometimes expects these objects to be contained in a folder.  For example, the PD file may have a line that says “cyclone/lrshift~”.  The problem is that any extras added to libPd with this method essentially become one with the base Pd library, so Pd will fail to initialize them.  To fix this, I simply run a search and replace on all my PD files, and replace any offending lines.  For example “cyclone/lrshift~” becomes “lrshift~”.  I recognize that this may not be the most convenient method and I’m listening for suggestions on how to fix it.

Where do I find extras?

All the extra objects included in PD-Extended can be found in the PD-Extended app.  To find them, download and install Pd-Extended.  Right-click on the app, and select “Show Package Contents”.  Under Contents > Resources there’s a folder called “extra”.  This is where all the extended objects are hiding.

On libPd

What is libPd?

libPD is the same Pure Data you know and love, but without a fancy UI.

What does that mean?

That means that libPD can do everything that Pure Data does, other than build patches.  That also means that libPD can be embedded into any software, which supercharges your application with Pure Data power.  Some ideas are:

• games
• audio software
• browser-based applications (ie. Chrome apps, Light Table)

That's cool, but why bother?

The most obvious reason is that audio programmers can build audio patches in a way that’s comfortable to them.  That means less time is spent learning new tools.  
It also means that all Pure Data’s resources are available to you.
Most of all, it means that you have the power (and responsibility) to subvert massively overpriced audio software by building your own software with the power of pure data.

DIY DDR Controller

I built this DIY Controller out of Play-Doh, hooked up to a Teensy++.  Touching the play-doh sends a signal to the Teensy, which acts as a USB Keyboard/Gamepad/Joystick plugged into a laptop.  The picture shows the play-doh controlling Stepmania, an open-source and cross-platform version of DDR.