## Monday, 9 July 2012

### Raspberry Pi and the Kinect

There are several forum posts asking if the Raspberry Pi could be used with the kinect so I decided to give it a try. It is important to note that you will need a powered USB hub, as whilst the kinect does have a PSU this is only used for the motor, the Camera and Audio sub systems still need more power than the Pi can produce.

First you will need to get a couple of libraries, first off libusb is required, I downloaded the latest 1.0.9 tarball and built it using the following commands

tar vfxj libusb-1.0.9.tar.bz2
cd libusb-1.0.9/
./configure
make
sudo make install

This should work fine for both of the debian versions and this will install the developer libraries and headers for libusb. Next I downloaded the OpenKinect source code from git hub unzip this file and change into the source directory.

You may need to install cmake if you have not already done so, this can be done by using sudo apt-get install cmake. Next we need to edit some of the cmake files as for this example I don't want to build the demos which require libraries which will not work properly on the pi.

If you edit the CMakeLists.txt file and search for the following line

OPTION(BUILD_EXAMPLES "Build example programs" ON)

And change the ON to OFF you should now be able to build by typing the following
cmake CMakeLists.txt
make
sudo make install

This will then install the following files
ls /usr/local/include/libfreenect/
libfreenect.h  
libfreenect-registration.h
libfreenect_sync.h

and
ls /usr/local/lib/libfree*
/usr/local/lib/libfreenect.a
/usr/local/lib/libfreenect.so.0.1
/usr/local/lib/libfreenect_sync.a
/usr/local/lib/libfreenect_sync.so.0.1
/usr/local/lib/libfreenect.so
/usr/local/lib/libfreenect.so.0.1.2

The first demo I've tried is a modified version of the tiltdemo.c There are reports of this working fine for some people, however it didn't for me under the latest wheezy build so I investigated more and found that the sync library wasn't working for me. The following program uses the Normal freenect library calls instead.
#include "libfreenect.h"
#include <cstdlib>
#include <ctime>
#include <iostream>

int main(int argc, char *argv[])
{
// seed rng generator
srand(time(0));
// pointer to the freenect context
freenect_context *ctx;
// pointer to the device
freenect_device *dev;

if (freenect_init(&ctx, NULL) < 0)
{
std::cout<<"freenect_init() failed\n";
exit(EXIT_FAILURE);
}
// set the highest log level so we can see what is going on
freenect_set_log_level(ctx, FREENECT_LOG_SPEW);

int nr_devices = freenect_num_devices (ctx);
std::cout<<"Number of devices found: "<<nr_devices<<"\n";
// I only have one kinect so open device 0
if (freenect_open_device(ctx, &dev, 0) < 0)
{
std::cout<<"could not open device error\n";
freenect_shutdown(ctx);
exit(EXIT_FAILURE);
}
// now I'm going to loop and set random value
// these are basically from the tiltdemo.c that comes with
// the freenect lib modified not to use the sync lib
while (1)
{
// Pick a random tilt and a random LED state
freenect_led_options led = (freenect_led_options) (rand() % 6); // explicit cast
int tilt = (rand() % 30)-15;
freenect_raw_tilt_state *state = 0;
double dx, dy, dz;
// Set the LEDs to one of the possible states
freenect_set_led(dev,led);
// Set the tilt angle (in degrees)
freenect_set_tilt_degs(dev,tilt);

// Get the raw accelerometer values and tilt data
state=freenect_get_tilt_state(dev);

std::cout<<"led["<<led<<"] tilt["<<tilt<<"]\r" ;
std::cout.flush();
sleep(1);
}
}


To build this I used the following makefile
CC=g++
CFLAGS=-c -Wall -O3 -I/usr/local/include/libfreenect
LDFLAGS=-L/usr/local/lib -lfreenect
SOURCES=tiltdemo.cpp
OBJECTS=$(SOURCES:%.cpp=%.o) EXECUTABLE=tiltdemo all:$(SOURCES) $(EXECUTABLE)$(EXECUTABLE): $(OBJECTS)$(CC) $(LDFLAGS)$(OBJECTS) -o $@ .cpp.o:$(CC) $(CFLAGS)$< -o $@ clean : rm -f *.o$(EXECUTABLE)


A video of it in action can be seen here

1. Have you managed to get video working?

2. Not yet still working on it, at present there seems to be a bottle neck in the USB transfer. Several people are working on it but nobody has any proof of it working as yet! Started using the new hardfloat raspbian build so hoping for better performance and to get it working.

3. I use this tiltdemo.cpp with the raspbian image and it works fine. I have a problem at first start "error while loading shared libraries" and solve this by refreshing my ldconfig cache.

4. yes you could also add

export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/usr/local/lib

to you .bashrc and typing source ~/.bashrc (only need to do this once)

5. Have you tried disassembling Kinect to see if internals can be pulled out and put into more compact device probably same size as RPI itself?

6. Not tried to do that as I still sometimes use the kinect on my xbox!

7. Any luck with getting the video working on the Kinect?

8. I'm also checking in to see if you've got video or IR working yet?

You mentioned "Several people are working on it" -- Is there a forum somewhere, or an IRC channel?

I'm tinkering around with my Raspberry Pi and Kinect as well. I'd love to contribute, but mostly want to keep track of the news as it comes out.

9. As far as I know nobody has had any luck with video or depth, I'm considering getting the new 512M version of the pi and giving it another go but think it is unlikely y to work as the bottleneck seems to be in the USB drivers

1. I see. Thanks for the reply.

10. Does any one know what is the reason it is not working? when I folow the init seqence for each control message I send though the magic number is given out it always gives an output of 2 0. where as desired output seems 0 0.

11. This comment has been removed by the author.

12. Any further development? I followed this tutorial, have the Kinect connected via powered USB hub, and am getting the following error when I run ./tiltdemo:

Number of devices found: 1
Could not open motor: -3
could not open device error

1. Fixed the errors by running under sudo

2. 10x

13. To run as a normal user you need to set the udev rules, see the openkinect website http://openkinect.org/wiki/Getting_Started#Use_as_normal_user

14. I'm actually more interested in the audio for voice recognition... Have you tried that (curious if the array of microphones the kinect uses can cancel out background noise for cleaner voice commands)

15. Not attempted the audio side of things yet. There are some issues as it has to have the firmware uploaded to the Kinect and I'm not sure how well this works yet. Have a look here http://openkinect.org/wiki/Protocol_Documentation#NUI_Audio

16. Hi,Jon
I run the tiltdemo.cpp,
at first I got a "libfreenect.so.0.1 cannot open shared object file" error
then I >sudo ldconfig
but got the result:
Number of devices found:1
Could not open motor: -3
could not open device error

17. I tried python with a basic example to control the motor and led, it doesn't work with the error that reports no device is plugged in.

I use the 2013-02-09-wheezy imagine

However, lsusb returns an entire list for
Xbox NUI Audio/Motor/Camera.

18. Can't find the CMakeLists.txt file to edit

19. Would I be able to just use the Kinect's depth camera in a mobile application using only the Raspberry Pi's usb to power the Kinect?

20. Hi
i installed it on my raspberry pi (arch linux) and dont get the whole list with lsusb just the motor and the hub... can someone help me plz?

21. Hallo Jon,
nice example and wonderfull description.
Have you tested the RGB or Depth camera of the kinect yet?

Why have you modified the tiltdemo?

Thanks