Monday 9 July 2012

Raspberry Pi and the Kinect

There are several forum posts asking if the Raspberry Pi could be used with the kinect so I decided to give it a try. It is important to note that you will need a powered USB hub, as whilst the kinect does have a PSU this is only used for the motor, the Camera and Audio sub systems still need more power than the Pi can produce.

First you will need to get a couple of libraries, first off libusb is required, I downloaded the latest 1.0.9 tarball and built it using the following commands

tar vfxj libusb-1.0.9.tar.bz2
cd libusb-1.0.9/
./configure
make
sudo make install
This should work fine for both of the debian versions and this will install the developer libraries and headers for libusb. Next I downloaded the OpenKinect source code from git hub unzip this file and change into the source directory.

You may need to install cmake if you have not already done so, this can be done by using sudo apt-get install cmake. Next we need to edit some of the cmake files as for this example I don't want to build the demos which require libraries which will not work properly on the pi.

If you edit the CMakeLists.txt file and search for the following line

OPTION(BUILD_EXAMPLES "Build example programs" ON)
And change the ON to OFF you should now be able to build by typing the following
cmake CMakeLists.txt
make
sudo make install
This will then install the following files
ls /usr/local/include/libfreenect/
libfreenect.h  
libfreenect-registration.h  
libfreenect_sync.h
and
ls /usr/local/lib/libfree*
/usr/local/lib/libfreenect.a
/usr/local/lib/libfreenect.so.0.1
/usr/local/lib/libfreenect_sync.a
/usr/local/lib/libfreenect_sync.so.0.1
/usr/local/lib/libfreenect.so
/usr/local/lib/libfreenect.so.0.1.2
The first demo I've tried is a modified version of the tiltdemo.c There are reports of this working fine for some people, however it didn't for me under the latest wheezy build so I investigated more and found that the sync library wasn't working for me. The following program uses the Normal freenect library calls instead.
#include "libfreenect.h"
#include <cstdlib>
#include <ctime>
#include <iostream>

int main(int argc, char *argv[])
{
 // seed rng generator
 srand(time(0));
 // pointer to the freenect context
 freenect_context *ctx;
 // pointer to the device
 freenect_device *dev;


 if (freenect_init(&ctx, NULL) < 0)
 {
  std::cout<<"freenect_init() failed\n";
   exit(EXIT_FAILURE);
 }
 // set the highest log level so we can see what is going on
 freenect_set_log_level(ctx, FREENECT_LOG_SPEW);

 int nr_devices = freenect_num_devices (ctx);
 std::cout<<"Number of devices found: "<<nr_devices<<"\n";
 // I only have one kinect so open device 0
 if (freenect_open_device(ctx, &dev, 0) < 0)
 {
  std::cout<<"could not open device error\n";
  freenect_shutdown(ctx);
  exit(EXIT_FAILURE);
 }
 // now I'm going to loop and set random value
 // these are basically from the tiltdemo.c that comes with
 // the freenect lib modified not to use the sync lib
 while (1)
 {
  // Pick a random tilt and a random LED state
  freenect_led_options led = (freenect_led_options) (rand() % 6); // explicit cast
  int tilt = (rand() % 30)-15;
  freenect_raw_tilt_state *state = 0;
  double dx, dy, dz;
  // Set the LEDs to one of the possible states
  freenect_set_led(dev,led);
  // Set the tilt angle (in degrees)
  freenect_set_tilt_degs(dev,tilt);

  // Get the raw accelerometer values and tilt data
  state=freenect_get_tilt_state(dev);

  std::cout<<"led["<<led<<"] tilt["<<tilt<<"]\r" ;
  std::cout.flush();
  sleep(1);
 }
}

To build this I used the following makefile
CC=g++
CFLAGS=-c -Wall -O3 -I/usr/local/include/libfreenect
LDFLAGS=-L/usr/local/lib -lfreenect
SOURCES=tiltdemo.cpp
OBJECTS=$(SOURCES:%.cpp=%.o)
EXECUTABLE=tiltdemo

all: $(SOURCES) $(EXECUTABLE)

$(EXECUTABLE): $(OBJECTS)
 $(CC) $(LDFLAGS) $(OBJECTS) -o $@

.cpp.o:
 $(CC) $(CFLAGS) $< -o $@

clean :
 rm -f *.o $(EXECUTABLE)

A video of it in action can be seen here

43 comments:

  1. Have you managed to get video working?

    ReplyDelete
    Replies
    1. I have! I successfully got the depth feed video out of the Kinect onto my Raspberry Pi recently. Check out my blog post w/ screenshot for more info here

      I can answer any questions quickly if you leave a comment.

      Delete
    2. how about the audio ... i have projects that could benefit from the full functionality in say a greenhouse controller ... can't say too much but i can come up with 5 gallons of molecularly pure water for ever kwh a municipality needs ... i want to make the greenhouses out of easily replaceable parts

      Delete
  2. Not yet still working on it, at present there seems to be a bottle neck in the USB transfer. Several people are working on it but nobody has any proof of it working as yet! Started using the new hardfloat raspbian build so hoping for better performance and to get it working.

    ReplyDelete
  3. I use this tiltdemo.cpp with the raspbian image and it works fine. I have a problem at first start "error while loading shared libraries" and solve this by refreshing my ldconfig cache.

    ReplyDelete
  4. yes you could also add

    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

    to you .bashrc and typing source ~/.bashrc (only need to do this once)

    ReplyDelete
  5. Have you tried disassembling Kinect to see if internals can be pulled out and put into more compact device probably same size as RPI itself?

    ReplyDelete
  6. Not tried to do that as I still sometimes use the kinect on my xbox!

    ReplyDelete
  7. Any luck with getting the video working on the Kinect?

    ReplyDelete
  8. I'm also checking in to see if you've got video or IR working yet?

    You mentioned "Several people are working on it" -- Is there a forum somewhere, or an IRC channel?

    I'm tinkering around with my Raspberry Pi and Kinect as well. I'd love to contribute, but mostly want to keep track of the news as it comes out.

    ReplyDelete
  9. As far as I know nobody has had any luck with video or depth, I'm considering getting the new 512M version of the pi and giving it another go but think it is unlikely y to work as the bottleneck seems to be in the USB drivers

    ReplyDelete
    Replies
    1. I see. Thanks for the reply.

      Delete
    2. Depth works well with this: https://github.com/xxorde/librekinect
      Video with linux/drivers/gspca/kinect

      Delete
  10. Does any one know what is the reason it is not working? when I folow the init seqence for each control message I send though the magic number is given out it always gives an output of 2 0. where as desired output seems 0 0.

    ReplyDelete
  11. This comment has been removed by the author.

    ReplyDelete
  12. Any further development? I followed this tutorial, have the Kinect connected via powered USB hub, and am getting the following error when I run ./tiltdemo:

    Number of devices found: 1
    Could not open motor: -3
    could not open device error

    ReplyDelete
  13. To run as a normal user you need to set the udev rules, see the openkinect website http://openkinect.org/wiki/Getting_Started#Use_as_normal_user

    ReplyDelete
  14. I'm actually more interested in the audio for voice recognition... Have you tried that (curious if the array of microphones the kinect uses can cancel out background noise for cleaner voice commands)

    ReplyDelete
    Replies
    1. Hi,
      You should find here any info on how to use the audio part : http://connectingstuff.net/blog/reconnaissance-vocale-sarah-raspberry-et-le-kinect/

      Delete
  15. Not attempted the audio side of things yet. There are some issues as it has to have the firmware uploaded to the Kinect and I'm not sure how well this works yet. Have a look here http://openkinect.org/wiki/Protocol_Documentation#NUI_Audio

    ReplyDelete
  16. Hi,Jon
    I run the tiltdemo.cpp,
    at first I got a "libfreenect.so.0.1 cannot open shared object file" error
    then I >sudo ldconfig
    but got the result:
    Number of devices found:1
    Could not open motor: -3
    could not open device error

    ReplyDelete

  17. I tried python with a basic example to control the motor and led, it doesn't work with the error that reports no device is plugged in.

    I use the 2013-02-09-wheezy imagine

    However, lsusb returns an entire list for
    Xbox NUI Audio/Motor/Camera.

    ReplyDelete
  18. Can't find the CMakeLists.txt file to edit

    ReplyDelete
  19. Would I be able to just use the Kinect's depth camera in a mobile application using only the Raspberry Pi's usb to power the Kinect?

    ReplyDelete
  20. Hi
    i installed it on my raspberry pi (arch linux) and dont get the whole list with lsusb just the motor and the hub... can someone help me plz?

    ReplyDelete
    Replies
    1. rpi is probably not powering USB sufficiently. Try a powered usb hub. Similar problem descriptions: https://groups.google.com/forum/#!topic/openni-dev/5N09f5St7gE

      Delete
  21. Hallo Jon,
    nice example and wonderfull description.
    Have you tested the RGB or Depth camera of the kinect yet?

    Why have you modified the tiltdemo?

    Thanks

    ReplyDelete
  22. This comment has been removed by the author.

    ReplyDelete
  23. Hello Jon,
    Thanks for such a great tutorial. I done everything and everything worked upto the point of compiling the code through the makefile. It is showing >>makefile:11: *** missing separator. Stop. So in 11th line of the makefile some error is there. But i am not able to find it out. It will be so helpful if you help me with this. Thank You.

    ReplyDelete
    Replies
    1. If I am removing the 11th line : "$(CC) $(LDFLAGS) $(OBJECTS) -o $@" from the code. Then same error is shown at "$(CC) $(CFLAGS) $< -o $@" th line.

      Delete
    2. I managed to manually compile the cpp file. But when i am trying to run it there is an error showing like this:
      ./nw: error while loading shared libraries: libfreenect.so.0.2: cannot open shared object file: No such file or directory
      Please help me with this.

      Delete
    3. I am not able to find any library starting with libfreenect in my /usr/lib/ directory of RPi

      Delete
    4. @FrleNd: I go the same error (makefile:11) and for me worked out to use a TAB in front the compile command instead of white spaces.
      To solve the shared Library issue I set variable LD_LIBRARY_PATH with the command export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

      Delete
    5. @FrieNd: you can fix the libfreenect.so.0.2 error as shown here: http://openkinect.org/wiki/Getting_Started#Testing_the_Kinect

      If glview gives a shared library error:
      $ glview
      glview: error while loading shared libraries: libfreenect.so.0.0: cannot open shared object file: No such file or directory
      glview: error while loading shared libraries: libusb-1.0.so.0: cannot open shared object file: No such file or directory
      You need to refresh your ldconfig cache. The easiest way to do this is to create a file usr-local-libs.conf (or whatever name you wish) with the following lines:
      /usr/local/lib64
      /usr/local/lib
      Switch to root account and move it to /etc/ld.so.conf.d/usr-local-libs.conf. Then update the ldconfig cache:
      $ su root
      $ mv ~/usr-local-libs.conf /etc/ld.so.conf.d/usr-local-libs.conf
      $ /sbin/ldconfig -v

      Delete
  24. no one of this librarys, already exist now, right?
    because, i try to install and I cant whit no one.

    ReplyDelete
  25. Depth on raspberry pi works well with this: https://github.com/xxorde/librekinect
    Video with linux/drivers/gspca/kinect

    ReplyDelete
    Replies
    1. Hi,
      I'm interested in librekinect as well, but I'd like to see a demo video if possible..do you perhaps have screenshots or a video of librekinect in action?

      Delete
  26. This comment has been removed by the author.

    ReplyDelete
  27. Check out my reply above for screenshots of the Kinect depth image on Raspberry Pi. Reply to the post on my blog if you have any questions.

    ReplyDelete
  28. thank you so much for sharing the code and the video

    raspberry pi boards in australia

    ReplyDelete
  29. Hello.
    Sorry, I have a problem when I try to compile, the make file show me fatal error: libfreenect.h: No such file or directory
    #include "libfreenect.h".

    ReplyDelete