Kinect Controlled AR. Drone Quadrocopter


Introduction

        Feeling cool using Kinect to control your avatar in Xbox games? Perhaps you’d be more excited controlling actual, physical things with your movements, rather than a virtual representation. “Kinect is a motion sensing input device by Microsoft for the Xbox 360 video game console and Windows PCs”1. With the Kinect, technology just gives another reason for the world to keep on believing, dreaming and achieving. Really enjoyed operating an AR.Drone quadrocopter with your iPhone or iPad? How cool is that if your can control it with your body movement captured with Kinect?

You must be wondering how it is implemented and make your own.



Implementation

With the Kinect SDK(software development kit), we are able to take the advantage of:

  • Depth data, which provides the distance of an object from the Kinect camera, as well as the raw audio and image data, which together open up opportunities for creating richer natural user interface experiences.
  • Highly performance and robust skeletal tracking capabilities for determining the body positions of one or two persons moving within the Kinect field of view.

First, let’s take a look at the controlling process from a high level architecture.
high-level architecture
The center (brain) of this architecture is the controller software, developed with
Microsoft Kinect SDK, sitting in the OS on the laptop. The controller software plays the following roles:

  • It communicates with the AR. Drone over Wi-Fi
  • It reads and analyzes human motion data get from the Microsoft Kinect connected with a USB cable

The Kinect motion sensors detect user movements and send data to the controller application. Whereas, AR. Drone receives commands from the controller and carries out the requested actions.

When it comes to implementing the controller software, we only need to worry about the user space programming with well-defined Drone DLL library functions and Kinect SDK. The library/SDK functions will respectively call the AR.Drone kernel level system calls and Kinect driver functions for you. (AR.Drone kernel level system calls that are functions directly communicating with the hardware, namely the AR.Drone. Kinect driver is also a kernel level system call library that communicates with Kinect directly, but with a fancy name.)
software layers

Graphic User Interface (GUI) is the view of the software, which the users see and control Kinect with. The software is programmed in C# with Visual Studio. Our software utilizes the handy functions provided by Kinect SDK and Drone DLL. Those functions then will call Kinect or AR.Drone kernel functions to get motion information from Kinect or send commands to AR.Drone.

The data flow can be simplified and abstracted as the following flow chart. Imagine a scenario that one moves his right hand towards left to make the quadrocopter go left, and then the quadrocopter does a flip in the air by raising his left hand. The quadrocopter keeps recording video with its front camera all this time and send it back to the controller software in real time. In this set of actions, the Kinect sensors keep sending depth and skeleton data to the controller for analysis all the time. The controller keeps analyzing changes of received depth and skeleton data over time. When it recognizes a right-hand-moving-left gesture, it signals the AR.Drone through Wi-Fi to fly towards left. Later on, the controller recognizes that the user raised his left hand, and signals the quadrocopter to do a flip.  During these processes, the camera keeps recording and sending data stream back to the controller in real time. The controller then decrypts the stream as video, and plays it on GUI (User Graphic Interface), which is the view and interface of our application.

Data Exchange

Brief Conclusion

The technology to control AR.Drone with Kinect is initially developed by hackers. However, it is widely used in research and education now.

Want to dig further on the ability dynamic controlled or autopilot quadrocopter? Check out this awesome video.



References & Additional Readings

Useful Links:
ARDrone 1.0 win32程序源码和ARDrone 2.0 C#控制程序源码—–PC控制飞机
Controlling an A.R. Drone with the Kinect and Clojure

Resources:

  1. Kinect, Wikipedia, http://en.wikipedia.org/wiki/Kinect

Video Credits:
Video 1. This video demo is the result of a ETH Zurich Bachelor Thesis by Armin Ambuhl, surpervised by Sergei Lupashin and Raffaello D’Andrea
Video 2. Video by Mark Muller, Sergei Lupashin and Raffaello D’Andrea