Leap Motion and the Emperor's New Drone Controller

Hand control of a drone using a Leap Motion sensor

After taking our Leap Motion controlled flight simulator to the Big Bang Fair and IET events (and the [FlyADroneWithYourHand] post), we always wondered whether it was possible to fly a real drone with your hand? This idea came around again after seeing the new DJI trigger controller for their cinematic drone, which is a new innovation in the field human computer interaction for piloting UAVs. For anyone researching this field, the ability to test out a new control system without actually having to build any new hardware is an interesting prospect. And so the "Emperor's New Drone Controller" project was born. You can skip to the videos below if you just want to see the flying.

The hardware follows our earlier post on computer control of a radio control transmitter here: http://casaucldrones.blogspot.com/2021/04/d4g-ai-racing-part-3-computer-control.html

The radio is bound to the drone and the trainer master/slave switch is used to switch between Leap Motion control and regular control. The Leap Motion sensor is plugged into the laptop, which is running some custom software that we developed in Unity, also available in the "ROGUE" GitHub repository under "LeapUnityDroneController": https://github.com/maptube/rogue

Originally, I had intended to write the Leap Motion code in Python to make it more portable, so there is a partially finished Python controller on GitHub too. Unfortunately, it was only when I got to the testing stage that I discovered that Leap Motion only ever supported Python 2.7 with an out of date SDK. At that point I decided to abandon Python support and develop using the latest Leap SDK (Orion 4.1.0) and Unity 2019 with the Leap plugin (4.8.0 for Unity 2019+). This turned out to be a really good decision, as I was able to show the stick and hand positions as a graphic (see below).

Just to recap what we have is Leap Motion => Laptop => Arduino => TX Trainer Port => Drone

This is how the control system works:

Tilting the hand back so the fingers point to the sky pulls the elevator stick back and causes the drone to translate backwards. The two squares and four sliders show a virtual radio control transmitter in mode 2, so the red stick on the right is aileron/elevator. NOTE: I had to reverse the elevator to match the Taranis radio, which is why the slider position on the right doesn't match the red stick position.

Pointing the fingers to the ground pushes the elevator stick forwards and causes the drone to tilt down and translate forwards.

Left and right aileron is achieved by tilting the hand left or right.

Throttle is controlled from the height of the hand above the sensor. Note the vertical position of the hand in the two pictures above. Mid throttle is with the hand about 30cm above the sensor, then 20cm up from there gives full throttle and 20cm down zero throttle.

Theory is all well and good, but does it fly? The video below shows the real first flight, which resulted in a rapid unscheduled disassembly due to impact with the wall. This was followed by a switch to one of our own 3D printed frame designs, which is more robust and stable in the air. The final 30 seconds show controlled flight with the new drone, which put a really big smile on my face afterwards.

OK, so to explain, it wasn't completely my fault. I did a systems test before turning on the camera to see that everything was working and I flew it using the sticks on the Taranis first. This style of frame has some velcro ties underneath that can stick to the carpet. I'm used to it happening, so I pulled the drone off and made sure it was free before switching the camera on and filming what you see above. The trouble was that, as I lifted my hand to power up, I tilted my hand to the right, pushing the right side of the drone back down onto the carpet and sticking it fast. As it was my first time flying with a Leap Motion, I hadn't quite got the hang of keeping my hand level and I was trying to watch the laptop screen to see what control inputs I was putting in, while at the same time watching the drone. I thought it hadn't got enough power to lift and that I wasn't holding my hand perfectly straight, not realising that the angle and the thrust have stuck it back down onto the carpet. I could actually feel it responding to my hand movements, so I continued with the flight, lifted my hand to put more power in to lift off and bang. It pings off the carpet and bounces off the wall just out of shot. I was actually quite happy with that as a first flight, as I had proved that everything works and the new form of control was exciting. So, then I disassembled the drone and switched the frame to our own design one, which doesn't have the exposed velcro underneath. Then I filmed the next flight, which was a big success.

The flying in the first video is very rough, but it was my first time and I was rather excited. A few days later I put together a proper test flight video, which you can see below.

I test fly a lot of radio controlled aircraft and drones, but this was something a bit different. Before flying, though, there are a couple of things I did to increase my chances of success:

Test pilot tip #1: Fly normally with the sticks first to get used to how the drone reacts and trim it to hover with the controls centred.

Test pilot tip #2: It's also a good idea to run the battery down a bit by flying on the sticks first, just to take the edge off the acceleration you get from a new battery so that it's easier to fly.

If you watch the two videos, it's apparent that I have two switches on the Taranis that I'm using to start and end flights. The two position left shoulder switch is programmed to switch between the trainer master and slave. Normally, this would be the spring loaded switch on the right shoulder, but I need to enable the slave control for the Leap Motion and then let go of the switch without it springing back to the master. The alternative trainer switch is programmed through the "Special Functions" menu on the Taranis to override the default assignment.

The other switch is a three position switch on the right top front panel of the transmitter and is programmed to arm the drone. I got this wrong initially, because I thought that there needed to be an arm command included in the data stream fed into the trainer port from the Arduino. The drone is running CleanFlight, with channel 5 the flight mode and channel 6 the arm channel. The trainer port mixing is only set up to take channels 1-4 from the trainer (TAER), so the arming is performed by the master i.e. the right hand switch on the Taranis. When the drone is armed, the rotors are set to spin at a low rate as an indication to the pilot that the throttle is now live. During the flights, I have the throttle stick to zero, so, by flicking the left switch back to master, I immediately go to throttle zero, although it is still armed until I flick the right switch. The flight mode channel is not used in this setup, so the flight mode remains at auto leveling. Depending on how the future flight tests go, I might change the flight mode at a later date.

One thing I am disappointed about is that the hand tracking and stick positions are not visible on the laptop screen in the video. The display it shows during flight is the same as the six stick position screen grabs from earlier. I was hoping to use that in conjunction with the hand and drone position to evaluate the controller. It's just a shame that the screen is too bright for the video to pick up clearly, otherwise the hand and stick positions would have been visible on the laptop screen.

Having reviewed all the footage, there are a few points worth making:

1. It takes a bit of practice. By the end I'm obviously getting a lot better at it and not over-controlling it as badly as in the earlier flights.
2. The drone's controls are too sensitive, leading to over-controlling it. I don't think it comes out on the video, but when I'm flying on the sticks near the end, I'm making very small movements in order to hover. I then copy that with the Leap Motion, but I'm moving tiny amounts.
3. Hand tracking by the Leap Motion is very hit or miss. OK, usually "hit", as in hit the tripod, hit the floor, hit the wall. I hit all these things when the hand tracking went berserk and started throwing out random control positions. It's worth pointing out that the height tracking (throttle) is quite reliable, while the hand vector to get the aileron and elevator is not so good. Also, the software is programmed to go to throttle zero if hand tracking is lost. The Leap API appears to be giving a good hand confidence, even when the hand tracking is suspect, so the throttle zero on loss of hand confidence is not working. This is different from the previous API, which gave a good hand confidence estimate. We used the older Leap API when we did the Big Bang Fair with the simulator, were it worked really well.
4. On the second video, at 19s, there's a point where the drone is speeding backwards towards me and I stop it from hitting myself with a downward motion of my hand. This was deliberate. There was never any point where I thought it was going to hit me. The control is very intuitive if you can get the hang of it. Really, I need a bigger space, but with a bit more work, I think you can do some interesting things with this technology.
5. There's no rudder. When I'm flying it with the sticks there's a point at 2m50s where I give the rudder a little right kick just to straighten things up. I haven't figured out how to do this with the hand control yet, but it's an essential feature. With the simulator at the Big Bang Fair we used coupled ailerons and rudder (CAR) to achieve this, but that's not an option flying a real drone in a confined space. You need separate controls.
6. In Leap Motion's own hand tracking visualiser on the diagnostics menu, it appears that their hand tracking is better than what I'm getting through Unity and the Leap API. Try a trigger action (i.e. air guitar with a pretend DJI controller) and see what you think? I'm probably doing something wrong, but it needs investigation.
7. Two handed control is a definite possibility for the future.
8. To make small corrections you can move just your fingers, which is a very intuitive way to fly.

So there you have it, the Emperor's very own drone controller that you can use to investigate new forms of human/drone interaction without actually having to make anything.