Leap Motion and Flying a Sim with Your Hand

In the last post we flew a real drone using the Leap Motion sensor to detect hand movements [post]. This post shows how to fly a simulator with your hand instead. While not immediately prompted by the crash at the end of the previous post's video, you can certainly do a lot less damage flying a simulator and it doesn't leave blade marks on the wallpaper. These two posts both demonstrate the control technology essential to the Drones4Good AI Racing concept. First, there's control of a real drone via Arduino, but, in order to train your AI to be able to fly, you need a flight simulator and virtual joystick emulation.

This is me flying a DreamFlight Weasel Trek off the slope and through a limbo gate using only my hand:

Things you will need:

1. A flight simulator. I used PicaSim, but any simulator with joystick control will work.
2. A Leap Motion Sensor.
3. The vJoy joystick emulator, available from sourceforge. I have version 2.18.
4. The vJoy SDK for the version of vJoy used in (3).
5. The Leap Motion SDK (free, but you have to sign up as a developer on UltraLeap.com).
6. Drones4Good software to take the Leap Motion inputs and feed them to the vJoy joystick (GitHub: https://github.com/maptube/rogue).

This build actually took me a lot longer to figure out everything about joystick emulators than it took to write the code. The Leap Motion part isn't a lot different from the previous post on flying a real drone, except that I've used the Leap C++ library this time instead of the Unity plugin. The vJoy part is basically a copy of their "feeder" example in the SDK, but with the force feedback bits removed. All the code is in the "rogue/LeapVJoystick" folder of the GitHub repository. Put simply, the program polls the Leap Motion for the hand orientation, converts this to an aileron and elevator stick position and sets this as the joystick position using vJoy. It's fun watching the joystick moving with your hand in the Windows joystick calibration app.

NOTE: I had never realised this before, but the Leap Motion has a setting called "Allow Background Tracking". This must be set to on, otherwise the Leap Motion stops tracking the hand when the application loses focus. You can find this option by right clicking the Leap Motion app in the windows tray in the bottom right and going to "Settings...".

Installation of vJoy is slightly tricky. The application is installed from the Windows installer package, following their instructions. This gives you a "vJoy Device" when you look at the Windows joysticks under "Setup USB Game Controllers" on the control panel. This is a software driver that pretends to be a hardware joystick (HID device). Then, the SDK is unpacked from a zip file so that it can be referenced by the C++ project. I have a "3rdparty" folder on the same level as the "LeapVJoystick" folder i.e. directly under "rogue" as cloned from GitHub. The Leap Motion SDK is also installed under this directory.

I'm using Visual Studio 2019 Community, but it's simple enough that any C++ compiler should work. The project file is included in the repository, which contains the include and library references, relative to the project file. If the Leap and vJoy SDKs are unzipped into the 3rdparty directory as suggested, then the paths should be correct. Otherwise simply change them in the project. Finally, in order to get the executable to run, you will need the "Leap.dll" and "vJoyInterface.dll" library files in the output directory where "LeapVJoystick.exe" is complied to.

And that's all there is to it. You can now fly off the slope using your hand.

It sure is good to fly something that isn't a quadcopter! But there is a very good reason for why a quadcopter wasn't used. I wanted to keep things simple and use single hand roll and pitch for two channel control. Typically, on a radio control aircraft, this would either be rudder/elevator or aileron/elevator with a 2 channel glider. So, no throttle control. PicaSim is one of the few simulators that had realistic slope soaring, and I got rather addicted to throwing the Weasel Trek around the sky.

To explain what slope soaring is, the phrase, "you mean they just chuck it off a cliff?", is rather appropriate. The way it works is that the wind blows towards the slope and you throw the aircraft off the edge directly into the wind. You can hear the wind from the simulator's audio in the recording, and it's blowing a gale. The reason we can use this to keep an aircraft in the sky is that the wind blows towards the slope, comes into contact with the ground and gets directed upwards. This means that there is good lift along the edge just in front of the slope. There is also some nasty turbulence, called "rotor", but we'll ignore that for now. The point is that there is a huge amount of wind energy that can be harvested if we put the aircraft in the right places. What I'm doing when I'm flying the Weasel is managing this energy by placing the aircraft to control its height and speed using only the ailerons and elevators.

The way I fly the limbo obstacle is to pick up speed and height over the slope edge, then fly a big left hand circuit, so as not to lose too much energy when I'm flying over land and I'm losing speed and height fast. I can control the radius of the turn so that I'm in the right position to go under the limbo without so much speed that I just balloon over the top (see video above).

Then I had a go at some thermal flying with a hand launch glider (HLG), often referred to as a discus launch glider, or DLG due to the spin and throw launch technique rather like an Olympic discus athlete. I had a bit of a problem with launching here, as it takes time to get my hand back over the Leap Motion and for the sensor to settle, by which time the glider has already pinged off the top of the launch and I've lost most of my launch height already.

OK, so about 1m50s in I make the beginners' mistake of flying over my head so that I lose track of my altitude and pitch, resulting in a rather rapid arrival with the ground. The large dark objects that you can see circling are birds. They're bigger and hungrier than me, and I'm flying in their sky, but they know where the thermals are so I can use them to stay aloft. Also, when you fly a real HLG, the wings really are so impossibly thin that flying directly towards or away from yourself causes the aircraft to disappear from view. Just don't panic and keep flying it. I can't see much on the simulator either.

I thought that filming the screen in the dark would work better, as I tried something similar for an earlier post: http://casaucldrones.blogspot.com/2018/03/fly-drone-with-your-hand.html

The Dell Laptop in the front left of the picture is running the simulator, while the monitor is just a second screen. It's a lot bigger than the 13in laptop screen, so I thought filming it in the dark with my RunCam would work better and highlight my hand hovering over the glowing Leap Motion sensor.

So, how does the hand control compare to a regular transmitter? Here's me flying with my Taranis QX7S so you can see for yourself:

It's obviously a lot more accurate flying with a real controller, so I'd say that the hand control is interesting but that it still needs a little work. Here are some of the things I learnt:

1. It's hard filming a computer screen with a tiny, fast moving, aircraft on it. I tried filming in daylight and in the dark, but the screen was either too bright or too dark. If you're wondering, I can't see the aircraft very well either, that's just the way it is.
2. The position of the camera and tripod means I'm having to stand behind the camera and stretch my arm out past it in order to reach the Leap Motion. This makes it really hard to see the screen and fly. I've had better results when I'm sitting at the desk with no camera. Next time, maybe I'll try capturing the game video directly, filming my hand with the camera, and then compositing the two video feeds? I need to improve my video editing skills first, though.
3. It's easier flying a real quadcopter hovering right in front of you using your hand than it is flying a simulator. With the real quadcopter you can match your hand to its drift to find the level position.
4. Looking back at the video I can see now that my outstretched arm is causing my palm to be at an angle to the sensor. This is causing the elevator response problems I was getting. I constantly felt as though the aircraft didn't want to lift its nose up, but it looks like this is caused by my hand position.
5. When I'm banking, the normal feeding in of elevator to keep the nose up is not at all intuitive. Also, as I'm lifting my palm to add elevator, it's affecting the ailerons so I'm adding a turning moment too. You can see this most clearly when I try to loop and it just screws out on the ailerons. I think it would be better if the two controls were on separate hands to avoid any unwanted coupling. Using hand motion in addition to palm angle is another possibility.
6. Filming people flying with this system and then looking at the hand motions that they make to try to control the aircraft would be a useful research exercise. I found myself moving my hand up and down to try to gain height and left/right to control position.
7. In the videos I'm constantly watching the cross hairs in the bottom right to see what control input I'm putting in, together with the main aircraft view and the zoomed in view in the top left for orientation. Pilot workload is rather high just to keep it in the sky.
8. If you compare my control inputs in the Taranis QX7S video to the Leap Motion videos, then the Taranis is obviously a lot more precise. There is noise on the Leap controls, but it's mainly the centre position and the control coupling that is causing the problems.

As I stated at the beginning, the Leap Motion control of a 3rd party flight simulator shows how an external program can be used to control a simulator. This doesn't have to be the Leap Motion controller, as this was just a demonstration. An AI algorithm using Reinforcement Learning can control the aircraft in the simulator using joystick emulation using exactly the same technique. The idea is to let the computer learn on the simulator before using the techniques from the previous post to let it loose on a real drone. This is how we teach the computer to fly. And teaching an AI how to manage the energy on either a thermal or slope soarer is an interesting problem and one which Microsoft looked at in their AI Powered Glider.