Gesture recognition has been a hot computer science topic for over a decade. Simply put it is the interpretation of human gestures, using advanced mathematical algorithms, to control and interact with devices without physically touching them.

In 2002 a science fiction thriller, directed by Steven Spielberg, hit the big screen. One of the films scenes has gone down in movie legend for its glimpse into how we might interact with technology in the future. The name of the film is of course Minority Report. 

PreCrime Captain John Anderton, played by Tom Cruise, uses a “spatial operating environment” interface to view future crimes. Audiences were amazed to see the character use “swiping” hand gestures to interact with the information on the “screens”.

In 2013 gesture technology is all around us…

Most commonly seen is gesture recognition from either the face or hand – although body gesture recognition is becoming hugely popular in the gaming world due to the success of Microsoft’s Kinect technology.

Samsung has created a growing range of “Smart TVs” that use hand gestures to interact with the dashboard via a small camera embedded into the TV frame. Facial gesture technology is used to great effect in helping with those with severe motor impairment use assistive and home automation technology to improve their lives.

The Leap Motion Controller

In July 2013 an American technology firm called Leap Motion released a small USB peripheral device called the Leap Motion Controller. (It was inspired by frustration surrounding 3D modeling using a mouse and keyboard.)

It is designed to track fingers, or similar items such as a pen, using three built in sensors and allow the user to point at, wave, reach, and even grab an object on a computer screen.

It has a 150 degree field of view, tracks movements at over 200 frames per second, is 0.5 inches (H) 1.2 inches (W) 3 inches (L), tracks all 8 fingers and 2 thumbs upto 1/100th of a millimeter and has 8 cubic feet of interactive 3 dimensional space.

Setup is easy if you meet the minimum system requirements

Windows 7 SP1 or Windows 8 (32 bit and 64 bit)
AMD Phenom II or Intel Core i3, i5 or i7 processor
2 GB RAM and a USB 2.0 port

or

Mac OS X 10.7 Lion or higher
Intel Core i3, i5 or i7 processor
2 GB RAM and a USB 2.0 port

…an Internet connection is also required and the device works in a USB 3.0 port with no issues.

Please note: some 3rd party apps may require additional downloads and higher specs such as Windows 64 bit only.

Once installed the controller can be used to access registered apps from its own app store called AirSpace – you will require a free account and some apps come with a small fee.

Popular apps include “Cut the Rope” “New York Times paper” and “COREL Painter Freestyle” and the “GameWave” app that allows you to control keyboard and mouse-based video games with your Leap Motion Controller

You can also use the Leap Motion controller to interact with your operating system (Win or Mac). To do this you will need to download and install the free “Touchless App” from the AirSpace store.

Leap Motion for Graphic Design?

We spent a morning working with the Leap Motion controller – specifically focused on design and illustration based tasks.

Our first port of call was to use the controller with Adobe Photoshop CS6 and see what we could accomplish. As you will see in the above video, this was sadly not very much. When using the Leap Motion controller with the Windows Touchless App, we could achieve basic tasks such as navigating menus, creating a new canvas and adding layers etc.

However we soon ran into problems when trying to use the brush and pencil tools and had to give up with the controller and revert back to a tablet and stylus.  

One option may be to use the left hand to navigate the software using gesture control, and with the right hand use a stylus and tablet to create and modify the content on the screen. 

Another option may be to use Dragon Naturally Speaking and the Dragon Productivity Pack by VoxEnable; that allows you to issue voice commands to Adobe Photoshop CS4/CS5/CS6. You could then feasibly combine voice control with gesture control from the Leap Motion controller.

A 3rd option may be to wait for a 3rd party Leap Motion app developer such as GameWare to build a profile that allows you to assign specific gestures as shortcuts within Adobe Photoshop CS6. But they would need to work with Adobe on this to gain better control over the brushes  etc.

However, I think these three “options” are not going to inspire many designers… they are too used to using either their favorite keyboard shortcuts or their programmed tablet shortcuts. Convincing them to use the controller with the above workarounds, would be a thankless task!

The problems with Adobe Photoshop CS6 did not stop there. Even if we had managed to get a brush to work, since there is NO emulation of the “pressure sensitivity” designers and illustrators are used to experiencing when working with a graphics tablet. Expecting them to work without this feature is pointless from a productivity angle.

Even as I write this blog post Wacom has just announced its new Intuos Creative Stylus for the iPad. It comes with integrated shortcut buttons and 2,048 levels of pressure sensitivity to create “a realistic pen-on-paper feel” when sketching, drawing, and painting on an iPad.

With these kind of tools available for designers and illustrators, the controller from Leap Motion is simply not an option worth considering.

Closing Thoughts

The Leap Motion controller opens up a world of potential for gesture control on Windows and Mac laptops/desktops. However I don’t yet feel this “potential” is actually being realised as the Leap Motion controller is NOT yet ready for the world of productivity – particularly from a design/illustration perspective.

When using the Leap Motion controller there are just too many niggles and issues to overcome, for this to be a serious workplace tool.

Chief amongst my requests to Leap Motion would be:

#1 – Addition of “Right Click” emulation…

#2 – Pressure sensitivity emulation…

#3 – Some intelligence built into the algorithm that can detect shaky hands/wrists that have been using the device for a while!

#4 – Self learning algorithms that gets to know the users gestures (everyone’s motor skills are not the same) and that adjust/compensate accordingly…

#5 – Profiles built into the “Touchless” app that directly work with the Adobe Suite of software…

#6 – Consider a hybrid model that combines gesture control and voice commands…

#7– Consider a partnership with Smart Glasses maker Vuzix.com and create the ultimate system for 3D modelling

Don’t misunderstand me. I like this device and if Leap Motion ever introduced “graphic tablet emulation” then this could be a game changer for the company. Until then I can not see this device being used by an army of designers and illustrators any time soon…