Why your finger will never replace your mouse; It’s hard to put a finger on what’s amiss with the new gesture-based computing peripheral from Leap Motion

Why your finger will never replace your mouse

August 6, 2013: 5:00 AM ET

It’s hard to put a finger on what’s amiss with the new gesture-based computing peripheral from Leap Motion.

By John Patrick Pullen

FORTUNE — He shot me. After a good eight seconds of flailing, grabbing, and poking at the air above my desk, Frank Welty finally unholstered his sidearm and put me out of my misery. Alas, it was only a game, but I never really stood a chance. My shooter, which in this case was my pointer finger, hadn’t hit a damn thing all day. The game, Fast Iron, is just one of dozens of apps available for the newly launched Leap Motion Controller. A peripheral that lets users control their computer through hand gestures, this device showed plenty of promise when it was announced in May 2012. Now, more than a year later, the $79 product has come to market, and after a week of feeling it out, it’s hard to point a finger at what exactly is wrong with gesture-based computing, at least in its current state.Setting up the Leap Motion Controller was unexpectedly easy. I had imagined having to input measurements like the controller’s distance to the screen, or heeding requirements like keeping the device at a certain height, but none of that was necessary. Other than downloading a software suite, the peripheral was more-or-less plug-and-play, the unit powered and connected to my Mac via a USB cable.

The Leap Motion software involved minimal hand-holding, and gave the impression that gesture-based controls would be easy to master, with the sensor seeming to pick up each finger and hand rotation cleanly. But the program’s lack of finish caught my eye — running on my Mac, the software’s full-screen capability resized all my other applications windows (a huge annoyance), and its graphics looked choppy, almost like they were low-resolution. The device couldn’t detect my hands shaking with worry over these concerns, but in hindsight, they were clear indicators of a lack of finish that would plague my experience with the controller.

For example, as I continued pawing through the device’s compatible software, poor graphics soon became the least of Leap Motion’s problems. The company’s Airspace Store, a proprietary app marketplace that sells third-party created software for the device, offers users a mix of free and paid apps for both Macs and PCs. The selection is mostly games, but there are learning and productivity titles already in the mix that show the range of what the Leap Motion Controller can do.

The first app I tried out was Swoosh. Upon launching the program, my computer totally freaked out, as if I was using the Leap to shadow box. At the same time, iTunes mysteriously launched and started playing the first movie file in my library (Anchorman, if you must know), blasting Swoosh’s music over the video. This happened every time I launched the third-party title, until finally I figured out the problem. Swoosh is a deejaying app that plays music from your iTunes library, and users can “spin” and “scratch” the tracks like a record with the Leap Motion Controller. I only discovered this after locating the app’s missing full-screen window, which for some unknown reason had been minimized. And when I finally loaded a song from my library into the software, the controls were unresponsive and unpredictable. I couldn’t do much of anything with it, except for throw my hands up and walk away.

That experience became a familiar one as I continued testing the device. I downloaded Frog Dissection, and managed to somehow zip open a virtual reptile using “wax on, wax off” gestures à la Karate Kid. I fiddled with Flocking, a visualization app that looks great but bores quickly, with simulated fish swimming to bits of light controlled by my fingers. I pinched and zoomed through the virtual universe with Exoplanet but had a heck of a time touching the stars. (Imagine trying to poke a pin with a wet noodle — it will make you reconsider any belief that God sticks his fingers in our tiny earthly concerns.). Meanwhile, another visualization app, Gravilux, worked as well as it does on other platforms (Android, iOS, Mac, and PC). Its developer, Scott Snibbe, also ported his iOS-based music app, OscilloScoop, to the Leap platform, proving perhaps that the controller’s problems are more about software execution than anything else.

While the Leap Motion’s games are underwhelming, there’s no denying that entertainment plays an important role in emerging technology. Just like computer gamers from the 1980s and 1990s evolved into the computer-savvy workforce at the 2000s tech boom, and the social media generation has matured into a crop of info-fueled workers today, forward-leaning products like the Leap Motion Controller have the potential to breed a future wave of dextrous, gesture hackers.

And though it is nowhere near as powerful (or expensive) as Microsoft’s (MSFT) Kinect sensor, the Leap Motion Controller did detect every movement — to a fault. If I pointed my finger out, the device often detected my thumb, too. If I was wearing my Nike+ (NKE) Fuelband, it would think the bracelet’s shiny metal clasp was another, mystery finger. Still, as an everyday Mac user, I am jealous of the Windows 8 touch interface, and I had hoped to use the Leap Motion as a smear-free touchscreen to drive my Apple (AAPL) computer — if only I could adjust to its overactive sensors.

To run the Mac operating system, Airspace offers two apps: Touchless for Mac (they have a PC equivalent, too) and Better Touch Tool. Both free programs, Touchless was made by Leap Motion, while Better Touch Tool was developed by German indie coder Andreas Hegenberg. Touchless consists of two zones, one for “hovering” (or moving the cursor) and the other for “touching” (or clicking the mouse). Using it was precise but awkward, akin to trying to use an iPhone with magnetic gloves. Sure, you can do it, but it wasn’t really meant to work like that. The Mac operating system’s buttons were too small to be pressed with a finger, let alone a quivering, airborne ghost digit that’s as fat as a hot dog. After about a half hour of trying to acclimate to the system, I finally gave the software the finger (totally undetected!) and went back to my trusty mouse.

Better Touch Tool returned much improved results but still has a way to go. By turning actual gestures (bringing both hands together like a slow clap, for example) into keyboard shortcuts (close window, sleep, etc.) at last the Leap sprang to life. I programmed a slew of moves that made sense to me, such as three finger wag left as “back” in Safari, but when it came time to use them, I swear the device wasn’t looking. Waving, I got nothing. Flicking, not a blip of recognition. Finally, I mimed choking the wretched thing, but my computer stared back, unamused.

So, I’ve decided to wave goodbye to the Leap for now. Hopefully, the next time I plug it in, I can give it a thumbs up — and maybe then, it will know exactly what I mean.