Gesture Based Interfaces–Bringing Technology Back to the Human Roots of Language

In the not-too-distant future, a gesture-based interface for personal computers could free us from keyboards and even touch screens, styluses or whatever other cumbersome and easy-to-lose devices are out there and trendy now. Warning: They may not be as sexy as Tom Cruise’s setup in the Minority Report, but Pattie Maes and Pranav Mistry at MIT are working on a technology that has the potential to become a “sixth sense,” as Maes puts it. She explains her work in a fascinating talk hosted by TED Conferences and posted online.

Think about how the iPhone touch screen works. To resize an image on the screen, users pinch their fingers together, or slide them apart. The gesture is completely natural. My mother is not usually the first one on board with a new technology, but even she had no trouble understanding such a simple command.

While I might get a call from her asking me to explain how to attach a picture to an email twenty times, simple gestures like finger slides stick with people in a way that file structures, sub menus and mouse clicks never will. This is a dramatic change from the crazy stylus languages that early Palm Pilot victims had to endure to accurately make use of the devices.

As humans we are predisposed to use language—languages that are both spoken and gestural. As early as half a centry ago, famed MIT linguistics professor Noam Chomsky championed the theory that all languages have a ‘universal grammar’ dictated by our genetic code, suggesting that we are literally wired to communicate in certain ways.

Whether we communicate through words or gestures, evolution has provided us with the tools to communicate with each other. So argues the linguist Richard P. Meier, of the University of Texas at Austin, who applies linguistic theory to American and Australian sign language in a fascinating book review. For those of you with an interest in linguistics, and with access to the journal Natural Language & Linguistic Theory, the 16-page review is a great read.

Communication with computers, however, is something we have to learn from scratch. There is nothing intuitive about the QWERTY keyboard – it may have actually been designed to slow typists. Back in my high school keyboarding days, the frustration of contorting my hands to type a ‘c’ or an ‘x’ drove me to keyboard-mashing fits, and I know I’m not alone.

Maes’ and Mistry’s research may never eliminate keyboarding classes, and I have no idea if the gestures employed by Tom Cruise are actually borne of our inherent language abilities. But my mom gets it, and I for one am happy to give up a few hours of acting as the family “tech guy.” If researchers really do tap into our natural tools, it could save us all a lot more than some time on the phone.

Related on Scienceline:

Green Brain Waves.

Farshad Khorrami’s Robotic Ray.