Talk and Point: Interactive Map Is Learning to Show the Way Untitled

September 16, 1999

Talk and Point: Interactive Map Is Learning to Show the Way

By MICHELLE SLATALLA
magine a lone tourist standing on a subway platform underneath Times Square, asking a map not only to show the fastest route to the Bronx Zoo but also to highlight the closest deli that sells pastrami sandwiches and the cleanest nearby public bathroom.

YOU ARE HERE - At Pennsylvania State University, a campus map prototype responds to speech and gestures.

The prototype of an interactive map that might one day be able to answer questions like those can be found in a laboratory at Pennsylvania State University. Researchers in the university's computer science and engineering department have been working for more than two years to create a computerized map that can respond not only to tourists' spoken questions but also to their hand gestures. Their prototype, a map of the university's main campus, in University Park, projects an image of buildings and grounds onto a 60-inch television monitor.
"You could create a map of any area you want, of course, including New York City," said Rajeev Sharma, the project's leader and an assistant professor of computer science and engineering, "and you could provide a limitless amount of information if you connected it to the Internet. We even want to develop a system that could handle having multiple people standing in front of it, everyone waving their hands, trying to figure out where to go. The map would decide who was in control and then answer that person."
For now, however, the map system's capabilities are far more modest. The prototype system consists of two computers -- a personal computer and a Silicon Graphics workstation -- that communicate, respectively, with a cordless microphone and a video camera to respond to a person standing in front of the map. A person can communicate with the map either by speaking or by using hand gestures, like pointing.
If, for example, a person speaks into the cordless microphone to ask, "Where is my dormitory?" the personal computer uses standard speech-recognition software, then responds with an on-screen image that shows the location of all the dormitories. The person is then prompted to choose one of them.
If instead of speaking the person points toward the map or waves a hand across its image, the workstation's video camera uses gesture-recognition software developed by Dr. Sharma's team, then responds with an on-screen image of a little red hand that moves across the map like a cursor.
Previous gesture-recognition projects have proved computers can learn to recognize static gestures. In 1966, for instance, researchers at Carnegie Mellon University taught a computer to recognize 14 gestures from the American Sign Language alphabet. Building on that, the Penn State team has developed a method to teach the computer to pick out the meaningful gestures among all the ones someone may be making.
"The lucky break came from watching the Weather Channel, when we realized that the kinds of gestures people make in front of a map are similar to the kinds of gestures the weather forecasters use," Dr. Sharma said.
Weather Channel recordings were used to teach the system to identify meaningful gestures.
Of course, the computer may never understand some things. "For example," Dr. Sharma said, "it doesn't understand a gesture that's obscene in one culture or another." Which, come to think of it, could make it less adaptable to Times Square.

Related Sites
These sites are not part of The New York Times on the Web, and The Times has no control over their content or availability.

Pennsylvania State University: interactive map project

Help/Feedback | Classifieds | Services | New York Today