Step aside, Siri: Flexible displays could mean smartphones speak through movement

When Apple added the virtual assistant Siri to the iPhone, consumers got to experience what it was like to have their mobile device talk back to them. Now researchers at the University of British Columbia’s SPIN Lab are investigating a world where smartphones communicate more subtly — by moving on their own in ways that mimic human beings.

In “It’s Alive! Exploring the Concept of a Living Phone,” (J. Dawson, O. Schneider, J. Ferstay, D. Toker, J. Link, S. Haddad and K.E. MacLean), a team from UBC’s Sensory Perception and Interaction research group discuss how participants in their study reacted to a set of prototype smartphones that could use gestures. They argue that with the rise of flexible OLEDs and polymer membranes that inflate or contract, the idea of such smartphones should be possible soon. For research purposes, however, they used simple things like wood, foam and rubber to create the kind of futuristic designs they envision, which they named Curl, Breathe and Crawl.

“The Curl prototype demonstrated a phone sitting up and looking at the user. Its hard, plastic body was segmented then bound with paper tape to support a smooth curl in one direction. Popsicle sticks were used for manual animation,” the researchers write. “The Breathe prototype was constructed with grey closed-cell foam containing a balloon; a hard plastic back resisted the balloon’s expansion. A puppeteer blowing in and sucking out of a breathing tube could animate the prototype to simulate breathing. The Crawl prototype was a flattened, flexible shape with popsicle sticks at its ends for manipulation. Its freeform visual gestures included lateral movement along a table’s edge (walking or crawling, inchworm-style) and looking around (by sitting or standing up).”

In videotaped sessions, study participants were presented with the prototype phones and asked to describe their reactions. According to the research paper, those who took part were quick to associate the movements of the smartphones with animals like caterpillars, and to interpret their movements as emotional cues like “it wants your attention.”

The researchers found that simple gestures like curling went a long way to signalling that the phone wanted to be picked up. However orientation played a major role: when Curl was facing away from participants, they thought it might be hiding something. And the researchers admitted there may be differences in how gestures are perceived if they are holding a phone versus watching it, or if it’s sitting in their pocket or a bag.

“While not all of the gestures and expressions in the two studies were always interpreted according to our expectations, our evaluations suggest that gestural displays have great potential to enrich our device-mediated communications through expressiveness and emotion,” the researchers wrote. And gestural displays would be a lot quieter about it than Siri.