User Interface—The next battlefield--Part 1

User Interface (UI) is the next battlefield in consumer electronics and, as products mature, the race to differentiate based on the UI will accelerate.  Rather than compare minor differences in features sets, consumers will increasingly make purchases based on ease of use and access of features. Specifically, when innovation in the core functionality of an end product approaches maturity, consumers increasingly turn to the industrial design and user interface of the product to guide purchasing decisions.  Further, the UI is now defining the personality of a device and becoming an integral part of the branding equation, thereby creating emotional ties between consumers and their devices, and ultimately building loyalty to a particular product. 

With the functionality of two devices being relatively equal, UI becomes the crucial differentiating factor.  This trend is particularly noticeable in the mobile phone market, and is now spreading to other, more mature electronics devices. Motorola’s RAZR, for example, had effectively the same functionality as other handsets, when introduced. However, its sleek clamshell design and unique keyboard set it apart from the competition. Initially, Motorola was expecting to ship 750K units of the RAZR per quarter. However, within one year, demand increased faster than expected and volume shipments climbed to over 6.5M units per quarter, making it one of the company’s fastest growing devices. Several years later, Apple took the UI to a new level with its iPhone, which revolutionized not only how consumers interacted with their phones but also how they related to them. Prior to the iPhone, Apple had no presence in the wireless handset market; the unique design and user interface of the product helped the iPhone adopt a cult-like following and ignited competition in the “Smart Phone” segment. Currently, Apple ships over 37M iPhones per quarter, in arguably one of the most hotly contested and competitive consumer electronics markets in the world. Additionally, the iPhone remains one of Apple’s most highly profitability product lines, according to industry sources, partially due to its smart choice of its touch screen user interface combined with its easy to use software applications. Today, we are seeing this trend proliferate to other maturing products such as notebook computers, which are now transforming into the new category of “ultra books” with integrated touch, speech and gesture recognition.

Given the fickle nature of consumers, features that are well designed become ubiquitous, while those that don’t work well typically stall. Users have come to expect devices to work simply, even when performing complex tasks. However, consumers’ low tolerance for sub-par user interface generally leads to decelerating adoption of a particular feature that does not meet their standards, thereby slowing the overall market adoption. Two good examples of these trends are touch screen and speech recognition (SR). While there is no question that consumer demand for both features remains high, each has taken a different route in its adoption curve. On one hand, touch screens have become an integral part of many consumer electronics products, largely due to their elegant design, but more importantly due to their ease of use. This is apparent in many end markets including mobile phones and tablet computers, among others, where effectively all major original equipment manufacturers (OEMs) and original design manufacturers (ODMs) are offering products based on touch screen input. On the other hand, speech recognition adoption has somewhat lagged in embedded applications and remains in its early stages of adoption. While it is clear that speech recognition as a UI will eventually be adopted widely, the only consumer end market that has started to use SR broadly is the automobile industry, primarily in infotainment applications. To date, only a handful of companies offer cars with embedded SR, while multiple others are planning to introduce systems with embedded basic SR features. Expectations remain high and the profile on SR is accelerating, as was apparent in the recent Consumer Electronics Show, where multiple OEMs and ODMs introduced other CE devices with integrated SR. Prominent market research firms have cited recent studies that indicate consumers generally remain somewhat dissatisfied with embedded SR, particularly in automobiles, resulting in a slowing of consumer usage or in many cases consumers simply stopping use of this feature.

As is the case with many innovations in the technology arena, where the initial products start with a simple concept and eventually develop into complex systems, so too user interface innovation has shifted from the simple keyboard to touch screen, and will eventually shift to speech recognition and other more complex UIs. As this shift occurs, consumers will be faced with adopting many new and exciting technologies over the next several years. In the next phase of UI adoption, rather than require user training, devices with advanced UIs will need to adapt to consumers so they can be used “out of the box”. This will include not only being adaptable to basic functions but also to more complex features such as the specific preferences of individuals over time. To achieve this, devices need to become more intelligent and, as such, will require more processing power and memory.

Intelligent User Interface

At the leading edge of technology are self-intelligence and predictive intelligence.  Self-intelligence is defined as the ability of a device to diagnose its own status and take preventive measures, while predictive intelligence, is defined as the ability of a system to predict a user’s desired outcome. In general, self-intelligence uses information from sensors embedded within the system to verify and maintain reliable operation.  Many devices use warning indicators to alert users to a condition based on sensor input that may lead to degraded performance or imminent system failure.  Automobiles, for example, use sensors to indicate fuel and oil levels, among other things.  With additional processing resources, more advanced diagnostics may be implemented. In the handset arena, for example, companies such as Apple have been developing patents that use the phone’s internal sensors to help identify various operational issues.

Predictive intelligence, in contrast, is more complex than self-intelligence because the system has to understand the different options available to a user and predict which is most likely to be selected.  There are two types of predictive intelligence: active and passive.  A good example of active intelligence is the auto correction feature implemented in mobile phone text and email functions, which actively replace misspelled words. On the other hand, passive prediction is often used to suggest selections to users.  Two good examples of this are Apple’s Genius and Pandora’s Music Genome Project, both of which use a listener’s selection history to create a list of new songs the listener is likely to enjoy.

The key to developing a system with solid predictive intelligence is accuracy.  The more data the system collects about a user, the more accurately it can predict user needs.  Most predictive technologies assume that people are consistent in their interactions, but in practice, a person’s emotions and level of distraction have a significant impact on their choices.  Consider an automotive speech recognition system. Current automotive SR technologies are language based (i.e. developed to recognize speaker language), and in some cases include dialects. However, the emotional state of the speaker and its impact on the speaker’s accent is rarely taken into account, if ever.  A person who is angry, for example, will likely begin to speak more quickly and perhaps more loudly, resulting in variations in tone, most likely leading to lower recognition accuracy, further frustrating the user.  A system that can recognize the emotional state of the driver could adjust the recognition algorithms to compensate for these responses, such as by lowering the microphone gain.  Additionally, as the driver calms down, the system could continue to dynamically readjust the recognition algorithms to maintain consistent accuracy.

While it may seem that predicting a user’s state of mind could be a daunting task, the basic technologies exist today to make this a reality. For example, the camera on a cell phone could both recognize the face of a user and his/her current emotional state from facial expressions and body language.  Additionally, the touchscreen could sense a user’s urgent or agitated state by the force and speed of keyboard use. Similarly, a speech recognition system could monitor changes in the user’s tone of voice.  Each of these systems could coordinate their analysis results to work together to further improve overall accuracy, and ultimately provide a better user experience.

Beyond the basic functionality of UI, determining a user’s state has further applications such as enabling e-commerce or facilitating communication with friends and family.  For example, companies would likely pay a premium to place their mobile advertisements when a person is emotionally more receptive.  Additionally, a user could allow friends and family to access his or her emotional state before they complete a call in order to refine their communication tactics or to avoid discussing sensitive topics.

The irony is that for devices to become easier to use, the user interface must become more complex.  For example, facial recognition would become more difficult to implement when using live video compared to a static image, but likely result in a better user experience.  Similarly, speech recognition will become more complex when using natural language understanding (NLU) vs. today’s command-and-control systems but result in far better results. Rather than asking consumers to use a limited vocabulary according to a strict grammar, which often results in a frustrating user experience, ideally, users would have an interactive conversation with a system, resulting in a richer user experience but a more complex system implementation.  Users can then replace simple and dull commands such as “Volume Up”, with more natural terms such as, “Turn the radio up,” or “Make the music louder”.  The system would also be able to offer more natural responses such as, “Is the music loud enough?”  Additionally, the system could take other variables such into account, such as multiple passengers talking at once, and offer predictive responses such as turning the volume down.  The UI could also profile each regular user and learn the ways each person interacts with the system in order to offer better predictive success. 

Conclusion

With regards to system design, adding intelligence to a system increases the need for additional processing capacity and memory, not just to diagnose the system or make a prediction but also to implement the complex responses the system needs to take.  The complexity of predictive intelligence requires more processing and memory resources than what is currently offered in embedded systems. As these embedded systems take on more UI functionality, this issue is becoming more of a challenge, both on the processing and memory side of the equation, given the resource-limited nature of embedded systems.

Next:  Part II - Potential solutions for these exciting new trends, including the rise of the UI Processor.

About the Author

Alvin Wong is vice president of marketing and business development for Spansion's Programmable System Solutions.  Wong has over 20 years of experience in strategic marketing and managing business units at semiconductor and technology companies. His most recent position was general manager of Integrated Device Technology for its Advanced User Interface and prior to that was vice president and general manager of Leadis Technologies? Touch Business Unit.  He has also held vice president of marketing positions at Xceive Corporation and Infineon Technologies, and several positions with Philips. Wong completed the Executive Management Program at Indiana University Graduate School of Business and holds a B.S.E.E. from San Jose State University.