Natural user interface

A natural user interface (NUI) is a style of human-computer interface in which people operate a computer through behaviors they already use in daily life, such as touch, hand and body gestures, speech, handwriting and eye gaze, instead of a keyboard, mouse or command language. From the late 2000s Microsoft promoted the term for its Surface tabletop computer, the Kinect motion sensor and related touch and speech work, and it is the subject of the 2011 design book Brave NUI World by Daniel Wigdor and Dennis Wixon. Wigdor and Wixon define the natural user interface as "one that leads users, at all times, to feel like they are effortlessly interacting with the technology".

The label has been contested from the start. In a 2010 column titled "Natural user interfaces are not natural", Donald A. Norman argued that "most gestures are neither natural nor easy to learn or remember" and that gestural systems still need feedback, visible options and undo. In virtual reality (VR) and augmented reality (AR) the idea is central to headset input: controller-free hand tracking, eye-gaze targeting combined with a finger pinch, and voice commands are all presented by headset makers as more natural alternatives to handheld controllers, while 3D interaction researchers study when such "natural" techniques actually help users and when they do not.

Reviewed 6 October 2026. Re-checked every quote, date, author and venue against the cited Microsoft, Apple, Meta, UploadVR, Road to VR, Web Directions, UX Magazine, Elsevier and jnd.org pages, the Bowman 2012, Jacob 1991/2008, Mann 2007, Pfeuffer 2024 and Consumed Endurance papers, and all 11 DOIs via Crossref. About review dates.

Definition

There is no standardized definition of a natural user interface. The term is usually explained by contrast with the earlier command-line interface (CLI) and graphical user interface (GUI). The description of a talk that August de los Reyes, then principal design director for Microsoft Surface, gave at Web Directions South in Sydney on 25 September 2008 stated that Surface "marks a departure from graphical user interface or GUI into the world of Natural User Interface or NUI".

Wigdor and Wixon, who had both worked on the Microsoft Surface product (Wigdor as its user experience architect, Wixon as its head of research), place the "natural" in the user's experience rather than in the input technology. In their account, "novice use is supported, and expert use is the goal", and the transition between the two should be effortless. They reject the idea that natural interaction can be achieved by copying the physical world: "This experience cannot be created simply by mimicry and metaphor. There does not exist some innate way of interacting with complex technology." They conclude that "new interfaces need to be learned", and that good design makes that learning rapid and effective.

Bill Buxton of Microsoft Research stressed combinations of input methods over any single one. In a 2010 Microsoft Research video on designing natural user interfaces, he said that "it's not about speech, it's not about gesture, it's not even about the phone, and it's not about human-to-human communication", and described the aim as how these things "work together in a natural and seamless way that reduces complexity for the users".

Academic HCI research describes the same family of interfaces with its own frameworks. Robert J. K. Jacob of Tufts University and colleagues proposed reality-based interaction (RBI) at CHI 2008 as a unifying concept for "post-WIMP" interfaces, a group in which they explicitly included virtual, augmented and mixed reality, tangible interfaces and ubiquitous computing. They argued that these styles build "on users' pre-existing knowledge of the everyday, non-digital world" and identified four themes they draw on: naive physics, body awareness and skills, environment awareness and skills, and social awareness and skills. The authors also held that "mimicking reality alone is not enough": designers should give up realism only explicitly and in exchange for qualities such as expressive power, efficiency, versatility, ergonomics, accessibility or practicality.

For 3D interfaces, Doug Bowman, Ryan McMahan and Eric Ragan defined the naturalism, or interaction fidelity, of an interface as "the objective degree with which the actions (characterized by movements, forces, or body parts in use) used for a task in the UI correspond to the actions used for that task in the real world". Under this definition, swinging the arms to hit a virtual golf ball is more natural than pressing a sequence of buttons.

History

Antecedents

Most of the input methods later grouped under the NUI label are much older than the term. In his 2010 column, Norman cited a 1998 review by Brad Myers that describes gesture work in the 1960s and its first commercial use in computer-aided design systems and the Apple Newton of 1992, and he pointed to Myron Krueger's "artificial reality" work of the early 1980s, which was his own first encounter with gestural interaction on large projected images. Citing a review by Bill Buxton, Norman dated the first multi-touch system designed for human-computer interaction to a 1982 master's thesis by Nimish Mehta, and he noted that the theremin, a gesture-controlled electronic instrument, was patented in 1928.

Steve Mann titled a 2007 conference paper "Natural Interfaces for Musical Expression". Mann noted that traditional instruments such as the harp, the guitar and the tin whistle have a "directness of user-interface", in which the musician's fingers are in direct physical contact with the sound-producing medium, and presented new instruments designed to keep that directness, such as the hydraulophone, in which sound is produced by pressurized hydraulic fluid that the player's fingers touch.

Microsoft Surface, Kinect and the NUI label

Microsoft announced Surface on 29 May 2007 at the D: All Things Digital conference. The company described it as a 30-inch display in a table-like form factor that "provides effortless interaction with all forms of digital content through natural gestures, touch and physical objects", recognizes many points of contact at once, and "works without a traditional mouse or keyboard".

At the Consumer Electronics Show in January 2010, Steve Ballmer and Robbie Bach used their keynote to announce that "Project Natal" for Xbox would ship that year. Microsoft's corporate blog described Natal as "part of a bigger effort around what scientists call Natural User Interface (NUI)", covering touch, speech, gestures, handwriting and vision, and listed Windows 7, Microsoft Surface and the Ford SYNC in-car system as products that already used parts of it. Norman opened his column with a statement he attributed to Ballmer, who predicted that 2010 would be remembered as the year the industry "expanded beyond the mouse and keyboard and started incorporating more natural forms of interaction such as touch, speech, gestures, handwriting, and vision".

Microsoft announced Natal's official name, Kinect for Xbox 360, at E3 in June 2010, and the device launched in North America on 4 November 2010. Microsoft marketed it as controller-free play in which players "use your body and voice to play your favorite games". The term also entered Microsoft's developer tools: in the native API of the Kinect for Windows SDK (versions 1.5 to 1.8), the sensor functions, headers and constants carry a "NUI" prefix, for example NuiInitialize, NuiApi.h and the NUI_SKELETON flags used for body tracking.

Wigdor and Wixon's Brave NUI World: Designing Natural User Interfaces for Touch and Gesture followed in 2011, published by Morgan Kaufmann. Its publisher describes it as a practical guide to designing touch- and gesture-based interfaces, written by members of the Microsoft team that developed the Surface tabletop.

Criticism

In his 2010 essay, Norman described "natural" as a "marketing name" and wrote that "marketing rhetoric is ahead of reality". His argument was that gestures lack properties that made the GUI learnable: because gestures are ephemeral, they leave no record when they fail, and a purely gestural system makes it hard to discover what actions exist. Gestures also vary across cultures, and a system must avoid reacting to movements that were not meant as input. Norman used the Nintendo Wii's bowling game as an example of a gesture that was "too natural": players released the controller's button as if letting go of a ball, and in the heat of the game some also relaxed their grip on the controller, which could fly through the air and sometimes broke the television screen. He concluded that natural user interfaces are not natural, "but they will be useful", and that conventions and standards for gestures would take time to develop. Norman and Jakob Nielsen extended the argument later the same year in "Gestural interfaces: a step backward in usability", which held that new gestural products were ignoring established interaction-design principles.

A related line of research asks users, rather than designers, which gestures feel natural. Jacob O. Wobbrock, Meredith Ringel Morris and Andrew D. Wilson noted that many surface-computing prototypes used gestures "created by system designers" that "are not necessarily reflective of user behavior". In a 2009 study they showed 20 participants the effect of a command and asked them to perform its cause, logging 1,080 gestures for 27 commands. They found that users rarely cared how many fingers they used, preferred one hand to two, were strongly influenced by desktop idioms, and showed little agreement on some commands, which suggested a need for on-screen widgets. Thammathip Piumsomboon, Adrian Clark, Mark Billinghurst and Andy Cockburn applied the method to AR in 2013, eliciting 800 gestures for 40 tasks from 20 participants and deriving a user-defined gesture set for AR from the gestures participants agreed on. Alessio Malizia and Andrea Bellucci made a related argument in a 2012 Communications of the ACM article, "The artificiality of natural user interfaces", whose summary line reads "Toward user-defined gestural interfaces".

Natural interaction in VR and AR

Naturalism in 3D user interfaces

Bowman, McMahan and Ragan's 2012 review article in Communications of the ACM examined naturalism in 3D interfaces such as those used in VR. Summarizing their own experiments at Virginia Tech and earlier studies, they reported that high levels of naturalism can enhance performance and the overall user experience, while traditional, less natural interaction styles can perform well but result in lower levels of presence, engagement and fun. Moderately natural techniques, however, "can be unfamiliar and detrimental to performance": in their Mario Kart Wii study, the less natural joystick controls were faster and more accurate than steering with the Wii Remote, with or without the Wii Wheel prop, although players rated the Wii Wheel the most fun; a moderately natural "human joystick" travel technique was hard to use even after practice. They concluded that "naturalism is most effective when very high levels of fidelity can be achieved, and when the resulting interface is familiar to users."

The same review describes "hyper-natural" techniques, which "use natural movements but make them more powerful by giving the user new abilities or intelligent guidance". In the Go-Go technique, normal arm extension makes the virtual arm extend far into the environment, so that distant objects can be selected by touching them; HOMER combines pointing-based selection with virtual-hand manipulation at a distance. The authors note that these enhanced abilities often come with a cost: in HOMER, the scaling applied to hand movements reduces precision. Their scaled HOMER variant applies a large scaling factor when the hand moves quickly and a small one when it moves slowly, which they report keeps the natural feel of HOMER while improving precision.

Headset input

Commercial headset makers adopted the language of natural interaction for controller-free input. When it announced hand tracking for the Oculus Quest on 25 September 2019, Facebook described it as enabling "natural interaction in VR using your own hands on an all-in-one device", with no extra hardware required. The feature reached Quest owners as an experimental option in the v12 software update in December 2019, initially for the home interface and a few first-party apps, with a pinch gesture acting as a click for button presses and as a click-and-hold for dragging scrollable areas. The hand-tracking SDK for developers became available the following week.

Microsoft's mixed reality design guidance calls the same goal "instinctual interactions". It states that designing within a multimodal framework, combining hand and eye tracking with natural language input, "is the key to creating instinctual experiences", and it groups mixed reality input into three primary models: hands and motion controllers, hands-free input (voice or gaze and dwell), and gaze and commit. For the HoloLens 2, Microsoft writes that it no longer asks users to memorize symbolic gestures, unlike the bloom and air tap gestures taught on the first-generation HoloLens. Instead, direct manipulation with articulated hand tracking relies on the design of UI affordances: an object meant to be grabbed with a two-finger pinch should be small, and one meant for a five-finger grab larger.

Apple introduced the Apple Vision Pro in June 2023 with a three-dimensional interface controlled by what the company called "the most natural and intuitive inputs possible", namely the user's eyes, hands and voice. Users look at an item to target it, tap their fingers to select, flick the wrist to scroll, and dictate with their voice. Apple's Human Interface Guidelines for visionOS split gestures into two categories. With an indirect gesture, a person looks at an object to target it and then manipulates it from a distance with the hands, for example by tapping finger and thumb together; Apple states that indirect gestures are comfortable at any distance. With a direct gesture, the person physically touches a virtual object, such as a key on the visionOS keyboard; because "people may find it tiring to keep their arms raised for extended periods, direct gestures are best for infrequent use."

Gaze and pinch

The eyes-plus-pinch model has an academic precedent. Ken Pfeuffer, Benedikt Mayer, Diako Mardanbegi and Hans Gellersen presented the Gaze + Pinch technique for VR at the 2017 ACM Symposium on Spatial User Interaction. It "integrates eye gaze to select targets, and indirect freehand gestures to manipulate them", so that a familiar pinch gesture can be applied to any object the user looks at, near or far, without a controller. In a 2024 article in IEEE Computer Graphics and Applications, Pfeuffer, Gellersen and Mar Gonzalez-Franco wrote that the term originated in that 2017 paper, that the paradigm was being adopted by headsets including the HoloLens 2 and Magic Leap devices, and that the Apple Vision Pro integrated it across the operating system. Their first design principle is a division of labor in which "the eyes perform selection tasks, the hands do the actual confirmation or manipulation work", which they argue avoids overloading the eyes with motor control, reduces physical fatigue and keeps each modality in its most naturalistic role.

Known problems

Research on natural input describes two problems that also apply to headsets. The first is the Midas touch problem of gaze input, described by Jacob in 1991: if looking at something triggers it, then "everywhere you look, another command is activated; you cannot look anywhere without issuing a command." For Gaze + Pinch, Pfeuffer, Gellersen and Gonzalez-Franco recommend that eye tracking act as input only at the moment a pinch is registered, so that eyes wandering around a scene do not accidentally select things. The second is arm fatigue during mid-air interaction, which Juan David Hincapié-Ramos and colleagues described in 2014 as "a feeling of heaviness in the upper limbs, a condition casually termed as the gorilla-arm effect", and for which they proposed a metric called Consumed Endurance. The same paper treats mid-air interaction as "a new class of natural user interface (NUI)". Apple's advice to reserve direct touch gestures for infrequent use reflects the same concern.

See also

• Human-computer interaction
• 3D user interface
• Hand tracking
• Eye tracking
• Gesture recognition
• Speech recognition
• Ray-casting selection
• VR UI/UX design
• Microsoft Kinect