Augmented virtuality

This page is a stub, please expand it if you have more information.

See also: Mixed reality and Augmented Reality

Augmented virtuality (AV) is a class of mixed reality in which a predominantly virtual environment is supplemented with real-world objects, people, or imagery. It is one of the regions of the reality-virtuality continuum defined by Paul Milgram and Fumio Kishino in 1994, and it sits between augmented reality and a fully virtual environment, closer to the virtual end of the scale.

Augmented virtuality is the converse of augmented reality (AR). Where AR adds virtual content to a view of the real world, AV adds real content to an otherwise computer-generated world. Common examples include a real person captured by a camera and composited into a virtual scene, and a live video texture mapped onto an object inside a virtual world.

Origin in the reality-virtuality continuum

The term comes from the taxonomy published by Milgram and Kishino, two researchers then associated with the ATR Communication Systems Research Laboratories in Japan, in the 1994 paper "A Taxonomy of Mixed Reality Visual Displays". The paper introduced both the term "mixed reality" and the reality-virtuality (RV) continuum, a one-dimensional scale running from a completely real environment at one end to a completely virtual environment at the other. Everything between the two extremes is mixed reality: a blend of real and virtual content.

On that continuum, augmented reality lies near the real end and augmented virtuality lies near the virtual end. AR describes cases in which a view of a real environment is augmented with virtual objects, and AV describes the symmetric case in which a virtual environment is augmented with real objects or stimuli. A 2021 review by Richard Skarbez, Missie Smith and Mary C. Whitton summarizes augmented virtuality as an environment "where most of the content is virtual but there is some awareness or inclusion of real world objects".

Milgram and Kishino did not treat mixed reality as a single point but classified its visual displays along three further dimensions: extent of world knowledge (how much the system knows about the scene), reproduction fidelity (the quality of the rendered imagery), and extent of presence metaphor (the degree to which the viewer feels present in the displayed scene). The same group, with Haruo Takemura and Akira Utsumi as co-authors, expanded the framework in the conference paper "Augmented Reality: A class of displays on the reality-virtuality continuum", published in the SPIE proceedings volume "Telemanipulator and Telepresence Technologies". Both papers have been cited thousands of times and are among the foundational references for VR, AR, and XR research.

Relationship to other terms

Augmented virtuality is distinguished from neighboring terms mainly by which kind of content dominates and which is added:

Term	Dominant environment	What is added	Position on the continuum
Virtual reality	Virtual	Nothing real (fully synthetic)	Virtual extreme
Augmented virtuality	Virtual	Real objects, people, or imagery	Near the virtual end
Augmented reality	Real	Virtual objects	Near the real end
Real environment	Real	Nothing virtual	Real extreme

Both augmented virtuality and augmented reality fall under mixed reality, the band of the continuum where real and virtual content coexist. In practice the boundary between AV and AR is not always sharp, and the term mixed reality is often used on its own when a system mixes real and virtual content without specifying which side dominates.

The 2021 Skarbez, Smith and Whitton review argued that the original continuum needs revision. It proposed that the continuum is discontinuous because a perfectly convincing virtual environment is not currently achievable, that mixed reality should be defined in terms of a single combined percept rather than a type of display, and that the three classifying dimensions be replaced with extent of world knowledge, immersion, and coherence. These revisions do not remove augmented virtuality as a category; they refine how its position and properties are described.

How it works

Producing augmented virtuality requires capturing real elements and placing them inside a rendered world in a way that is spatially consistent with the virtual content. Systems commonly use cameras, depth sensors, and computer-vision techniques to capture physical objects or people and represent them in the digital environment in real time.

A frequent technique is chroma keying, in which a subject is filmed against a uniform background (usually a green screen) and that background color is removed so the remaining image can be inserted into a virtual scene. This is the same compositing method used by television weather presenters, who appear to stand in front of computer-generated maps, and by video-conferencing tools such as Zoom and FaceTime, which replace a participant's real surroundings with a chosen backdrop. A widely cited illustration of AV is a video of a real human face mapped onto the three-dimensional mesh of an avatar's head, so that a virtual character carries the live appearance of a real person.

When the inserted element must occlude and be occluded by virtual geometry correctly, a flat keyed image is not enough, because it carries no depth information. Adding a depth sensor lets the system record how far each part of the real subject is from the camera and represent it correctly relative to the virtual content, rather than as a flat cut-out.

Uses

Augmented virtuality is used to bring real people, tools, or data into a virtual environment when full synthesis would lose useful realism. Reported application areas include manufacturing and training, where physical control panels or instruments are combined with simulated machinery so that operators handle real hardware inside a virtual workstation, and education, where real objects are studied within a virtual setting.

A consumer example is mixed reality capture for VR. An external camera films a player wearing a head-mounted display in front of a green screen, software keys out the background, and the player's real image is composited into the virtual game world to produce third-person footage that shows the person inside the experience. The social VR application Bigscreen added a green screen environment in November 2020 that uses third-person capture tools and external software to remove a chroma-key background and place a user's representation over chosen footage for video calls, streaming, and content creation.

In research, the same idea underlies telepresence systems, in which live imagery of a remote person or place is brought into a shared virtual space, a use case the original SPIE paper situated directly within the reality-virtuality continuum.