Markerless outside-in tracking

This page is a stub, please expand it if you have more information.

See also Outside-in tracking, Markerless tracking, Positional tracking

Introduction

Markerless outside-in tracking is a subtype of positional tracking used in both virtual reality (VR) and augmented reality (AR). It places external cameras or other depth sensing devices around the play area and estimates a user’s six-degree-of-freedom pose without any worn fiducial markers. Instead, the system runs computer vision algorithms—most famously the per-pixel body-part classifier introduced for Microsoft’s Kinect—to create a real-time motion capture skeleton.

Underlying technology

A typical markerless outside-in pipeline includes:

• Sensing layer – One or more fixed RGB-D or infrared depth cameras (e.g., the first-generation Kinect) acquire point-cloud frames. Depth is measured with structured light or time-of-flight illumination.
• Segmentation – Foreground extraction isolates user pixels from the static background.
• Body-part classification – A decision-forest classifier labels each depth pixel as head, hand, torso, and so on, following Shotton et al.
• Skeletal fitting and filtering – Joint hypotheses are fitted to a kinematic model and temporally smoothed, generating continuous head- and hand-pose streams.

Open software stacks such as OpenNI/NITE expose these joint streams to developers.

Markerless vs. marker-based tracking

Marker-based outside-in systems (HTC Vive Lighthouse, PlayStation VR) attach active LEDs or reflective spheres to the headset or controllers, achieving millimetre-level accuracy. Markerless systems remove that hardware layer but incur:

• Susceptibility to occlusion and environmental lighting.
• Higher positional noise and latency (~20–30 ms end-to-end).

History and notable systems

Year	System	Technical note
2003	EyeToy (PlayStation 2)	2-D silhouette tracking with a single RGB webcam.
2010	Kinect for Xbox 360	Structured-light depth sensor providing full-body skeletons for up to six users.
2011	Kinect + FAAST middleware	Demonstrated low-cost VR interaction with markerless tracking.
2017	Kinect production ends	Microsoft ceased manufacturing Kinect as industry moved to other tracking paradigms.

Applications

• **Gaming and entertainment** – Titles such as Kinect Sports map whole-body gestures to avatars; hobbyists still use Kinect for full-body VR chat avatars.
• **Rehabilitation and exercise** – Depth-based pose tracking supports remote physiotherapy and balance-training systems.
• **Interactive exhibits** – Museums mount depth cameras to create “magic-mirror” AR overlays.
• **Telepresence** – Multi-camera arrays stream volumetric avatars into shared virtual spaces.

Advantages

• No wearable markers, enhancing comfort.
• Quick single-sensor setup and lower hardware cost.
• Ability to track multiple users at once.

Disadvantages

• Occlusion sensitivity and limited camera field-of-view.
• Lower accuracy than marker-based alternatives.
• Performance degradation in bright sunlight or on reflective surfaces.