Markerless outside-in tracking: Difference between revisions

'''[[Markerless outside-in tracking]]''' is a subtype of [[positional tracking]] used in [[virtual reality]] (VR) and [[augmented reality]] (AR). In this approach, external [[camera]]s or other [[depth sensing]] devices positioned in the environment estimate the six-degree-of-freedom pose of a user or object without relying on any [[fiducial marker]]s. Instead, [[computer vision]] algorithms analyse the incoming colour or depth stream to detect and follow natural scene features or the user’s own body, enabling real-time [[motion capture]] and interaction.<ref name="Shotton2011" />

* **Sensing layer** – One or more fixed [[RGB-D]] or [[infrared]] depth cameras acquire per-frame point clouds. Commodity devices such as the Microsoft Kinect project a [[structured light]] pattern or use [[time-of-flight]] methods to compute depth maps.<ref name="Zhang2012" />

* **Segmentation** – Foreground extraction or person segmentation isolates user pixels from the static background.

* **Per-pixel body-part classification** – A machine-learning model labels each pixel as “head”, “hand”, “torso”, and so on (e.g., the Randomised Decision Forest used in the original Kinect).<ref name="Shotton2011" />

* **Skeletal reconstruction and filtering** – The system fits a kinematic skeleton to the classified pixels and applies temporal filtering to reduce jitter, producing smooth head- and hand-pose data that can drive VR/AR applications.

Although a single camera can suffice, multi-camera rigs extend coverage and mitigate occlusion problems. Open source and proprietary middleware (e.g., [[OpenNI]]/NITE, the Microsoft Kinect SDK) expose joint-stream APIs for developers.<ref name="OpenNI2013" />

Marker-based outside-in systems (HTC Vive Lighthouse, PlayStation VR) attach active LEDs or retro-reflective spheres to the headset or controllers; external sensors triangulate these explicit targets, achieving sub-millimetre precision and sub-10 ms latency. Markerless alternatives dispense with physical targets, improving user comfort and reducing setup time, but at the cost of:

* **Lower positional accuracy and higher latency** – Depth-sensor noise and computational overhead introduce millimetre- to centimetre-level error and ~20–30 ms end-to-end latency.<ref name="Baker2016" />

* **Sensitivity to occlusion** – If a body part leaves the camera’s line of sight, the model loses track until the part re-enters view.

| 2003 || [[EyeToy]] (PlayStation 2) || 2-D silhouette tracking with a single RGB camera for casual gesture-based games.<ref name="Sony2003" />

| 2014 – 2016 || Research prototypes || Studies showed Kinect V2 could supply 6-DOF head, hand, and body input to DIY VR HMDs.<ref name="KinectVRStudy" />

* **Gaming and Entertainment** – Titles like ''Kinect Sports'' mapped whole-body actions directly onto avatars. Enthusiast VR chat platforms still use Kinect skeletons to animate full-body avatars.

* **Rehabilitation and Exercise** – Clinicians employ depth-based pose tracking to monitor range-of-motion exercises without encumbering patients with sensors.<ref name="Baker2016" />

* **Interactive installations** – Museums deploy wall-mounted depth cameras to create “magic-mirror” AR exhibits that overlay virtual costumes onto visitors in real time.

* **Telepresence** – Multi-Kinect arrays stream volumetric representations of remote participants into shared virtual spaces.

* '''Reduced accuracy and jitter''' – Compared with marker-based solutions, joint estimates exhibit higher positional noise, especially during fast or complex motion.<ref name="Baker2016" />

<ref>Shotton, Jamie; Fitzgibbon, Andrew; Cook, Mat; Sharp, Toby; Finocchio, Mark; Moore, Bob; Kipman, Alex; Blake, Andrew (2011). Real-Time Human Pose Recognition in Parts from a Single Depth Image. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2011​

microsoft.com

.</ref>

<ref>Zeng, Wenjun; Zhang, Zhengyou (2012). Microsoft Kinect Sensor and Its Effect. IEEE MultiMedia, 19(2):4–10​

.</ref>

<ref>OpenNI Foundation (2010). OpenNI 1.5.2 User Guide. “OpenNI is an open source API that is publicly available at www.OpenNI.org.”&#8203;:contentReference[oaicite:2]{index=2}.</ref>

<ref>Pfister, Andreas; West, Niels; et al. (2022). Applications and limitations of current markerless motion capture methods for clinical gait biomechanics. Journal of Biomechanics, 129:110844. “While markerless temporospatial measures generally appear equivalent to marker-based systems, joint center locations and joint angles are not yet sufficiently accurate for clinical applications.”​

pmc.ncbi.nlm.nih.gov

.</ref>

<ref>Pham, Alex (2004-01-18). EyeToy Springs From One Man’s Vision. Los Angeles Times. “the $50 EyeToy, a tiny camera that enables video game players to control the action by jumping around and waving their arms…”​

latimes.com

.</ref>

<ref>Microsoft News Center (2010-11-04). The Future of Entertainment Starts Today as Kinect for Xbox 360 Leaps and Lands at Retailers Nationwide. “Kinect for Xbox 360 lets you use your body and voice to play your favorite games... No buttons. No barriers. Just you.”​

news.microsoft.com

.</ref>

<ref>Lange, Belinda; Rizzo, Skip; Chang, Chien-Yen; Suma, Evan A.; Bolas, Mark (2011). Markerless Full Body Tracking: Depth-Sensing Technology within Virtual Environments. Proc. I/ITSEC 2011. “FAAST is middleware to facilitate integration of full-body control with virtual reality applications... (e.g. Microsoft Kinect).”​

.</ref>

<ref>Good, Owen S. (2017-10-25). Kinect is officially dead. Really. Officially. It’s dead. Polygon. “Microsoft has confirmed it is no longer manufacturing Kinect and none will be sold once retailers run out.”​

polygon.com

.</ref>