Egocentric vision
More actions
Egocentric vision, also called first-person vision (FPV), is the branch of computer vision that analyses images and video recorded by a camera worn on the body, so that the footage shows the world from the wearer's own point of view. In their 2012 paper in the Proceedings of the IEEE, Takeo Kanade and Martial Hebert describe first-person vision as sensing "the environment and the subject's activities from a wearable sensor", in contrast to the traditional approach of observing a person with fixed surveillance cameras; such a system can also record the wearer's head motion and, with eye tracking, where the wearer is looking.[1][2]
The field matters to augmented reality and virtual reality because head-worn devices see the world the same way. The authors of the Ego4D dataset paper note that "in both robotics and augmented reality, the input is a long, fluid video stream from the first-person or 'egocentric' point of view".[3] Large egocentric datasets such as EPIC-KITCHENS, Ego4D, Ego-Exo4D and Microsoft's HoloAssist, recorded on HoloLens 2, and recording devices such as Meta's Project Aria research glasses support research on perception for AR assistants and other head-worn computing.[3][4][5]
Terminology
No single name has been settled on. In a 2015 survey in IEEE Transactions on Circuits and Systems for Video Technology, Alejandro Betancourt, Pietro Morerio, Carlo Regazzoni and Matthias Rauterberg wrote that "First Person Vision (FPV) is arguably the most commonly used" name, while "Egocentric Vision or Ego-vision" had also grown in popularity.[6] The large datasets of the late 2010s and 2020s, such as EPIC-KITCHENS, Ego4D and HoloAssist, use "egocentric" in their titles.[7][3][5] Video from cameras placed around a person, rather than on them, is called third-person or exocentric video; the Ego-Exo4D dataset records both views of the same activity at once.[8]
Characteristics of egocentric video
Betancourt and colleagues list several advantages of the first-person view. The wearer, often without meaning to, keeps the most relevant part of the scene in frame, which reduces the need for controlled multi-camera setups. Hands and the objects being handled appear in predictable places: active objects tend to sit near the centre of the frame, while hands tend to appear in the bottom left and bottom right. Eye and head movements can also carry information about the wearer's internal state.[6]
The same survey lists the main difficulties. The camera is always moving, which makes it hard to separate background from foreground. Lighting varies widely between places and times of day. Many applications need real-time processing, and processing on smart glasses has to fit within limited battery life, compute and communication bandwidth.[6] The Ego4D authors add that passively captured wearable video "entails unusual viewpoints, motion blur, and lacks temporal curation", unlike internet video chosen and framed by a photographer, and that models pre-trained on third-person data suffer from a sizeable domain mismatch.[3]
A 2024 survey by Chiara Plizzari, Gabriele Goletto, Antonino Furnari and colleagues, published in the International Journal of Computer Vision, describes further problems. For 3D scene understanding, sharp head rotations and mostly forward motion produce short, noisy feature tracks. For action recognition, the camera wearer is largely outside the field of view, so the main actor is only partly observed.[9]
History
Wearable computing origins
Betancourt's survey traces the analysis of first-person video back to 1997 and singles out work at the MIT Media Lab in the late 1990s and early 2000s and the wearable devices built by Steve Mann. It cites Mann's 1997 article "Wearable Computing: A First Step Toward Personal Imaging" in the IEEE magazine Computer, in which he described an always-ready computer "worn like clothing" whose screen "also serves as a viewfinder".[6] In 1998, Thad Starner, Bernt Schiele and Alex Pentland of the MIT Media Laboratory presented "Visual Contextual Awareness in Wearable Computing" at the second International Symposium on Wearable Computers. The paper argued that small, body-mounted video cameras would let a wearable computer observe its user, and it introduced a vision-based assistant that tracked the wearer's location and task during the real-space game Patrol without off-body infrastructure. The authors wrote that wearable computers could experience the life of the user in a "first-person" sense, and that the work continued augmented reality research begun in 1995 on binding virtual data to physical locations.[10]
Formation of a research field
The first Workshop on Egocentric Vision was held at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in 2009. A second IEEE workshop on egocentric (first-person) vision followed at CVPR 2012 in Providence, Rhode Island, on 17 June 2012; its organisers noted that advances in camera miniaturisation and mobile computing had made it feasible to capture and process such footage.[11]
Several papers from this period set out the core tasks. At CVPR 2011, Alireza Fathi, Xiaofeng Ren and James Rehg learned object models from egocentric video of household activities when only the names of the objects in each sequence were known, using a bottom-up segmentation that split each frame into hand, object and background.[12] At CVPR 2012, Hamed Pirsiavash and Deva Ramanan introduced a dataset and methods for detecting activities of daily living in first-person views, concluding that real-world recognition of these activities is "all about the objects being interacted with".[13] At the same conference, Yong Jae Lee, Joydeep Ghosh and Kristen Grauman of the University of Texas at Austin summarised a camera wearer's day as a storyboard built around the important people and objects the wearer interacted with.[14] In 2013, Yin Li, Alireza Fathi and James Rehg predicted where the wearer looks by combining head motion and hand location computed from the video itself, and reported that plugging the predicted gaze into existing methods improved the recognition of daily actions and the segmentation of foreground objects.[15]
Betancourt's survey groups the hardware of the early 2010s into three classes: smart glasses with sensors, processing and a head-up display, which the authors saw as "the starting point of an augmented reality system"; action cameras used by athletes and lifeloggers, which researchers adopted while waiting for smart glasses; and eye trackers.[6]
Large-scale datasets
From 2018 the field moved to much larger, crowd-collected datasets. EPIC-KITCHENS, presented at the European Conference on Computer Vision (ECCV) in 2018 by Dima Damen and colleagues from the University of Bristol, the University of Catania and the University of Toronto, was recorded by 32 participants in their own kitchens in four cities using a head-mounted GoPro camera. Participants narrated their own videos after recording, and these narrations were the basis for the crowd-sourced annotations.[7] Its extension, EPIC-KITCHENS-100, published in the International Journal of Computer Vision, grew the collection to 100 hours in 45 environments and added a "test of time" evaluation of whether models trained on the 2018 footage generalise to footage collected two years later.[16]
Facebook AI announced Ego4D on 14 October 2021, describing it as a project with a consortium of 13 universities and labs and framing egocentric perception as something future AR glasses and VR headsets would need.[17] The CVPR 2022 paper reports 3,670 hours of daily-life video from 931 camera wearers in 74 locations across nine countries. To avoid models overfitting to one device, the teams used seven head-mounted cameras: GoPro, Vuzix Blade, Pupil Labs, ZShades, ORDRO EP6, iVue Rincon 1080 and Weeview. The paper notes that a downward-pointing head-mounted GoPro gives a high-resolution view of the hands, while a heads-up camera such as the Vuzix shares the vantage of the eyes but misses interactions close to the body.[3] The dataset was released on 17 February 2022 to researchers who sign its license agreement.[18]
Ego-Exo4D, released on 13 December 2023, pairs first-person video captured with Project Aria glasses with four to five stationary GoPro cameras around the participant. It holds 1,286 hours of video of skilled activities, such as sports, music, dance and bicycle repair, from 740 participants in 13 cities, together with eye gaze, 3D point clouds, camera poses and expert commentary.[8][19] The University of Bristol described the project as "the largest ever deployment of the Aria glasses in the academic research community".[20]
| Dataset | Year | Capture device | Scale and content |
|---|---|---|---|
| GTEA (Georgia Tech Egocentric Activities) | 2011 | Camera mounted on a cap | 7 types of daily activity, each performed by 4 subjects[12][21] |
| ADL (Activities of Daily Living) | 2012 | Chest-mounted GoPro | More than 10 hours and over a million frames from 20 people in their own homes[13] |
| EGTEA Gaze+ | 2017 (last update) | Eye-tracking glasses with HD video | 28 hours of cooking from 86 sessions of 32 subjects, with gaze tracking at 30 Hz[21] |
| EPIC-KITCHENS | 2018 | Head-mounted GoPro | 55 hours, 32 kitchens, 39,600 action segments, 454,300 object bounding boxes[7] |
| EPIC-KITCHENS-100 | 2020 (preprint), 2022 (journal) | Head-mounted cameras | 100 hours, 20 million frames, 90,000 actions, 45 environments[16] |
| Ego4D | 2021 (announced), 2022 (released) | Seven head-mounted camera models | 3,670 hours, 931 camera wearers, 74 locations, 9 countries[3] |
| HoloAssist | 2023 | Microsoft HoloLens 2 | 166 hours from 222 participants in 350 instructor-performer pairs, 20 manipulation tasks[5] |
| Ego-Exo4D | 2023 | Project Aria glasses plus 4-5 static GoPro cameras | 1,286 hours, 740 participants, 13 cities[8][19] |
Tasks and benchmarks
Research tasks in egocentric vision include recognising and anticipating actions, detecting the objects a person handles, segmenting and tracking hands, predicting gaze, summarising long recordings and localising the wearer in 3D. EPIC-KITCHENS-100 defines challenges for action recognition, action detection, action anticipation, cross-modal retrieval from captions and unsupervised domain adaptation.[16]
Ego4D organises its benchmarks around past, present and future. Episodic memory asks a system to find where in a user's past video a query can be answered (for example "Where did I leave my keys?"); other benchmarks cover hand and object interaction, audio-visual diarization (who spoke and when), social interaction (who is talking to or looking at the wearer) and forecasting of future movement and interactions.[3][17] The paper describes an augmented reality assistant that processes the egocentric video stream as a possible way to give users "super-human memory" if it could index their visual experience and answer queries.[3] Ego-Exo4D adds benchmarks for fine-grained activity understanding, proficiency estimation, translation between first- and third-person views, and 3D hand and body pose.[8]
Applications in VR and AR
Hand tracking in headsets
Hand tracking in standalone headsets is an egocentric vision problem, since the cameras sit on the user's head. MEgATrack, a system presented at SIGGRAPH 2020 by Shangchen Han and colleagues at Facebook (now Meta), tracks hands in real time using four fisheye monochrome cameras, with neural networks for hand detection and keypoint estimation. The authors report that it runs at 60 Hz on a PC and 30 Hz on a mobile processor, and that it is the default hand tracking feature on the Oculus Quest, where it powers input and social presence.[22]
Research glasses and headsets
Meta's Reality Labs built Project Aria as what its 2023 device paper calls "an egocentric, multi-modal data recording and streaming device", intended to support "always available, context-aware and personalized AI applications" on future AR glasses that are comfortable enough to wear all day.[4] When it announced the program in September 2020, Meta (then Facebook) said the glasses would capture video and audio from the wearer's point of view along with eye movement and location data, and described the device as a research tool that was not for sale.[23]
Mixed reality headsets are also used as egocentric capture tools. Microsoft's 2020 technical report on HoloLens 2 Research Mode describes such headsets as "powerful sensing devices with integrated compute capabilities" and documents an API that gives researchers the raw streams of the device's four visible-light tracking cameras, depth camera and inertial measurement unit.[24] Researchers at Microsoft and ETH Zurich used HoloLens 2 to record HoloAssist, in which a performer wearing the headset completes a physical task while a remote instructor watches the performer's egocentric video in real time and gives spoken guidance. The dataset captures RGB, depth, head pose, 3D hand pose, eye gaze, audio and IMU data and was built as a step toward AI assistants that guide people through tasks in the physical world; its benchmarks cover mistake detection, intervention type prediction and hand forecasting.[5]
Assistive scenarios
The 2024 survey by Plizzari and colleagues imagines a future wearable it calls EgoAI, "a wearable device that enables in-situ multimodal sensing from the wearer's perspective", and maps scenarios for it (at home, at work in a factory, as a tourist, in policing and in film production design) to existing research tasks, noting where current methods and datasets fall short. The authors also point to practical hardware limits, such as battery life of only a few hours under continuous capture.[9]
Privacy and ethics
Always-on wearable cameras record bystanders and private spaces. Betancourt's survey already listed privacy and battery life as issues raised by the first-person perspective.[6] For Ego4D, each partner institution developed its own policy, which generally required complying with institutional research review, obtaining informed consent from camera wearers (who could withdraw and could review and redact their own video), respecting the rights of others in private spaces and following de-identification requirements for personally identifiable information. Some social recordings show participants who agreed to remain unblurred.[3] In EPIC-KITCHENS, participants were asked to record alone in the kitchen and to remove items such as portraits or mirrors that could disclose their identity.[7] See Privacy in virtual and augmented reality for the wider debate around camera-equipped glasses and headsets.
See also
References
- ↑ Takeo Kanade, Martial Hebert (2012-08). "First-Person Vision". Proceedings of the IEEE, vol. 100, no. 8, pp. 2442-2453. https://doi.org/10.1109/JPROC.2012.2200554. Retrieved 2026-10-15.
- ↑ "First-Person Vision". Carnegie Mellon University Robotics Institute. https://publications.ri.cmu.edu/first-person-vision. Retrieved 2026-10-15.
- ↑ 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, et al. (2022-03-11). "Ego4D: Around the World in 3,000 Hours of Egocentric Video". Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2022 (arXiv:2110.07058). https://arxiv.org/abs/2110.07058. Retrieved 2026-10-15.
- ↑ 4.0 4.1 Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, et al. (2023-10-01). "Project Aria: A New Tool for Egocentric Multi-Modal AI Research". arXiv. https://arxiv.org/abs/2308.13561. Retrieved 2026-10-15.
- ↑ 5.0 5.1 5.2 5.3 Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, et al. (2023). "HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World". Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) 2023. Computer Vision Foundation. https://openaccess.thecvf.com/content/ICCV2023/papers/Wang_HoloAssist_an_Egocentric_Human_Interaction_Dataset_for_Interactive_AI_Assistants_ICCV_2023_paper.pdf. Retrieved 2026-10-15.
- ↑ 6.0 6.1 6.2 6.3 6.4 6.5 Alejandro Betancourt, Pietro Morerio, Carlo S. Regazzoni, Matthias Rauterberg (2015). "The Evolution of First Person Vision Methods: A Survey". IEEE Transactions on Circuits and Systems for Video Technology, vol. 25, no. 5, pp. 744-760 (arXiv:1409.1484). doi:10.1109/TCSVT.2015.2409731. https://arxiv.org/abs/1409.1484. Retrieved 2026-10-15.
- ↑ 7.0 7.1 7.2 7.3 Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, et al. (2018-07-31). "Scaling Egocentric Vision: The EPIC-KITCHENS Dataset". European Conference on Computer Vision (ECCV) 2018 (arXiv:1804.02748). https://arxiv.org/abs/1804.02748. Retrieved 2026-10-15.
- ↑ 8.0 8.1 8.2 8.3 Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, et al. (2024-09-25). "Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives". arXiv (expanded version of the CVPR 2024 paper). https://arxiv.org/abs/2311.18259. Retrieved 2026-10-15.
- ↑ 9.0 9.1 Chiara Plizzari, Gabriele Goletto, Antonino Furnari, Siddhant Bansal, Francesco Ragusa, Giovanni Maria Farinella, Dima Damen, Tatiana Tommasi (2024). "An Outlook into the Future of Egocentric Vision". International Journal of Computer Vision, vol. 132, pp. 4880-4936 (arXiv:2308.07123). doi:10.1007/s11263-024-02095-7. https://arxiv.org/abs/2308.07123. Retrieved 2026-10-15.
- ↑ Thad Starner, Bernt Schiele, Alex Pentland (1998). "Visual Contextual Awareness in Wearable Computing". Digest of Papers, Second International Symposium on Wearable Computers, pp. 50-57. doi:10.1109/ISWC.1998.729529. https://faculty.cc.gatech.edu/~thad/p/031_20_Activity/visual-context-awareness_ISWC98.pdf. Retrieved 2026-10-15.
- ↑ "Second IEEE Workshop on Egocentric (First-Person) Vision". Georgia Institute of Technology. 2012. https://egovision12.cc.gatech.edu/. Retrieved 2026-10-15.
- ↑ 12.0 12.1 Alireza Fathi, Xiaofeng Ren, James M. Rehg (2011). "Learning to Recognize Objects in Egocentric Activities". IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2011, pp. 3281-3288. doi:10.1109/CVPR.2011.5995444. https://mlanthology.org/cvpr/2011/fathi2011cvpr-learning/. Retrieved 2026-10-15.
- ↑ 13.0 13.1 Hamed Pirsiavash, Deva Ramanan (2012). "Detecting Activities of Daily Living in First-person Camera Views". IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2012. University of California, Irvine. doi:10.1109/CVPR.2012.6248010. https://vision.ics.uci.edu/papers/PirsiavashR_CVPR_2012_1/PirsiavashR_CVPR_2012_1.pdf. Retrieved 2026-10-15.
- ↑ Yong Jae Lee, Joydeep Ghosh, Kristen Grauman (2012). "Discovering Important People and Objects for Egocentric Video Summarization". University of Texas at Austin. doi:10.1109/CVPR.2012.6247820. https://vision.cs.utexas.edu/projects/egocentric/. Retrieved 2026-10-15.
- ↑ Yin Li, Alireza Fathi, James M. Rehg (2013). "Learning to Predict Gaze in Egocentric Video". IEEE International Conference on Computer Vision (ICCV) 2013, pp. 3216-3223. doi:10.1109/ICCV.2013.399. https://mlanthology.org/iccv/2013/li2013iccv-learning/. Retrieved 2026-10-15.
- ↑ 16.0 16.1 16.2 Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, et al. (2022). "Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100". International Journal of Computer Vision, vol. 130, no. 1, pp. 33-55 (arXiv:2006.13256). doi:10.1007/s11263-021-01531-2. https://arxiv.org/abs/2006.13256. Retrieved 2026-10-15.
- ↑ 17.0 17.1 "Teaching AI to perceive the world through your eyes". Meta AI. 2021-10-14. https://ai.meta.com/blog/teaching-ai-to-perceive-the-world-through-your-eyes/. Retrieved 2026-10-15.
- ↑ "Ego4D". Ego4D Consortium. https://ego4d-data.org/. Retrieved 2026-10-15.
- ↑ 19.0 19.1 "Ego-Exo4D". Ego-Exo4D Consortium. https://ego-exo4d-data.org/. Retrieved 2026-10-15.
- ↑ "Bristol co-development will give AI window into complex human activity". University of Bristol. 2023-12-01. https://bristol.ac.uk/news/2023/december/ai-window-into-complex-human-activity.html. Retrieved 2026-10-15.
- ↑ 21.0 21.1 "Georgia Tech Egocentric Activity Datasets". Georgia Institute of Technology (archived copy). https://web.archive.org/web/20250201044720/https://cbs.ic.gatech.edu/fpv/. Retrieved 2026-10-15.
- ↑ Shangchen Han, Beibei Liu, Randi Cabezas, Christopher D. Twigg, et al. (2020-08-17). "MEgATrack: Monochrome Egocentric Articulated Hand-Tracking for Virtual Reality". ACM Transactions on Graphics, vol. 39, no. 4 (SIGGRAPH 2020). Meta. doi:10.1145/3386569.3392452. https://ai.meta.com/research/publications/megatrack-monochrome-egocentric-articulated-hand-tracking-for-virtual-reality/. Retrieved 2026-10-15.
- ↑ "Privacy Matters: Project Aria". Meta Newsroom. 2020-09-16. https://about.fb.com/news/2020/09/privacy-matters-project-aria/. Retrieved 2026-10-15.
- ↑ Dorin Ungureanu, Federica Bogo, Silvano Galliani, Pooja Sama, Xin Duan, et al. (2020-08-25). "HoloLens 2 Research Mode as a Tool for Computer Vision Research". arXiv technical report. https://arxiv.org/abs/2008.11239. Retrieved 2026-10-15.