RGB camera

An RGB camera is a digital camera that records visible light as a color image, capturing the three additive primary colors (red, green, and blue) at each point in the scene. Most consumer RGB cameras use a single image sensor (CCD or, more commonly today, CMOS) overlaid with a color filter array, together with image-processing electronics that reconstruct a full-color picture from the sensor's raw output. In virtual reality (VR), augmented reality (AR), and mixed reality (MR) headsets, RGB cameras are the sensors that supply color information for video passthrough of the real world, for spatial photo and video capture, and, in some designs, as one input to environment tracking.

The term distinguishes a color camera from the monochrome (black and white) and infrared cameras that headsets also use, mainly for positional tracking and depth sensing. Because a color sensor spends part of its resolution and light sensitivity on separating colors, headset makers often pair RGB passthrough cameras with separate monochrome cameras tuned for tracking, rather than using one camera type for everything.

How an RGB camera works

A digital image sensor is a grid of light-sensitive photosites (pixels). A bare photosite measures only the intensity of the light that reaches it, not its color, so an unmodified sensor produces a grayscale image. To record color, almost all single-sensor cameras place a color filter array over the pixel grid so that each pixel sees light of only one primary color.

The most common color filter array is the Bayer filter, patented by Bryce Bayer of Eastman Kodak in 1976 (U.S. patent 3,971,065). In a Bayer mosaic, half of the pixels carry green filters and a quarter each carry red and blue filters, arranged in a repeating 2x2 pattern of one red, two green, and one blue. Green is favored because the human eye is more sensitive to green light, and the extra green samples yield an image that appears sharper and less noisy.

Each pixel therefore measures only one of the three colors directly; the two missing color values at every pixel are estimated from neighboring pixels by an interpolation process called demosaicing. The unprocessed sensor output before demosaicing is called raw image data. Simple demosaicing copies or averages nearby same-color pixels, while more advanced methods follow image edges to reduce color artifacts. After demosaicing the camera applies white balance, color correction, gamma, and noise reduction to produce the final RGB image.

A variant relevant to headsets is the RGB-IR sensor, whose filter array adds infrared-passing pixels alongside the red, green, and blue ones, so a single camera can output both a color image and a near-infrared image. RGB-IR modules use a dual-bandpass lens filter that admits both visible and near-infrared light, instead of the infrared-cut filter used on ordinary color cameras, and separate the two images with dedicated processing.

RGB cameras versus monochrome and depth cameras

A VR or AR headset typically carries several camera types whose jobs differ.

An RGB camera records color and is used where the image is shown to the user or stored as a photo or video, principally video passthrough. A monochrome camera records only brightness. Removing the color filter array lets every pixel collect light of all visible wavelengths, which improves low-light sensitivity and avoids the resolution loss of demosaicing, and the smaller data stream is faster to process. These properties suit head and controller tracking, where geometry matters more than color. Tracking cameras commonly use a global shutter, which exposes the whole frame at once and so avoids the skew that a rolling shutter introduces during fast head or hand motion. The OmniVision OG0TC, announced in July 2024, is an example of a dedicated tracking-camera sensor: a 400x400 pixel (about 0.16 megapixel) black-and-white global-shutter CMOS sensor in a 1/14.46-inch format, marketed primarily for inward-facing tracking cameras such as eye and face tracking in AR, VR, and MR headsets.

A depth camera (a depth sensor or RGB-D camera) measures the distance to surfaces rather than their color, using techniques such as structured light, time of flight, or LiDAR. When depth data is combined with a color image, the result is an RGB-D image, which pairs color with a per-pixel distance value. The Microsoft Kinect popularized low-cost RGB-D sensing and made the format common in research on three-dimensional scene reconstruction, where multiple RGB-D views are merged into colored meshes or point clouds for VR. Because these roles trade off against one another, modern headsets often combine separate camera types and fuse their outputs through sensor fusion so the passthrough view stays color-correct, stable, and aligned to the user's head.

Role in VR and AR

Color video passthrough and mixed reality

Video passthrough shows the wearer the outside world on the headset's internal displays using its outward-facing cameras, which lets a fully enclosed VR headset present mixed reality by blending camera video with rendered graphics. The quality of that view depends directly on the RGB cameras: their resolution sets how sharp the real world looks, and their light sensitivity sets how well it holds up in dim rooms.

Early standalone headsets used only monochrome cameras for passthrough. The Meta Quest 2 (2020) carried four monochrome fisheye cameras at the front corners, intended mainly for tracking, and reused them for a low-resolution black-and-white passthrough view used to define the play boundary and to glance at the real world. The Meta Quest 3 (2023) moved to color passthrough using two 4-megapixel RGB cameras plus a depth sensor, which Meta said gave roughly ten times the passthrough resolution of the Quest 2. Reviewers found the color view clear enough to read a phone screen, though still somewhat soft.

The Quest 3's camera layout shows the division of labor described above: two forward-facing rolling-shutter RGB cameras handle color passthrough, while two forward-facing global-shutter grayscale cameras handle visual tracking, supported by infrared cameras and an infrared depth projector.

High-end and standalone examples

Higher-resolution RGB passthrough is used in headsets aimed at professional and industrial work. The Varjo XR-3 (2021) used dual 12-megapixel passthrough cameras running at 90 Hz with under 20 ms of latency, combined with LiDAR-and-RGB depth fusion over a 0.4 to 5 m range. The Apple Vision Pro (2024) has a stereoscopic main camera system rated at 6.5 stereo megapixels with an 18 mm, f/2.00 lens for spatial photo and video, part of a sensor array that Apple lists as two main cameras, six world-facing tracking cameras, four eye-tracking cameras, a TrueDepth camera, and a LiDAR scanner. Apple states that a pair of high-resolution cameras send more than one billion pixels per second to the displays so the wearer can see the surroundings.

The table below summarizes the outward-facing RGB passthrough cameras of several headsets. Values for tracking and depth sensors are omitted.

Headset	Year	RGB passthrough cameras	Notes
Meta Quest 2	2020	None (monochrome only)	Low-resolution black-and-white passthrough from tracking cameras
Varjo XR-3	2021	2 x 12 MP at 90 Hz	Under 20 ms latency; LiDAR + RGB depth fusion
Meta Quest 3	2023	2 x 4 MP	Color passthrough plus a depth sensor; about 10x Quest 2 resolution
Apple Vision Pro	2024	2 main cameras, 6.5 stereo MP	18 mm f/2.00; over 1 billion pixels/sec to displays

Spatial capture and tracking input

Beyond live passthrough, RGB cameras let headsets record spatial photos and videos that store a stereo color image of a scene for later viewing in three dimensions; the Apple Vision Pro's main camera system is built for this. RGB images can also feed tracking and scene understanding. Visible-light features support computer vision tasks such as hand tracking and recognizing objects and surfaces, and color frames can be one input to a SLAM (simultaneous localization and mapping) pipeline, although headsets often prefer monochrome global-shutter cameras for the core inside-out tracking loop because of their speed and low-light behavior.

Limitations in headsets

RGB passthrough cameras have practical limits. They need adequate light: the Meta Quest 3's color cameras require reasonable room lighting to avoid a grainy image. The cameras sit a few centimeters in front of and apart from the eyes, so their viewpoint does not match the eyes' natural one, and the system must reproject the images to the correct perspective, a step that can add blur, warping, and depth errors. Demosaicing and the color filter array mean a color sensor delivers less effective spatial resolution and less light per pixel than a monochrome sensor of the same pixel count, one reason tracking is usually handled by separate cameras. Finally, passthrough adds latency between the camera and the display, which Varjo, for example, holds below 20 ms on the XR-3 to keep the view comfortable.